Major release and update cycle for Fedora CoreOS

[sinnykumari]

For Fedora Atomic Host, we do Atomic Host release along with Fedora major release. This follows with new release every Two Weeks with updated content from Fedora updates. This helps users to receives updated (including security fixes) and tested content every Two Weeks. For a major CVE fix, we make exception and do an in-between releases.

For FCOS, as per my knowledge we will have our first official release around Fedora 30 release based on be f30 tagged built packages (correct me if I am wrong). It will be nice to discuss and define how frequently we are going to do releases in between with updated content.

[bgilbert]

As part of this, we should also discuss the stream structure.

Container Linux

On Container Linux, we have these channels:

• alpha: Updated frequently. Contains the latest versions of software, including (for Docker and the kernel) versions we don't necessarily intend to promote to other channels.
• beta: Updated less frequently. Essentially a stable release candidate. Consists of a release branch promoted from alpha, probably with changes.
• stable: Updated less frequently than beta, but more frequently than a Fedora release. Promoted from beta, often with fixes.

Not every branch promotes to stable; half of all alpha branches and half of all beta branches are not promoted. Serious bugfixes and security fixes are backported to existing release branches via out-of-cycle releases.

We encourage users to run some production nodes on alpha and some on beta in order to help catch regressions. In particular, this means that serious fixes must be applied to all channels.

The channel names are not especially descriptive and should not be carried forward without some thought; in particular, beta is not merely a promoted alpha.

Fedora CoreOS

We'll probably want to sync with the Fedora release cycle, even if it's not mandatory. If we branched from Rawhide independently from the rest of Fedora, we'd end up responsible for backporting fixes, which is too much work.

There's still value in maintaining a multiple-stream structure, however. If we don't have something like a beta channel where users can get an early preview of updates, we're more likely to push regressions to stable. Avoiding this is especially important with an automatically-updating production OS.

Straw proposal

• next: Preview of the next Fedora release when that has stabilized "enough". We'd want to define the cutover point: Bodhi enablement? the Fedora alpha release? During the rest of the cycle, next would track testing.
• testing: Periodic snapshot of the current Fedora release + updates, perhaps every two weeks.
• stable: Promotion of a snapshot that's baked in testing for two weeks, including any needed fixes.

Out-of-cycle backports of important security fixes and bugfixes would also occur. If a fix is important enough to backport directly to stable for an out-of-cycle release, it should also be backported to testing.

I've excluded updates-testing from the above, on the theory that packages retracted from updates-testing are considered not to have been deployed, so packages will not carry workarounds for any resulting breakage. I'm not sure whether that's the right idea, though.

I'm also not sure of a reasonable cadence for stable.

[dustymabe]

thanks for a well thought out reply @bgilbert. I didn't get the notification and just now stumbled upon it (see GH notifications issue I've been having).

A few comments:

• I like the idea of next.
• testing sounds like the current FAH stable (AKA fedora/28/x86_64/atomic-host). Is that right?
• should we version our refs (i.e. include major version in the name) like we have done in the past for FAH? There is a long discussion here about what a "rolling" release could have looked like for FAH.
• There will also be the rawhide ref that will exist if people want to go real leading edge.
• are you suggesting we don't advertise the updates-testing ref to the users?

[bgilbert]

testing sounds like the current FAH stable (AKA fedora/28/x86_64/atomic-host). Is that right?

AIUI yes.

should we version our refs (i.e. include major version in the name) like we have done in the past for FAH? There is a long discussion here about what a "rolling" release could have looked like for FAH.

No. Users shouldn't have to think about Fedora versions or to manually update between them.

Relatedly, though, it's worth considering how we assign version numbers to trees, as a friendly name for the user. <Fedora release>.<snapshot serial number>.<patch release>?

There will also be the rawhide ref that will exist if people want to go real leading edge.

Is there value to continuing that? In the proposal above, FCOS won't ever be tracking Rawhide or attempting to keep it working.

are you suggesting we don't advertise the updates-testing ref to the users?

That's the proposal. I don't know whether it's the right idea, and I'd love to hear opinions.

[mscherer]

So what would be the ABI/API stability promise ?

If I am using next/testing/stable, what and how often should I expect breakage for each, and effort to fix those breakages ?

[ajeddeloh]

For API stability: With CL it's always been "Stable unless otherwise noted" which is something we should continue. Bugs aside, there should never be breaking changes without an announcement and deprecation window. Users should not have to worry about updates (but should run a bit of beta/whatever we call the prerelease channel in their clusters to detect upcoming bugs).

For ABI stability: You shouldn't care. Run your stuff in containers. Running on the host directly is unsupported. We will break you and not care if you run on the host directly.

[mscherer]

Running in container do not prevent to have a ABI. I do remember a article that made the round about how containers were not as portable as we tought, due to /proc being somehow exposed and being used for detecting selinux, etc and so triggering non standard code paths with Ubuntu container on RHEL. So the conclusion was that we can't really abstract the kernel from that. And even today, that's a point that is pushed by RH Folks like http://crunchtools.com/portability-not-compatibility/

So if I am being told "there is no ABI promise", how can I be sure that stuff will not break as a user ? And if that's not a problem, can someone tell to RH why they pretend otherwise and tell me I should care ?

[bgilbert]

Fedora CoreOS is a community software project. There is no ironclad way to "be sure that stuff will not break". As @ajeddeloh said, the best approach is running some of your nodes on preproduction streams and reporting bugs you encounter.

The kernel's ABI stability rules are also not ironclad, but they generally work pretty well. If you need stronger ABI guarantees than the upstream kernel can provide, neither Container Linux nor Fedora CoreOS nor any of the other Fedora editions are likely to meet your needs.

[bgilbert]

To @ajeddeloh's point, we'll need ways to carry downstream reversions of breaking changes in Fedora. Eventually there will be something like another cgroups v2 that we'll have to work around.

[ajeddeloh]

I was more thinking ABI in terms of C/C++ ABI (don't link against our stuff and run it on the host please), but the kernel ABI is a good point. I don't think there's anything we can do there really other than say "run a little preprod if you want to catch bugs before they're a problem." We're not going to ship ancient or out of date kernels. That being said, breakage from the kernel, systemd, etc, is 99% bugs, not intenional changes.

[mscherer]

But Fedora has a process to get kernel tested before being pushed with the updates-testing system and a policy to not break ABI in stable release (as much as possible, that's not perfect, but people get annoyed when it happen).

https://fedoraproject.org/wiki/Updates_Policy

And my understanding is that such promise is not on the table, and @bgilbert explictely say in this thread that users shouldn't care about version. That's a goal I agree with, but if we rely on Fedora release as the basis and switch every 6 months, that is where things will break and where I think people would need to care about version.

So while Fedora do not promise much, as a user, I know when I can expect breakage and wait before upgrading because there is a time where I can use both. I do have 6 months to do the upgrade, which is fine for me.

My understanding of the proposal is that as a user, I would have 2 weeks to fix if I see anything broken on my side in testing before it get to stable, which is a much shorter timeframe.

And for the kernel ABI, while it is stable when the configuration do not change, we may also do configuration change in the future. So no matter how ironclad are the kernel devs, our packager might be as strict.

And API/ABI is also stuff like the Docker API, switching from Docker to podman/buildah, etc. Admins should relay on the binary API and run thing on the host for sure, but in the end, they also need to interact with the host somehow, using a API. Again, if you tell "this can be broken any time", this is not very compelling to me as a user.

[cgwalters]

Again, if you tell "this can be broken any time", this is not very compelling to me as a user.

If you look at how Container Linux has been managed, some things on the host have been removed, such as fleet with a year-long deprecation window. That's two Fedora release cycles.

I am sure that if FCOS made any such changes (e.g. dropping docker) it would make total sense to do so in concert with a Fedora major window.

As far as the rest of your comment - nothing is perfect, but the basic premise here is that value of containerization far outweighs concerns about corner case kernel ABIs affecting apps.

[bgilbert]

But Fedora has a process to get kernel tested before being pushed with the updates-testing system and a policy to not break ABI in stable release (as much as possible, that's not perfect, but people get annoyed when it happen).

Stable Fedora releases update to new kernels with new ABIs all the time. Kernel 4.17 will go EOL soon, so Fedora 28 will bump to 4.18.

My understanding of the proposal is that as a user, I would have 2 weeks to fix if I see anything broken on my side in testing before it get to stable, which is a much shorter timeframe.

You'd have two weeks to report regressions that reach testing. If you're running a few testing nodes in production, as you should, hopefully that will be enough time to catch issues. If you want additional time to test packages from the following version of Fedora, that's what next is for.

[ajeddeloh]

Does Fedora have planned breaking changes that are announced anywhere? If so we should probably work with Fedora to either announce those explicitly for FCOS (since it's supposed to be "set and forget*") or ensure that the change won't impact users.
* except when we email you

[dustymabe]

Does Fedora have planned breaking changes that are announced anywhere?

Fedora does have a "changes" process where large changes are announced and the implications of changes are considered. I'm sure not everything that changes gets reported, but it's a good way to at least socialize changes that we would like to make or to also monitor them so we are aware of changes that will affect us.

Here are links to all the fedora changes for the last 10 releases.