ostree delivery model

[cgwalters]

For Red Hat CoreOS, we are experimenting with doing ostree-inside-container as a delivery model. For people running a Kubernetes/OpenShift cluster, dealing with container images is...a known quantity. A critical aspect is that there a metric ton of tools that know how to mirror container images (and "offline"/"behind the firewall" usage is absolutely critical to support).

(Aside - one might ask: why ostree-in-container? Why can't the container filesystem itself just be the OS filesystem? Well...one issue there is SELinux labeling; container files don't carry labels today. We might be able to create some sort of new, special extensions to the OCI format but for now it's way easier to just embed an ostree repo inside a container image)

I propose that Fedora CoreOS support this - i.e. we produce an "oscontainer" image for each release. We should document how to pull it down and deploy it inside a cluster, and have the cluster use it for updates as a local mirror.

However, that still leaves the question of the default. One downside of OCI today is that there's no deltas. This is generally irrelevant for clusters (particularly deployed in the public cloud), but...for small scale usage it does matter. Now, "pure ostree" mode does have deltas. However, if one wants to use any package layering...this is where rojig comes into play. (On a semi-related topic, I will be pushing for Silverblue to use rojig by default)

One advantage of offering rojig as well for FCOS is that it'd be trivial to spin up a container and get the same RPM packages that went into the host OS - it's just a yum repo and you can yum distro-sync to it or whatever.

So in the end that's my proposal: We offer both rojig and oscontainer for FCOS. The default is rojig, but we make it falling-off-a-log-easy to switch to the oscontainer.

But that's just a proposal. Any other opinions?

[bgilbert]

I think it'd be better to start with one good distribution model, rather than two different ones which each have advantages under some circumstances. This is the best opportunity we'll get to be legacy-free.

(For what it's worth, I'm not especially excited about either reassembling-an-ostree-from-RPMs or extracting-an-ostree-from-a-container. AIUI, they both seem focused primarily on reuse of unrelated technologies in order to gain access to existing tooling. That may end up being necessary under the circumstances, but it seems regrettable.)

[dustymabe]

My personal opinion is that I prefer the pure OSTree repository model (as it maps very easily to a git repo) and it's rather easy to manipulate branches and perform releases that way.

However, if one wants to use any package layering...this is where rojig comes into play.

I think we're going to try to generally discourage the use of package layering except for maybe a few select use cases, so maybe the benefit isn't as great?

[cgwalters]

My personal opinion is that I prefer the pure OSTree repository model

OK! That's fair enough. And then are you thinking just ostree? At least rojig today supports a rpm-ostree ex commit2rojig operation which is designed to support a world where the ostree repository is canonical, and make it easy to maintain the rojig format. Doing something similar with our oscontainer process probably wouldn't be hard.

[dustymabe]

OK! That's fair enough. And then are you thinking just ostree?

I'm really not trying to dictate anything, just stating my personal opinion. I really think the technology behind rojig is amazing. You and jonathan have done an incredible job implementing a "jigdo" like system for rpms where the original commit can be bit-for-bit reconstructed from the original rpm artifacts. However, I do think that having an ostree server repository as the source of truth and "augmenting" the delivery mechanism with rojig when necessary (i.e. when users much prefer moving around content in rpms vs just having files on a webserver somewhere) is the way to go. Again, this is just my personal preference.

At least rojig today supports a rpm-ostree ex commit2rojig operation which is designed to support a world where the ostree repository is canonical, and make it easy to maintain the rojig format. Doing something similar with our oscontainer process probably wouldn't be hard.

Yep, a cojig for container-ostree jigdo would probably be useful. I'd really love having some smart delta mechanism for container registries as well. I'm sure someone will write one soon enough.

[ajeddeloh]

I agree with @bgilbert and @dustymabe. For one I think we should emphasize the ostree bits and not the rpm bits. From a user perspective rpm's should be invisible; they shouldn't have to think about what their OS is built from. Granted if we need to do package layering to support things like multiple dockers/cri-o's then that breaks down a bit, but wherever possible we should make that invisible. For the user FCOS shouldn't "feel like" fedora in the same way that CL doesn't "feel like" Gentoo. Ideally the fact that the OS is fedora based shouldn't matter to the user, it's just where we get our packages.

Getting back to the point, the pure ostree model is very not-fedora-specific and not-redhat-specific. Other distribution models are (well maybe not oci, but I don't see a strong reason to use it over the pure ostree model).

[dustymabe]

Added the meeting label to this ticket so we can discuss it today during the community meeting. Regardless of the path we choose there is going to be work to do so the sooner we decide the sooner we can start knocking out that work.

[dustymabe]

ok had a lot of discussion about this in the community meeting today (see the log starting at 16:41:49).

Generally we agreed upon:

• The general sentiment is that we should target plain OSTree repo + enhanced mirroring support (not implemented today) and then augment with rojig in the future if needed. We will try to get buy in from IoT/Silverblue teams so that we can have a common strategy for all of Fedora delivered OSTree content.

I'd like to invite Silverblue (@owtaylor + @matthiasclasen), Fedora IoT (@nullr0ute), and Fedora Infra (@puiterwijk + @nirik) to weigh in. Ideally we would agree on a strategy so that we can unify the way we deliver OSTree content to end users.

[cgwalters]

For Silverblue, I have a fairly strong opinion that we should use rojig by default. The reason is it simply works fundamentally better when you have layered packages, and particularly with FSB29 since we dropped e.g. libvirt and docker by default, we'll have things people want to layer.

Layering for FCOS and IoT is perhaps less of a use case - and particularly for IoT I suspect most real users will want to build their own - so our choice of delivery format is more "what choices do we offer them".

[dustymabe]

For Silverblue, I have a fairly strong opinion that we should use rojig by default. The reason is it simply works fundamentally better when you have layered packages, and particularly with FSB29 since we dropped e.g. libvirt and docker by default, we'll have things people want to layer.

warning dumb questions inbound

I think there are two arguments for rojig being better assuming someone is package layering:

• you now download updates only from yum repos rather than both an ostree repo and a yum repo
• you don't get package layering conflicts (like split base/layered versionlocked package problem (krb5-workstation) rpm-ostree#415 and Clearer error message in caseof layered package conflict rpm-ostree#1325)

I agree for the first point (one repo is nicer), but don't think I agree for the second point (at least I haven't in the past). Today i'd say we shouldn't get package layer conflicts on silverblue if the operation you are doing is rpm-ostree upgrade --install foorpm rather than just rpm-ostree install foorpm, do you agree? I'm not sure how rojig solves that problem without also forcing an upgrade. Does it look at what version of the yum repo the rojig RPM (and corresponding ostree commit) came from and only use that version of the repo to pull packages from? That would require versioned repos (which would be nice to have but I don't think we have today do we?), but if we had that I'd say we could also probably bake that information into an ostree commit and use it for non-rojig mode too (i.e. this ostree was built using yum repo xyz let's try to package layer from xyz repo too).

EDIT: follow up discussion to this point happening here

[cgwalters]

(This is totally a tangent but...I find it fascinating. Building off the IoT thing, in the United States at least, car manufacturers are prohibited from offering OTA updates because of dealerships. So even if a manufacturer wanted to add containers on top of FCOS and somehow use it as is, they couldn't actually have their cars update from our servers. My car updates from a USB stick. Big picture I am certain even if they could do it they'd want to do their own QA/patches so - back to custom builds)

[matthiasclasen]

I prefer a pure ostree model as well. rojig seems just ... iffy, conceptually. And package layering is prone to give you the worst of both worlds, in my experience. I'd rather see us strongly encourage do-everything-in-a-container, for Silverblue.

[owtaylor]

The reason is it simply works fundamentally better when you have layered packages
I'd be interested in learning more about this? Why is this the case?

rojig is most interesting to me as an optimization of distribution - where we have a QA'ed ostree image, and the exact same ostree is recreated on the users system. In this role, it has the advantage that the content is already on fedora mirrors. The thing I worry about with rojig is how robust its going to be - we're downloading different things from different repositories, and assembling them on the users system. is it always going to work? what happens when an RPM that is needed is missing?

Going back to the beginning of this thread:

Why can't the container filesystem itself just be the OS filesystem? Well...one issue there is SELinux labeling; container files don't carry labels today. We might be able to create some sort of new, special extensions to the OCI format but for now it's way easier to just embed an ostree repo inside a container image

Hmm - so we're talking about two inconsistent representations of an ostree as an OCI image:

• This ostree-inside-container model
• The Flatpak model where the ostree filesystem is the OCI image filesystem, and commit metadata is carried as OCI annotations.

OCI image layer specification specifically says that xattrs can be included (in fact it says they MUST be included). Isn't that sufficient for the representation of selinux labels?

[cgwalters]

OCI image layer specification specifically says that xattrs can be included (in fact it says they MUST be included). Isn't that sufficient for the representation of selinux labels?

Perhaps - but when we're pulling application containers we shouldn't respect any security.selinux labels that exist there, as that could violate confinement. I would think containers/storage would be filtering this out...but if it is doing so I don't see the code that would be doing it...