Default filesystem choices for FCOS

[ajeddeloh]

Container Llinux uses ext4 by default, Fedora Workstation uses ext4, Fedora Server uses xfs. We should pick one for / and /var (if /var is a seperate partition). Some people like btrfs, so maybe consider that too?

I personally don't have any strong opinion.

Assuming we can setup ostree between the disks and files stage, users would be able to change this via Ignition as well, so this would only affect the default image we ship.

[cgwalters]

In https://pagure.io/pungi-fedora/pull-request/257 one reason to use XFS is that it doesn't have fixed inode limits - and that's highly relevant for containers.

Another good reason to use XFS is it will soon have reflinks and that's also quite useful for overlayfs (and in general).

The other reason to use XFS is that it's the default for Red Hat Enterprise Linux 7 today.

[cgwalters]

Fedora Atomic host uses XFS for these reasons today.

[cgwalters]

(If one is comparing ext4 vs XFS, the main thing ext4 has today that XFS doesn't is encryption...that's a whole big discussion)

[dustymabe]

I don't see a good reason to go against what Fedora Server (and RHEL) are doing today with XFS. The only gripe I have with XFS is the inability to shrink a filesystem (which is annoying, but in practice you don't really need it that often).

[mskarbek]

btrfs is and will be laden with bad PR (fully deserved IMO) and hard to sell to certain groups of people. Personally, will never use it again, I value my data and don't want to go through btrfs hell again.
XFS is also a base file system for stratis which is supposed to land in Fedora 29 as 1.0 release and may be interesting in terms of future storage solutions in FCOS.

As soon as there will be something to work with I will start to experiment with combining ZFS and FCOS but that is a very limited use case, I know that. (Yes, I'm throwing that here as a statement, there are people using ZFS on Linux in production ;))

[dustymabe]

added meeting label so we can grab some consensus on this one.

[jasonbrooks]

I'm +1 to xfs

[mscherer]

So, what would be the plan for upgrading the on disk format if needed ?

I do remember that we did face a bug with RHOSP 12 or 13 with the introduction of container on XFS with a FS that was created with RHEL 7.0 and later upgraded to 7.5 (using every intermediate step). Openstack folks gave me those 2 links: https://bugzilla.redhat.com/show_bug.cgi?id=1564671 and https://bugzilla.redhat.com/show_bug.cgi?id=1309498 for people who want more details, I can give links to the internal discussion if needed.

Since the issue did only appear on a upgraded setup who was installed on 7.0, since the goal is to have continuous upgrade, is this something that need to be taken in account in the XFS vs ext4 (especially since Colin point out there is new features coming that we may need) ?

[dustymabe]

is this something that need to be taken in account in the XFS vs ext4

is the question the stability of XFS vs ext4 ?

[mscherer]

No, more on the question of on disk upgrade if needed. I do guess all suffer from the same issue anyway, but I kinda feel this is something to look for a continuous upgrade system, and something that would influence the choice.

Like would we need to change something, and if so "how", and can it be done safely in a cloud world (aka, with server that can reboot at the wrong moment, etc, etc)

[cgwalters]

No, more on the question of on disk upgrade if needed.

https://marc.info/?l=linux-ext4&m=149520766520774&w=2

is a bit related here (of course Ignition does do filesystem stuff from the initramfs). But this starts to get quite filesystem specific as far as what transitions can be made.

In the big picture, no upgrade system is really going to get you out of a need to reprovision at some point - at least for "long-lived" servers/desktops etc.

For Red Hat CoreOS, we are building technology that will do in-place updates where we can, or ensure nodes are reprovisioned as necessary. Particularly in virtualized infrastructure, reprovisioning isn't that expensive. For bare metal, one can amortize it.

The RHEL XFS thing was quite unfortunate. On the other hand, if you used LVM and didn't allocate all the disk space, you can easily make a new filesystem.

There's no magical solutions to this - there's really no getting around trying to avoid having long-lived pet filesystems. You want backups anyways, and if you have those you can reprovision and restore from backup.

[ajeddeloh]

Historically this hasn't been an issue with CL, at least in my time working on it (which is ext4 by default, not sure if XFS will cause more concern). We should not try to update the FS itself automatically (not sure how possible that even is), because it's just far too risky. Number one rule of automatic updates is "don't do things that you think might break other people's systems" and number zero rule is "Don't delete their data". If you want an updated filesystem, reprovision.

This does mean that we need to make sure that the kernel/userspace tools continue to work with older version filesystems.

[bgilbert]

Historically this hasn't been an issue with CL, at least in my time working on it (which is ext4 by default, not sure if XFS will cause more concern).

Also, we originally defaulted to btrfs for the root FS and there are still systems out there that use it.