Stream and release metadata

[bgilbert]

There are two types of metadata relevant to FCOS releases. We should publish each type in a unified machine-readable format.

Proposal

Release metadata

Release metadata is associated with a particular OS version. It should include:

• The stream name, version number, and release notes
• The Pungi compose ID from which packages were sourced
• Image identifiers for each tuple of (platform, CPU architecture, region). For example, on AWS there is an AMI ID per region. On GCP there is one image name across the entire platform. On all platforms there is an associated image artifact; we could also include its content hash.

Release metadata is populated incrementally, but each field should be immutable once added. For example, we might need to publish an urgent security release while an AWS region is down. In this case, we'd provide the available AMI IDs, and backfill the metadata once we got an AMI published into the lagging region. Platforms with slow publishing processes (such as AWS Marketplace) might always need their image IDs backfilled after the release.

There's been some discussion of storing release metadata as ostree detached metadata (coreos/coreos-assembler#189 (comment)) or in a Git repo (coreos/coreos-assembler#203). Perhaps release metadata should be signed.

Stream metadata

Stream metadata is associated with a stream. It should give the currently-recommended OS version, image ID, and artifact URL for the stream, separately for each platform (and region, where appropriate). Most of the time, stream metadata will just be a refactored version of the release metadata for the current release on the stream. However, operational issues may require divergence from the current release, such as:

• Holding certain platforms or regions at a previous release
• Reverting certain platforms or regions to a previous release after updating them to the current release

Stream metadata might also include the target OS version for updates, to be consumed by the Cincinnati graph builder (#83). Operational issues may sometimes require this to be different from the target version for new installs.

Tooling should perform on-demand sync of stream metadata from current release metadata, subject to overrides specified via explicit configuration. Those overrides should allow replacing certain stream metadata elements with corresponding elements from a specified older release, or from manually specified metadata.

Stream metadata for each production stream should be available at a well-known HTTP endpoint, perhaps in the form of a static JSON document generated by a site builder. Perhaps it should be signed. Metadata history should be recorded, perhaps in a Git repo.

cc @cgwalters

[bgilbert]

If we want to ship ostree static deltas, release metadata should also record the expected predecessor(s) of a release. This will usually be the previous release on the stream, not the previous release on the snapshot branch. (For example, the predecessor of 30.5.2 might be 30.4.3.) Note that "release" really means release; abandoned unreleased versions (which can occur in CL; not sure if they will with FCOS) shouldn't count as predecessors.

In some cases there will be multiple predecessors, notably when rollout of the previous release was halted due to a regression.

cc @sinnykumari

[cgwalters]

coreos-assembler's builds.json + meta.json is one model. It's OK. The git model could also be nice although it requires more work for endpoints (it's not just a REST API to consume), though obviously one could write one on top. Modeling the releases in ostree itself has some nice properties although it's also not REST.

I think coreos-assembler should probably define a schema for "streams" that point to builds.json; obviously one can model it as just separate dirs but having it be discoverable (like ostree summary files list multiple refs) would be nice.

I guess all of this gets to Dusty's push to model as much data as possible in ostree. That's certainly possible - but it would mean e.g. ostree commit metadata would have e.g. AMI ids. Which...is probably fine, in the end the data isn't going to be more than 10k I'd guess which is way way smaller than the statically linked binaries in the OS.

[ashcrow]

/cc @dustymabe @sinnykumari

[brianredbeard]

Let me just add two concrete use cases we had with CoreOS/Container Linux (CL) which may be useful as a litmus test:

With CL it was challenging to discover all of the related files to an OEM to know which files one would need to sync. When reflecting on the distribution repos there have been times when file name changes were necessitated or when new platforms had files added.

Using OpenStack and PXE platforms as an example, in the case of OpenStack all that is needed is to pull down the virtual machine image while with PXE one has to pull down the Kernel and the Initramfs.

While one may pull down a specific version of an OpenStack image it will quickly get out of date meaning that immediately after install a host has to go down for an update needlessly wasting time during provisioning.

Our solution was to present two scripts. A generic one which could check the ETag on version.txt so that it could easily be run via cron and only react when version.txt changed. The second was to then trigger the execution of an OEM specific script (glance_load.sh in this case) which will then perform the specific logic of loading in a new version to glance*.

In the case of PXE we then had two distinct paths of execution: Either using a PXE load to then bootstrap the host or using it as a live running system. In the latter case this allowed organizations to use Container Linux as a bootstrapping platform for their own clouds. Multiple public cloud providers used Container Linux in combination with a containerized agent to perform dynamic bootstrapping of hosts (at least one of them used this with OpenStack Ironic).

This presented an interesting problem though, if a user was not persisting data to disk then there is no mechanism for Omaha (or Cincinnati) based updates. This meant that the synchronization of OS components MUST occur at the infrastructure level. For this users would then be required to synchronize down the components of an OEM. As (throughout the life of Container Linux) metadata was not generally presented on a per-OEM basis, this meant users had to report to string matching of content to identify the content they needed to mirror. One example is this script - coreos_mirror.py.

• We had differing opinions about the "right" way to go about this, so I personally had a copy of my script with additional functionality around tagging of images so that one could boot the latest image in a "channel" using the os_family, os_distro, os_release, or os_version metadata.

[bgilbert]

Here are some sample metadata blobs for discussion. I've written them in YAML for clarity, but the published documents would be JSON.

Stream metadata

# Include stream name so the document is self-contained
stream: stable
metadata:
  last-modified: "2019-06-04T16:18:34Z"
architectures:
  x86_64:
    artifacts:
      # Some of these will be useful for many users, such as qemu or
      # openstack. Some will likely only be useful for cloud operators,
      # such as digitalocean or packet.  Some, such as aws, are useful
      # for users in special situations.
      aws:
        release: 30.1.2.3
        formats:
          # Generally one format per platform, but allow for future expansion
          # without obscuring the platform ID (as on Container Linux)
          "vmdk.xz":
            # Generally only one artifact, but not always
            disk:
              location: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz
              signature: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      azure:
        release: 30.1.2.3
        formats:
          "vdi.xz":
            disk:
              location: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz
              signature: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      digitalocean:
        release: 30.1.2.3
        formats:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz
              signature: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      gcp:
        release: 30.1.2.3
        formats:
          "tar.gz":
            disk:
              location: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz
              signature: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      metal:
        release: 30.1.2.3
        formats:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz
              signature: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          iso:
            disk:
              location: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso
              signature: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          pxe:
            kernel:
              location: https://artifacts.example.com/hkIj8FkCydT3lV9h
              signature: https://artifacts.example.com/hkIj8FkCydT3lV9h.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
            initramfs:
              location: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz
              signature: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          "installer.iso":
            disk:
              location: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso
              signature: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          installer-pxe:
            kernel:
              location: https://artifacts.example.com/EtqI0KsLIwZOHlCx
              signature: https://artifacts.example.com/EtqI0KsLIwZOHlCx.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
            initramfs:
              location: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz
              signature: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      openstack:
        release: 30.1.2.3
        formats:
          "qcow.xz":
            disk:
              location: https://artifacts.example.com/oKooheogobofai8l.qcow.xz
              signature: https://artifacts.example.com/oKooheogobofai8l.qcow.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      packet:
        release: 30.1.2.3
        formats:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz
              signature: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      qemu:
        release: 30.1.2.3
        formats:
          "qcow.xz":
            disk:
              location: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz
              signature: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      virtualbox:
        release: 30.1.2.3
        formats:
          ova:
            disk:
              location: https://artifacts.example.com/yohsh2haiquaeYah.ova
              signature: https://artifacts.example.com/yohsh2haiquaeYah.ova.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      vmware:
        release: 30.1.2.3
        formats:
          ova:
            disk:
              location: https://artifacts.example.com/quohgh8ei0uzaD5a.ova
              signature: https://artifacts.example.com/quohgh8ei0uzaD5a.ova.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

    images:
      # Cloud images to be launched directly by users.  These are in a
      # separate section because they might not always in sync with the
      # release artifacts above.
      aws:
        regions:
          us-east-1:
            # We know the release because we uploaded it, so might as well
            # list it.
            release: 30.1.2.3
            image: ami-0123456789abcdef
          us-east-2:
            release: 30.1.2.3
            image: ami-0123456789abcdef
      azure:
        # We could give a specific image URN here, but we probably want
        # users to always use a Marketplace URN.  So this is a static
        # string, and represents advice rather than a value we might
        # change.
        image: Fedora:CoreOS:stable:latest
      gcp:
        # We could give a specific image name here, but we probably want
        # users to always use an image family.  So this is a static string,
        # and represents advice rather than a value we might change.
        image: projects/fedora-cloud/global/images/family/fedora-coreos-stable
      digitalocean:
        # We don't control platform ingest, so an image slug is probably
        # the best we can do.
        image: fedora-coreos-stable
      packet:
        # Images don't have addressable versions, so an operating system
        # slug is the best we can do.
        image: fedora_coreos_stable

    updates:
      # Primarily meant as input to Cincinnati
      release: 30.1.2.3

We could also include artifact size/uncompressed-size/uncompressed-sha256 from meta.json, if desired.

Override file

The simplest possible implementation of the override file would just be partial stream metadata, overriding corresponding parts of the automatically generated one. A fancier one would allow references to older releases, but that functionality could be added later if needed.

Release metadata

This seems largely redundant with meta.json, so in principle we might skip it. But it could also be a good opportunity to clean up that format a bit.

release: 30.1.2.3
stream: stable
metadata:
  last-modified: "2019-06-04T16:18:34Z"
architectures:
  x86_64:
    commit: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    media:
      aws:
        artifacts:
          "vmdk.xz":
            disk:
              location: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz
              signature: https://artifacts.example.com/dsB2fnzP7KhqzQ5a.vmdk.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        images:
          us-east-1:
            image: ami-0123456789abcdef
          us-east-2:
            image: ami-0123456789abcdef
      azure:
        artifacts:
          "vdi.xz":
            disk:
              location: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz
              signature: https://artifacts.example.com/aeng0xah6vaaVosh.vdi.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        images:
          global:
            image: Fedora:CoreOS:Stable:30.1.2.3
      digitalocean:
        artifacts:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz
              signature: https://artifacts.example.com/ichaloomuHax9ahR.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      gcp:
        artifacts:
          "tar.gz":
            disk:
              location: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz
              signature: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        image: projects/fedora-cloud/global/images/fedora-coreos-stable-30-1-2-3
      metal:
        artifacts:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz
              signature: https://artifacts.example.com/xTqYJZKCPNvoNs6B.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          iso:
            disk:
              location: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso
              signature: https://artifacts.example.com/ADE5GO3bjAXeDcLO.iso.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          pxe:
            kernel:
              location: https://artifacts.example.com/hkIj8FkCydT3lV9h
              signature: https://artifacts.example.com/hkIj8FkCydT3lV9h.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
            initramfs:
              location: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz
              signature: https://artifacts.example.com/a9ytS8yB4cGZpca1.cpio.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          "installer.iso":
            disk:
              location: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso
              signature: https://artifacts.example.com/KwKye6YW4SIIPrhY.iso.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
          installer-pxe:
            kernel:
              location: https://artifacts.example.com/EtqI0KsLIwZOHlCx
              signature: https://artifacts.example.com/EtqI0KsLIwZOHlCx.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
            initramfs:
              location: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz
              signature: https://artifacts.example.com/EhoS1x66RVA2k8y6.cpio.gz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      openstack:
        artifacts:
          "qcow.xz":
            disk:
              location: https://artifacts.example.com/oKooheogobofai8l.qcow.xz
              signature: https://artifacts.example.com/oKooheogobofai8l.qcow.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      packet:
        artifacts:
          "raw.xz":
            disk:
              location: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz
              signature: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      qemu:
        artifacts:
          "qcow.xz":
            disk:
              location: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz
              signature: https://artifacts.example.com/Siejeeb6ohpu8Eel.qcow.xz.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      virtualbox:
        artifacts:
          ova:
            disk:
              location: https://artifacts.example.com/yohsh2haiquaeYah.ova
              signature: https://artifacts.example.com/yohsh2haiquaeYah.ova.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      vmware:
        artifacts:
          ova:
            disk:
              location: https://artifacts.example.com/quohgh8ei0uzaD5a.ova
              signature: https://artifacts.example.com/quohgh8ei0uzaD5a.ova.sig
              sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

[bgilbert]

Some discussion in #187.

[arithx]

In an OOB discussion we were discussing having an index for the release metadata that would be consumed by tooling to determine the release metadata endpoint for specific OSTree commits / versions. This index will be generated by plume as part of the release process and I'm currently planning on using the following JSON layout:

{
   "releases": [
        {
            "commit": "<hash>",
            "version": "<version>",
            "metadata": "<url endpoint to build release metadata>"
        },
        ...
    ],
    "metadata": {
        "last-modified": "<timestamp>"
    }
}