Bucket file layout

[jlebon]

How should our various files (metadata & build artifacts) be laid out in the bucket? Note this isn't something that we expect users to care about. Some pieces of published metadata will hold links to files into the bucket, but we should be able to change the bucket layout itself without impacting users. As such, we should also strongly discourage perusing the bucket (which without indexing it would be cumbersome anyway) or predicting locations for new builds.

Here's a strawman based on some initial thoughts in coreos/coreos-assembler#159:

/
    prod/
        streams/
            stable/
                stream.json                   # holds stream metadata
                builds/
                    builds.json               # raw cosa builds listing metadata
                    30.1234-5/
                        release.json          # holds release metadata
                        x86_64/
                            meta.json         # raw cosa build metadata
                            commitmeta.json
                            fedora-coreos-30.8-qemu.qcow2.gz
                            ostree-commit-object
                            ostree-commit.tar
                            ...
                        ppc64le/
                            ...
                        ...
            testing/
            next/
            ...

[dustymabe]

Looks mostly good. A few comments:

• i'd prefer to have filename uniqueness so maybe let's not have two meta.json filenames
• could we add architecture awareness to the example in the description?

[jlebon]

i'd prefer to have filename uniqueness so maybe let's not have two meta.json filenames

Yeah fair. Maybe just stream.json in that case?

could we add architecture awareness to the example in the description?

This is mostly dependent on coreos/coreos-assembler#463; though I guess how the various archful builds are laid out are related but somewhat independent of how they get there, so we could strawman that part now too.

[dustymabe]

so we could strawman that part now too.

Yeah - a concrete example (like the one in your description) is the best way I can think to actually hash it out. My proposal is we simply add arch directories under the buildid:

        builds/
          30.1234-5/      # holds COSA build artifacts
                x86_64/
                aarch64/
                ppc64le/ 

[dustymabe]

Yeah fair. Maybe just stream.json in that case?

forgot to respond to this: 👍

[jlebon]

OK, updated initial comment with those modifications!

[arithx]

Couple of questions I need answered for the plume work:

1. Is it intended that there is a different bucket location that these artifacts are initially written to and then copied to this structure via plume in the release process?
2. Where are we storing the release metadata index in this structure (right under prod/)?
3. Are all files/artifacts inside of the individual build level buckets things that are meant to be publicly readable (e.x.: should meta.json be fetchable by end users)?

[bgilbert]

3. Yes.

Note that if the release.json is staged somewhere by COSA and then copied over by plume, incremental updates to it (e.g. to backfill an AMI ID) will need to be copied over again.

[sinnykumari]

Few things need to understand to work on Stream Metadata Generator:

1. Is stream.json mentioned in design is where Stream Metadata Generator output will be uploaded and made available?
2. release metadata (release.json ) is available for each builds in a stream. Which one of them stream metadata generator will be using as input for building stream metadata? Is there something like latest/ under builds/ directory pointing to the latest one to be utilized?

[jlebon]

Is stream.json mentioned in design is where Stream Metadata Generator output will be uploaded and made available?

Yes, I think that's correct.

Is there something like latest/ under builds/ directory pointing to the latest one to be utilized?

Whoops, I had missed the builds.json here from COSA. I updated the first comment with that. So e.g. you can determine what the latest build is by looking at builds.json.

[sinnykumari]

Makes sense, thanks @jlebon

[bgilbert]

The current plan is not to use builds.json for that, but instead to use a separate JSON document (#98 (comment)) pointing to the release metadata of released builds.

[jlebon]

Ahh yeah, right. The builds.json is essentially a raw listing of all the cosa builds stored in that directory. Whereas the "release index" (do we have another name for this yet?) actually represents releases we want users to consume.

2. Where are we storing the release metadata index in this structure (right under prod/)?

Hmm, seems like this should be a per-stream thing, right? So maybe at /prod/streams/$stream/releases.json ? (Essentially sitting side by side with stream.json).

[bgilbert]

We don't have another name. "Release index" works for me. And yes, per-stream.