diff --git a/.github/ISSUE_TEMPLATE/implementing-new-emerging-platform.md b/.github/ISSUE_TEMPLATE/implementing-new-emerging-platform.md new file mode 100644 index 0000000..de334f4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/implementing-new-emerging-platform.md @@ -0,0 +1,41 @@ +# Implementing a new emerging platform + +This template is a simplified version of the +[full template](https://github.com/coreos/fedora-coreos-tracker/blob/main/.github/ISSUE_TEMPLATE/implementing-new-platform.md) +that only includes what is strictly needed to get initial support for a new +platform in Fedora CoreOS. This simplified version notably does not include the +steps needed to add new boot images to the release process. + +Platforms added via this process are labelled "emerging" and users will have to +get boot images for them by converting existing images in the right format and +changing the `ignition.platform.id=` command line parameter. + +This process will be documented using `guestfish` as an example. + +## During Development + +Create PRs addressing the following: + +- [ ] [Ignition](https://github.com/coreos/ignition/) ([example PR](https://github.com/coreos/ignition/pull/918)) + - [ ] Add userdata fetch + - [ ] If the platform supports it (unlikely), add userdata deletion +- [ ] [Afterburn](https://github.com/coreos/afterburn/) ([example PR](https://github.com/coreos/afterburn/pull/451)) + - [ ] (Cloud Only) Add relevant attributes + - [ ] (Cloud Only) Add SSH key support if available + - [ ] (Cloud Only) Add hostname support if available + - [ ] (Cloud Only) Add check-in if needed (unlikely) +- [ ] [fedora-coreos-docs](https://github.com/coreos/fedora-coreos-docs) ([example PR](https://github.com/coreos/fedora-coreos-docs/pull/377)) + - [ ] Add a `provisioning-.adoc` that walks through how to setup the new platform + - [ ] Add an entry in the `modules/ROOT/nav.adoc` that points to new documentation +- [ ] (Optional but recommended) Add support for the platform to [kola](https://github.com/coreos/coreos-assembler) to simplify testing +- Create or ask for a new upstream releases for: + - [ ] Ignition + - [ ] Afterburn +- Wait for new images with updated Ignition and Afterburn to reach stable then + merge documentation with `guestfish` commands: + - [ ] fedora-coreos-docs + +## At Release + +There are no "At Release" steps as we do not produce new boot images for +emerging platforms/ diff --git a/.github/ISSUE_TEMPLATE/implementing-new-platform.md b/.github/ISSUE_TEMPLATE/implementing-new-platform.md index ea51af7..4ede3f2 100644 --- a/.github/ISSUE_TEMPLATE/implementing-new-platform.md +++ b/.github/ISSUE_TEMPLATE/implementing-new-platform.md @@ -1,88 +1,63 @@ # Implementing a new supported platform - ## During Development - Create PR's addressing the following: - - - [ ] [stream-metadata-go](https://github.com/coreos/stream-metadata-go) - - [Example PR](https://github.com/coreos/stream-metadata-go/pull/45/) - - [ ] Add platform to the `Media` struct in `release/release.go` - - [ ] Add supporting code for new platform to `toStreamArch` func in `release/translate.go` - - [ ] (Cloud Only) Cloud Images need to have an `Images` field - - - [ ] (Cloud Only) [stream-metadata-rust](https://github.com/coreos/stream-metadata-rust/) - - [Example PR](https://github.com/coreos/stream-metadata-rust/pull/16) - - - [ ] [fedora-coreos-tracker](https://github.com/coreos/fedora-coreos-tracker/) - - [Example PR](https://github.com/coreos/fedora-coreos-tracker/pull/1213/) - - [ ] Update the metadata for the new platform - - - [ ] [coreos-assembler](https://github.com/coreos/coreos-assembler) - - [Example PR](https://github.com/coreos/coreos-assembler/pull/2489) - - [ ] Implement required functionality to support new platform - - - [ ] [fedora-web](https://pagure.io/fedora-web/websites) - - [Example PR](https://pagure.io/fedora-web/websites/pull-request/221#request_diff) - - [ ] Add platform to `sites/static/js/coreos-download.js` - -- [ ] [fedora-coreos-browser](https://github.com/coreos/fedora-coreos-browser) - - [Example PR](https://github.com/coreos/fedora-coreos-browser/pull/35) - - [ ] Add a list element for new platform in `browser/index.html` - -- [ ] [build pipeline](https://github.com/coreos/fedora-coreos-pipeline) - - [Example PR](https://github.com/coreos/fedora-coreos-pipeline/pull/500) - - [ ] Add platform to the list found in `jobs/build.Jenkinsfile` for building the new artifact - - - [ ] [fedora-coreos-docs](https://github.com/coreos/fedora-coreos-docs) - - [Example PR](https://github.com/coreos/fedora-coreos-docs/pull/377) - - [ ] Add a `provisioning-.adoc` that walks through how to setup the new platform - - [ ] Add an entry in the `modules/ROOT/nav.adoc` that points to new documentation - - - - ## At Release - -1. Merge metadata changes: - - - [ ] stream-metadata-go - - [ ] stream-metadata-rust - - [ ] fedora-coreos-tracker - - [ ] fedora website - - [ ] fedora-coreos-browser - - -2. Create and push signed tags with appropriate versions - - ``` - # Ensure gpg key for signing in github settings that is associated to redhat email. - # Verify you are on the upstream repo's main branch. - - git status - - RELEASE_VER=vx.y.z - # Replace 'x.y.z' with the appropriate numbers. - - git tag -s ${RELEASE_VER} - # Give appropriate detail to tag, check previous tags with 'git show ${RELEASE_VER}' - - git push git@github.com:coreos/targeted-repo.git ${RELEASE_VER} - # Navigate to the targeted-repo's tag section to ensure a valid signed tag is listed. - # e.g. https://github.com/...repo/tags - ``` - - 1. [ ] Tag stream-metadata-go following the above steps. After tagging, ensure that dependabot has picked up latest version, and merged it into fedora-coreos-stream-generator && coreos-assembler. - - These can be triggered manually by navigating to [fedora-coreos-stream-generator's Dependabot](https://github.com/coreos/fedora-coreos-stream-generator/network/updates/) and [coreos-assembler's Dependabot](https://github.com/coreos/coreos-assembler/network/updates) respectively; then, clicking "Check for updates". - - This might need to be done a few times, as the Dependabot might not pickup tag changes for a few attempts after initial tagging. - - 2. [ ] Tag fedora-coreos-stream-generator following the above steps. - -3. Merge the following changes: - - [ ] coreos-assembler - -4. Wait for updates made to coreos-assembler to be propagated to latest container - - [ ] Download latest version of coreos-assembler container. Verify platform support functionality. - -5. Merge changes for: - - [ ] Build pipeline - -6. Wait for new images to reach stable then merge documentation. - - [ ] fedora-coreos-docs merged \ No newline at end of file +## During Development + +Create PRs addressing the following: + +- [ ] [Ignition](https://github.com/coreos/ignition/) ([example PR](https://github.com/coreos/ignition/pull/918)) + - [ ] Add userdata fetch + - [ ] If the platform supports it (unlikely), add userdata deletion +- [ ] [Afterburn](https://github.com/coreos/afterburn/) ([example PR](https://github.com/coreos/afterburn/pull/451)) + - [ ] (Cloud Only) Add relevant attributes + - [ ] (Cloud Only) Add SSH key support if available + - [ ] (Cloud Only) Add hostname support if available + - [ ] (Cloud Only) Add check-in if needed (unlikely) +- [ ] [stream-metadata-go](https://github.com/coreos/stream-metadata-go) ([example PR](https://github.com/coreos/stream-metadata-go/pull/45/)) + - [ ] Add platform to the `Media` struct in `release/release.go` + - [ ] Add supporting code for new platform to `toStreamArch` func in `release/translate.go` + - [ ] (Cloud Only) Cloud Images need to have an `Images` field +- [ ] (Cloud Only) [stream-metadata-rust](https://github.com/coreos/stream-metadata-rust/) ([example PR](https://github.com/coreos/stream-metadata-rust/pull/16)) +- [ ] [fedora-coreos-tracker](https://github.com/coreos/fedora-coreos-tracker/) ([example PR](https://github.com/coreos/fedora-coreos-tracker/pull/1213)) + - [ ] Update the metadata for the new platform +- [ ] [coreos-assembler](https://github.com/coreos/coreos-assembler) ([example PR](https://github.com/coreos/coreos-assembler/pull/2489)) + - [ ] Updated `cmd-generate-release-meta` + - [ ] `cosa osbuild ` works +- [ ] [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config/) + - [ ] Add a stanza to `platforms.yaml` if the system should use a serial console, or both serial and graphical consoles +- [ ] [fedora-websites-3.0](https://gitlab.com/fedora/websites-apps/fedora-websites/fedora-websites-3.0/) + - [ ] Add friendly name for platform to `components/utilities/FpDownloadItem.vue` + - [ ] Add artifact to `pages/coreos/download.vue` + - [ ] Possibly add logo to `content/editions/coreos/home.yml` +- [ ] [fedora-coreos-browser](https://github.com/coreos/fedora-coreos-browser) ([example PR](https://github.com/coreos/fedora-coreos-browser/pull/35)) + - [ ] Add a list element for new platform in `browser/index.html` +- [ ] [build pipeline](https://github.com/coreos/fedora-coreos-pipeline) ([example PR](https://github.com/coreos/fedora-coreos-pipeline/pull/815)) + - [ ] Add platform to the list found in `config.yaml` for building the new artifact +- [ ] [fedora-coreos-docs](https://github.com/coreos/fedora-coreos-docs) ([example PR](https://github.com/coreos/fedora-coreos-docs/pull/377)) + - [ ] Add a `provisioning-.adoc` that walks through how to setup the new platform + - [ ] Add an entry in the `modules/ROOT/nav.adoc` that points to new documentation + +## At Release + +1. Merge upstream changes and put out a release: + - [ ] Ignition + - [ ] Afterburn +1. Merge metadata changes: + - [ ] stream-metadata-go + - [ ] stream-metadata-rust + - [ ] fedora-coreos-tracker + - [ ] fedora website + - [ ] fedora-coreos-browser +1. Release updated components + - [ ] Create and follow release checklist for [stream-metadata-go](https://github.com/coreos/stream-metadata-go/blob/main/docs/development.md#release-process) + - [ ] Ensure that Dependabot has PRed stream-metadata-go into fedora-coreos-stream-generator and coreos-assembler. Merge the update PRs. + - This can be triggered manually by navigating to [fedora-coreos-stream-generator's Dependabot](https://github.com/coreos/fedora-coreos-stream-generator/network/updates/) and [coreos-assembler's Dependabot](https://github.com/coreos/coreos-assembler/network/updates) respectively, then clicking "Check for updates". + - This might need to be done a few times, as Dependabot might not pick up tag changes for a few attempts after initial tagging. + - [ ] Create and follow release checklist for [fedora-coreos-stream-generator](https://github.com/coreos/fedora-coreos-stream-generator/blob/main/docs/development.md#release-process) +1. Merge the following changes: + - [ ] coreos-assembler +1. Wait for updates made to coreos-assembler to be propagated to latest container + - [ ] Download latest version of coreos-assembler container. Verify platform support functionality. +1. Merge changes for: + - [ ] Build pipeline +1. Wait for new images to reach stable then merge documentation. + - [ ] fedora-coreos-docs merged diff --git a/.github/ISSUE_TEMPLATE/new-feature.md b/.github/ISSUE_TEMPLATE/new-feature.md new file mode 100644 index 0000000..808e7b6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/new-feature.md @@ -0,0 +1,59 @@ +--- +name: Implement a feature +about: Propose a design for a new feature +--- + +# Feature proposal + +## Description + + + +## Implementation PRs + + + +## Did you consider? + + + +- Storage + - [ ] Disk space usage + - [ ] Behavior on 4Kn disks + - [ ] Compatibility with multiple ESPs (Butane `boot_device.mirror`) +- First boot + - [ ] Behavior on first boot vs. second boot + - [ ] initrd networking requirements + - [ ] Reprovisioned systems that reused existing storage devices +- OS update + - [ ] Behavior after an OS rollback + - [ ] Compatibility with old bootloaders +- Architectures + - aarch64 + - [ ] Compatibility with non-UEFI boot + - ppc64le + - [ ] Whether new GRUB directives are supported by petitboot + - s390x + - [ ] Endianness issues + - [ ] Need to rerun `zipl` to update kernel or kargs + - [ ] ECKD/MBR lack of partition labels + - [ ] ECKD maximum partition count +- Implementation + - [ ] How interlocking PRs will be ratcheted into repos + +## Implementation steps + +- [ ] Create tracker ticket with initial design (above) +- [ ] Initial discussion and refinement in the ticket +- [ ] Add `meeting` label +- [ ] Discuss at community meeting +- [ ] Further refinement + - [ ] Post draft [fedora-coreos-docs](https://github.com/coreos/fedora-coreos-docs/) PR, ideally before doing any implementation, to help identify design problems. +- [ ] Update issue description with final proposal and post a comment saying that you did +- [ ] Verify that rough consensus exists +- [ ] Implement. Post PR links in the section above. In the description of each PR, link to this issue and specify the prerequisites for merging. + - [ ] Add kola test(s) for new feature +- [ ] Land implementation PRs, in order +- [ ] Wait for the functionality to reach FCOS stable +- [ ] Land docs PR +- [ ] Remove any ratcheting glue (e.g. workarounds in coreos-assembler) diff --git a/.github/ISSUE_TEMPLATE/rebase.md b/.github/ISSUE_TEMPLATE/rebase.md index e8615ad..ff2700e 100644 --- a/.github/ISSUE_TEMPLATE/rebase.md +++ b/.github/ISSUE_TEMPLATE/rebase.md @@ -7,6 +7,62 @@ - [ ] Fedora Changes Considerations ([example](https://github.com/coreos/fedora-coreos-tracker/issues/1222)) - [ ] Package Additions/Removals ([example](https://github.com/coreos/fedora-coreos-tracker/issues/1221)) - [ ] Test Week ([template](https://github.com/coreos/fedora-coreos-tracker/issues/new?template=test-week.md&title=tracker:+FN+Test+Week)) +- [ ] Communications Tracker ([example](https://github.com/coreos/fedora-coreos-tracker/issues/1655)) + +## At the first change checkpoint + +### Untag old packages + +`koji untag` packages from the pool that are not signed by any "active" release key (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). Active releases are N-1 and N. If branching for N+1 has already occurred, N+1 is also considered active. Use this process: + +- [ ] First, coordinate with releng to let them know you'll be untagging N-2 packages and that you'll need them to update `tag2distrepo.keys` on the `coreos-pool` tag promptly afterward (to drop the N-2 key). This minimizes the window where the key list is out of sync with what's tagged. Check the current state with: + - `koji taginfo coreos-pool` + +- [ ] Find the key short hashes for all active releases. Usually found [here](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/bodhi2/backend/templates/pungi.rpm.conf.j2). Then build the untaglist of packages not signed by any active key: + +``` +fN2key= # the key being removed +fN1key= # active +fNkey= # active +fNp1key= # active if branching has already occurred + +echo > untaglist # create or empty out file +for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do + if ! koji buildinfo $build | grep -iE "($fN1key|$fNkey|$fNp1key)" 1>/dev/null; then + echo "Adding $build to untag list" + echo "${build}" >> untaglist + fi +done +``` + +Now we have a list of builds to untag. But we need a few more sanity checks. + +- [ ] Check the current `testing-devel` image for packages signed with the N-2 key: + +``` +fN2key= +podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -i -B 9 $fN2key +podman rmi quay.io/fedora/fedora-coreos:testing-devel +``` + +If there are any packages in the output, there are two possible cases: + +1. **The package is signed by multiple keys** including an older one. This is OK, but add it to the untaglist anyway so it gets retagged into `coreos-pool` with a newer key/signature. You'll need to untag the RPM from `coreos-pool`, run a `koji dist-repo --non-latest coreos-pool key1 key2 key3` (which will remove that RPM from the repo metadata), and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key. + +2. **The package is only signed by an older (non-active) key**. This needs investigation. Something in the compose may be pulling in a stale package (dep issue?). It should not be untagged until the root cause is understood. + +- [ ] After verifying the list looks good, untag: + +``` +# use xargs so we don't exhaust bash string limit +cat untaglist | xargs -L50 koji untag-build -v coreos-pool +``` + +- [ ] Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit. + +- [ ] Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the N-1/N/N+1 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`). An example request looks [like this](https://forge.fedoraproject.org/releng/tickets/issues/13316). + - `koji taginfo coreos-pool` + - `koji edit-tag coreos-pool -x tag2distrepo.keys=" "` ## At Branching @@ -27,30 +83,34 @@ Branching is when a new stream is "branched" off of `rawhide`. This eventually b - `koji tag-build f${N+1}-coreos-continuous $BUILD` - example: `koji tag-build f36-coreos-continuous fedora-release-36-0.16` -- [ ] Add the N+1 signing key short hash (usually found [here](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2)) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`). +- [ ] Add the N+1 signing key short hash (usually found [here](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/bodhi2/backend/templates/pungi.rpm.conf.j2)) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`). An example request looks [like this](https://forge.fedoraproject.org/releng/tickets/issues/13213). - `koji taginfo coreos-pool` - `koji edit-tag coreos-pool -x tag2distrepo.keys="12c944d0 9570ff31 45719a39 9867c58f"` ### coreos-installer changes - [ ] Update coreos-installer to know about the signing key used for the future new major version of Fedora (N+1). + - The current set of trusted signing keys is available at https://fedoraproject.org/security/. + - If the Fedora (N+1) signing key isn't available yet at that site, you can also get it from https://src.fedoraproject.org/rpms/fedora-repos/tree/rawhide. - [ ] Drop the signing key for the obsolete stable release (N-2). +Example PR: https://github.com/coreos/coreos-installer/pull/1113 + ### Update `rawhide` stream -- [ ] Update [manifest.yaml](https://github.com/coreos/fedora-coreos-config/blob/rawhide/manifest.yaml) to list N+1 as the releasever. +- [ ] Update `VERSION`, `MUTATE_OS_RELEASE`, `BUILDER_IMG` in [build-args.conf](https://github.com/coreos/fedora-coreos-config/blob/rawhide/build-args.conf) ([example PR](https://github.com/coreos/fedora-coreos-config/pull/4003)) ### Enable `branched` stream -- [ ] Update [manifest.yaml](https://github.com/coreos/fedora-coreos-config/blob/branched/manifest.yaml) to list N as the releasever. -- [ ] Update [config.yaml](https://github.com/coreos/fedora-coreos-pipeline/blob/main/config.yaml) to un-comment out the `branched` stream definition. - +- [ ] Update `VERSION`, `MUTATE_OS_RELEASE`, `BUILDER_IMG` in [build-args.conf](https://github.com/coreos/fedora-coreos-config/blob/branched/build-args.conf) ([example PR](https://github.com/coreos/fedora-coreos-config/pull/4005)) +- [ ] Update [config.yaml](https://github.com/coreos/fedora-coreos-pipeline/blob/main/config.yaml) to un-comment out the `branched` stream definition ([example PR](https://github.com/coreos/fedora-coreos-pipeline/pull/904)) ## At Fedora (N) Beta ### Update [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config/) `next-devel` -- [ ] Bump `releasever` in `manifest.yaml` +- [ ] Bump `VERSION`, `MUTATE_OS_RELEASE`, `BUILDER_IMG` in `build-args.conf` +- [ ] Add the `fedora-candidate-compose` repo in `manifest.yaml` ([example PR](https://github.com/coreos/fedora-coreos-config/pull/2706)) - [ ] Update the repos in `manifest.yaml` if needed - [ ] Run `cosa fetch --dry-run --update-lockfile` - this updates the x86_64 lockfile - the others will get updated when `bump-lockfile` runs. @@ -61,6 +121,10 @@ Branching is when a new stream is "branched" off of `rawhide`. This eventually b - [ ] Disable `branched` stream since it is no longer needed. - Update [config.yaml](https://github.com/coreos/fedora-coreos-pipeline/blob/main/config.yaml) to comment out the `branched` stream definition. +### Prepare Fedora CoreOS (N) Beta announcement + +- [ ] Draft an announcement that contains information found in the "Communications Tracker", created in a step above, to inform users of Fedora CoreOS of upcoming changes in the Fedora (N) version. [(example)](https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/GK4RMQ3UFMLJGKMBUVTTWGNFVFNNSH5E/) + ### Ship rebased `next` - [ ] Ship `next` @@ -69,85 +133,39 @@ Branching is when a new stream is "branched" off of `rawhide`. This eventually b ## Preparing for Fedora (N) GA -### Update [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config/) `testing-devel` +Do these steps as soon as we have a Go confirmation for GA, usually the Thursday of the week before GA. -- [ ] Bump `releasever` in `manifest.yaml` -- [ ] Update the repos in `manifest.yaml` if needed -- [ ] Sync the lockfiles for all arches from `next-devel` -- [ ] Bump the base Fedora version in `ci/buildroot/Dockerfile` -- [ ] PR the result +### Ship a final `next` release +If the packages in `next-devel` don't exactly match the last `next` release that was done, we need to do a release with the final GA content. This ensures that what we'll promote to `testing` has the exact content in GA (plus version fast-tracks). This usually happens on the Thursday of the announcement of Go. -## At Fedora (N) GA +- [ ] Ensure final `next` release has GA content -### Ship rebased `testing` +### Build rebased `testing` and final `stable` release on N-1 -- [ ] Ship `testing` -- [ ] Set a new update barrier for the final release of N-1 on `testing`. In the barrier entry set a link to [the docs](https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/). See [discussion](https://github.com/coreos/fedora-coreos-tracker/issues/480#issuecomment-1247314065) +- [ ] Build `stable`; promote it from the `testing` branch, which should still be on N-1. Don't release it yet (i.e. don't run the `release` job). +- [ ] Build `testing`; promote it from the `next` branch instead of `testing-devel`. Don't release it yet (i.e. don't run the `release` job). -### Disable `branched` stream - -- [ ] Update [streams.groovy](https://github.com/coreos/fedora-coreos-pipeline/blob/main/streams.groovy) to remove the `branched` stream in the list of mechanical refs. - -### Untag old packages - -`koji untag` N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process: - -- [ ] Find the key short hash. Usually found [here](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2). Then: - -``` -f32key=12c944d0 -key=$f32key -echo > untaglist # create or empty out file -for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do - if koji buildinfo $build | grep $key 1>/dev/null; then - echo "Adding $build to untag list" - echo "${build}" >> untaglist - fi -done -``` - -Now we have a list of builds to untag. But we need a few more sanity checks. - -- [ ] Make sure none of the builds are used in `N` based FCOS. Check by running: - -``` -f32key=12c944d0 -key=$f32key -podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -B 9 $key -podman rmi quay.io/fedora/fedora-coreos:testing-devel -``` - -If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. - -- [ ] For any RPMS still used by `N-1` based FCOS let's remove them from the untaglist. Check by running: - -``` -f32key=12c944d0 -key=$f32key -podman run -it --rm quay.io/fedora/fedora-coreos:stable rpm -qai | grep -B 9 $key -podman rmi quay.io/fedora/fedora-coreos:stable -``` - -NOTE: This assumes `stable` is still on `N-1`. +### Update [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config/) `testing-devel` -Remove any entries from the `untaglist` file that are still being used. +- [ ] Bump `VERSION`, `MUTATE_OS_RELEASE`, `BUILDER_IMG` in `build-args.conf` +- [ ] Update the repos in `manifest.yaml` if needed +- [ ] Sync the lockfiles for all arches from `next-devel` +- [ ] Bump the base Fedora version in `ci/buildroot/Dockerfile` +- [ ] Bump the Fedora version for the test containers in `tests/kola/data/commonlib.sh` +- [ ] PR the result -- [ ] After verifying the list looks good, untag: -``` -# use xargs so we don't exhaust bash string limit -cat untaglist | xargs -L50 koji untag-build coreos-pool -``` +## At Fedora (N) GA -- [ ] Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit. +Do these steps on GA day. -- [ ] Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`). - - `koji taginfo coreos-pool` - - `koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"` +### Release rebased `testing` and final `stable` release on N-1 +- [ ] Run the `release` job for the staged `testing` and `stable` builds and start rollout. +- [ ] Set a new update barrier for the final release of N-1 on `testing`. In the barrier entry set a link to [the docs](https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/). See [discussion](https://github.com/coreos/fedora-coreos-tracker/issues/480#issuecomment-1247314065) -### Disable `next-devel` stream +### Disable `next-devel` stream if not needed We prefer to disable `next-devel` when there is no difference between `testing-devel` and `next-devel`. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N if `next-devel` and `testing-devel` are in lockstep, then disable `next-devel`. @@ -157,6 +175,13 @@ We prefer to disable `next-devel` when there is no difference between `testing-d - [ ] Update [repo-templates](https://github.com/coreos/repo-templates) [config.yaml](https://github.com/coreos/repo-templates/blob/main/config.yaml) with the version number and GPG key ID for Fedora (N). +### Disable the `fedora-candidate-compose` repo + +- [ ] Remove from the `manifest.yaml` of `next-devel` the `fedora-candidate-compose` repo + +### Prepare Fedora CoreOS (N) announcement + +- [ ] Draft an announcement that contains information found in the "Communications Tracker" and the Beta announcement, created in steps above, to inform users of Fedora CoreOS of upcoming changes in the Fedora (N) version. [(example)](https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/AEZV7QKPNXGECMYQO4T4W5IQR4X5B62F/) ## After Fedora (N) GA @@ -170,13 +195,23 @@ We prefer to disable `next-devel` when there is no difference between `testing-d - [ ] Create a new ticket from the [rebase template](https://github.com/coreos/fedora-coreos-tracker/issues/new?assignees=&labels=area%2Fplatforms%2C+kind%2Fenhancement&template=rebase.md&title=tracker:+Rebase+onto+Fedora+N) - label with `FN` label where `N` is the Fedora version. +### Update the FCOS Meeting-Action Template + +Now that Fedora N is GA, we need to start tracking the release schedule of Fedora N+1 during the weekly [Fedora CoreOS Community Meeting](https://github.com/coreos/fedora-coreos-tracker/blob/main/README.md#meetings). + +- [ ] Update the "Review Fedora N Release Schedule" topic and link to point to Fedora N+1 in the [FCOS meeting-action template](https://github.com/coreos/fcos-meeting-action/blob/main/static/meeting-template.md) -## Miscellaneous container updates -These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment. +## Miscellaneous ecosystem updates + +These are various projects (primarily containers) in use throughout our ecosystem which require updating on new Fedora versions. +We should update or open a ticket to track updating them once a new Fedora release is out. +If you open a ticket instead of doing the update add a link to the ticket as comment. - [ ] Update coreos-assembler or open ticket to update: - [Dockerfile](https://github.com/coreos/coreos-assembler/blob/main/Dockerfile) + - [Dockerfiles for kola test containers](https://github.com/coreos/coreos-assembler/tree/main/tests/containers) + - [Dockerfile for the OpenShift CI buildroot image](https://github.com/openshift/release/blob/master/ci-operator/config/coreos/coreos-assembler/coreos-coreos-assembler-main.yaml) - [ ] Update coreos-installer - [Dockerfile](https://github.com/coreos/coreos-installer/blob/main/Dockerfile) - [ ] Update Ignition @@ -185,13 +220,19 @@ These are various containers in use throughout our ecosystem. We should update o - [Dockerfile](https://github.com/coreos/butane/blob/main/Dockerfile) - [ ] Update fedora-coreos-cincinnati - [Dockerfile](https://github.com/coreos/fedora-coreos-cincinnati/blob/main/dist/fedora-infra/Dockerfile) + - [ImageStream](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/coreos-cincinnati/templates/imagestream.yml.j2) + - [BuildConfig](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/coreos-cincinnati/templates/buildconfig.yml.j2) + - [Git Hash Variables (Optional)](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/coreos-cincinnati/vars) - [ ] Update config-bot - [Dockerfile](https://github.com/coreos/fedora-coreos-releng-automation/blob/main/config-bot/Dockerfile) - [ ] Update coreos-koji-tagger - [Dockerfile](https://github.com/coreos/fedora-coreos-releng-automation/blob/main/coreos-koji-tagger/Dockerfile) - - [ImageStream](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/coreos-koji-tagger/templates/imagestream.yml) - - [BuildConfig](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/coreos-koji-tagger/templates/buildconfig.yml) -- [ ] Update coreos-ostree-importer - - [Dockerfile](https://github.com/coreos/fedora-coreos-releng-automation/blob/main/coreos-ostree-importer/Dockerfile) - - [ImageStream](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/coreos-ostree-importer/templates/imagestream.yml) - - [BuildConfig](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/coreos-ostree-importer/templates/buildconfig.yml) + - [ImageStream](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/coreos-koji-tagger/templates/imagestream.yml.j2) + - [BuildConfig](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/coreos-koji-tagger/templates/buildconfig.yml.j2) +- [ ] Update fedora-ostree-pruner + - [Dockerfile](https://github.com/coreos/fedora-coreos-releng-automation/blob/main/fedora-ostree-pruner/Dockerfile) + - [ImageStream](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/fedora-ostree-pruner/templates/imagestream.yml.j2) + - [BuildConfig](https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshift-apps/fedora-ostree-pruner/templates/buildconfig.yml.j2) +- [ ] Update the grub2 versions in the bootupd e2e test + - [test script](https://github.com/coreos/bootupd/blob/main/tests/e2e-update/e2e-update.sh) + diff --git a/Design.md b/Design.md index af85d0c..6b83bb1 100644 --- a/Design.md +++ b/Design.md @@ -18,7 +18,7 @@ conclusion should be summarized here with a link to the issue. ## OSTree Delivery Format -- Originally discussed in issue [#23](https://github.com/coreos/fedora-coreos-tracker/issues/23). +- Originally discussed in issue [#23](https://github.com/coreos/fedora-coreos-tracker/issues/23). ### Summary: @@ -29,7 +29,7 @@ end user systems: repo) on a server and fetched via HTTP requests. - rojig: uses a special rojig RPM and re-assembles OSTree commit from RPMs already on mirrors. -- OCI: OSTree commits are packaged up in OCI container images and delivered +- OCI: OSTree commits are packaged up in OCI container images and delivered via a container registry. Currently the plan in Fedora CoreOS is to deliver content via a plain @@ -87,6 +87,7 @@ The release process integrates with Fedora's release milestones in the following - Week -1 (Fedora "Go" Decision): `next` release: - `next` release with final Fedora GA content - Week 0 (GA release): triple release: + - `stable` release promoted from previous `testing` (on N-1) - `testing` release promoted from previous `next` - `next` release contains latest Fedora N content, including Bodhi updates - Week 2: triple release: @@ -101,7 +102,7 @@ Because production refs are unversioned, users will seamlessly upgrade between F ## Disk Layout -- Originally discussed in issue [#18](https://github.com/coreos/fedora-coreos-tracker/issues/18). +- Originally discussed in issue [#18](https://github.com/coreos/fedora-coreos-tracker/issues/18). See also [dustymabe's comment](https://github.com/coreos/fedora-coreos-tracker/issues/18#issuecomment-409668929) summarizing the discussion in the FCOS meeting. - Filesystem details were discussed in [#33](https://github.com/coreos/fedora-coreos-tracker/issues/33). @@ -227,16 +228,7 @@ Originally discussed in [#71](https://github.com/coreos/fedora-coreos-tracker/is Originally discussed in [#68](https://github.com/coreos/fedora-coreos-tracker/issues/68). - OpenStack environments do not require a cloud agent -- We will provide any base level of functionality with ignition and coreos-metadata - -### Packet: - -Originally discussed in [#69](https://github.com/coreos/fedora-coreos-tracker/issues/69). - -- On the first boot, Packet requires the machine to phone home to report a successful boot. This will be [handled by coreos-metadata](https://github.com/coreos/coreos-metadata/issues/120). -- Packet provides the IPv4 public address via DHCP, allowing a machine to acquire network via standard mechanisms. However, to obtain a private IPv4 address or a public IPv6 address (on the same interface), networking must be configured using metadata from an HTTP metadata service. This can be handled by coreos-metadata in the initramfs, but it [may need to learn to configure NetworkManager or nm-state](https://github.com/coreos/fedora-coreos-tracker/issues/111) depending on the outcome of [#24](https://github.com/coreos/fedora-coreos-tracker/issues/24). -- Packet needs the serial console on x86 to be directed to `ttyS1`, not `ttyS0`, requiring [cloud-specific bootloader configuration](https://github.com/coreos/fedora-coreos-tracker/issues/110). A different serial console configuration is required on ARM64. -- On many Linux OSes, Packet sets a randomized root password which is then available from the Packet console for 24 hours. This allows the serial (SOS) console to be used for interactive debugging. Container Linux, instead, enables autologin on the console by default. To avoid surprising users, Fedora CoreOS will do neither. For interactive console access, users can use Ignition to enable autologin or to set a password on the `core` account, and we'll document how to do that. +- We will provide any base level of functionality with ignition and coreos-metadata ### Open questions: @@ -274,7 +266,7 @@ This means: Originally discussed in [#114](https://github.com/coreos/fedora-coreos-tracker/issues/114). -We will not enable autologin on serial or VGA consoles by default, even on platforms (e.g. Azure, DigitalOcean, GCP, Packet) which provide authenticated console access. Doing so would provide an access vector that could surprise users unfamiliar with their platform's console access mechanism and access control policy. For users who wish to use the console for debugging, we will provide documentation for using Ignition to enable autologin or to set a user password. +We will not enable autologin on serial or VGA consoles by default, even on platforms (e.g. Azure, DigitalOcean, GCP) which provide authenticated console access. Doing so would provide an access vector that could surprise users unfamiliar with their platform's console access mechanism and access control policy. For users who wish to use the console for debugging, we will provide documentation for using Ignition to enable autologin or to set a user password. ### Automatically disable SMT when needed to address vulnerabilities @@ -344,8 +336,6 @@ next-devel | 10 testing-devel | 20 rawhide | 91 branched | 92 -bodhi-updates-testing | 93 -bodhi-updates | 94 For developer builds (those not produced by the official pipeline), Z is always `dev`. @@ -364,8 +354,6 @@ next-devel | 31.20191018.10.10 | 11th build of the day testing-devel | 31.20191018.20.0 | rawhide | 33.20191018.91.0 | F33-based, first build of the day branched | 32.20191018.92.0 | -bodhi-updates-testing | 31.20191018.93.0 | -bodhi-updates | 31.20191018.94.0 | (any developer build) | 31.20191018.dev.2 | Third build of the day We are not committing to this version scheme indefinitely, and may change it in future if it proves unworkable. A new Fedora major release (X bump) would be a good time to make such a change. We don't intend Fedora CoreOS version numbers to be parsed by machine; they're meant to help humans quickly determine the salient properties of a release. diff --git a/PRD.txt b/PRD.txt index 1cd5a63..a09dd46 100644 --- a/PRD.txt +++ b/PRD.txt @@ -117,7 +117,7 @@ All artifacts will be downloadable from the getfedora.org website. Similarly, fo === Delivery Format === -Artifacts will be delivered as cloud images on Amazon EC2, Azure, DigitalOcean, Google Compute Engine, and Packet; as downloadable images for OpenStack, QEMU, VirtualBox, and VMware; and as ISO images, netboot images, and installable raw images for bare metal systems. We may add other public cloud images and downloadable formats to meet demand or anticipated need. +Artifacts will be delivered as cloud images on Amazon EC2, Azure, DigitalOcean, and Google Compute Engine; as downloadable images for OpenStack, QEMU, VirtualBox, and VMware; and as ISO images, netboot images, and installable raw images for bare metal systems. We may add other public cloud images and downloadable formats to meet demand or anticipated need. === Architectures === diff --git a/README.md b/README.md index d01c229..decca97 100644 --- a/README.md +++ b/README.md @@ -25,13 +25,13 @@ technologies and produce Fedora CoreOS. # Communication channels for Fedora CoreOS -- main mailing list: [coreos@lists.fedoraproject.org](https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/) -- status mailing list: [coreos-status@lists.fedoraproject.org](https://lists.fedoraproject.org/archives/list/coreos-status@lists.fedoraproject.org/) (announcements/important messages) -- IRC/Matrix: [`#fedora-coreos` on Libera.Chat](https://web.libera.chat/#fedora-coreos) (ircs://irc.libera.chat:6697/#fedora-coreos) or [`#coreos:fedoraproject.org` on Matrix](https://matrix.to/#/#coreos:fedoraproject.org) -- forum at [https://discussion.fedoraproject.org/tag/coreos](https://discussion.fedoraproject.org/tag/coreos) -- feature planning and important issue tracking at [github.com/coreos/fedora-coreos-tracker](https://github.com/coreos/fedora-coreos-tracker) -- website at [https://getfedora.org/coreos/](https://getfedora.org/coreos/) -- documentation at [https://docs.fedoraproject.org/en-US/fedora-coreos/](https://docs.fedoraproject.org/en-US/fedora-coreos/) +- Main mailing list: [coreos@lists.fedoraproject.org](https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/) +- Status mailing list: [coreos-status@lists.fedoraproject.org](https://lists.fedoraproject.org/archives/list/coreos-status@lists.fedoraproject.org/) (announcements/important messages) +- Chat room: [`#coreos:fedoraproject.org` on Matrix](https://chat.fedoraproject.org/#/room/#coreos:fedoraproject.org) +- Forum at [https://discussion.fedoraproject.org/tag/coreos-wg](https://discussion.fedoraproject.org/tag/coreos-wg) +- Feature planning and important issue tracking at [github.com/coreos/fedora-coreos-tracker](https://github.com/coreos/fedora-coreos-tracker) +- Website at [https://getfedora.org/coreos/](https://getfedora.org/coreos/) +- Documentation at [https://docs.fedoraproject.org/en-US/fedora-coreos/](https://docs.fedoraproject.org/en-US/fedora-coreos/) - Twitter: [@fedoracoreos](https://twitter.com/fedoracoreos) # Roadmap/Plans @@ -49,7 +49,7 @@ Thus, new package requests are carefully scrutinized to weigh the benefits and drawbacks of adding an additional package. If you would like to propose the inclusion of a new package in the base set of packages, -please file a [new package request](https://github.com/coreos/fedora-coreos-tracker/issues/new?labels=kind/enhancement&template=new-package.md&title=New+Package+Request%3A+%3Cpackage+name%3E). +please file a [new package request](https://github.com/coreos/fedora-coreos-tracker/issues/new/choose). # Releases @@ -58,56 +58,72 @@ See [RELEASES.md](RELEASES.md). # Meetings The Fedora CoreOS Working Group has a weekly meeting. The meeting usually -happens in `#fedora-meeting-1` on irc.libera.chat and the schedule for the -meeting can be found here: https://calendar.fedoraproject.org/CoreOS/ -Currently, meetings are at `16:30 UTC` on Wednesdays. +happens in +[#meeting-1:fedoraproject.org](https://matrix.to/#/#meeting-1:fedoraproject.org) +on Matrix and the schedule for the meeting can be found here: +https://calendar.fedoraproject.org/CoreOS/ Currently, meetings are at +[`15:30 UTC`](https://time.is/15:30+UTC) on Wednesdays. + +As the +[Matrix bridge to Libera Chat is shutdown](https://matrix.org/blog/2023/11/28/shutting-down-bridge-to-libera-chat/), +you can not attend the meeting from IRC and you have to join using Matrix. ## Steps to run the meeting +The fedora meeting host can follow the guide which is curated by the [fcos-meeting-action](https://github.com/coreos/fcos-meeting-action) repo. +Every Wednesday a new checklist will be available in the form of a issue in the fcos-meeting-action repo, which can be used to run the meeting. + +If the action meeting repo is not available for some reason, the host can follow the below steps to run the meeting. + +
+Legacy Meeting steps + - `cd` to a local checkout of this repo and `git pull` - Ping [meeting people](https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt) in `#fedora-coreos` on libera.chat - `bash meeting-people.txt` - - copy lines of output and paste into `#fedora-coreos` channel -- Navigate to `#fedora-meeting-1` on libera.chat -- Type `#startmeeting fedora_coreos_meeting` -- `#topic roll call` + - copy lines of output and paste into + [`#coreos:fedoraproject.org`](https://chat.fedoraproject.org/#/room/#coreos:fedoraproject.org) + on Matrix +- Navigate to + [`#meeting-1:fedoraproject.org`](https://matrix.to/#/#meeting-1:fedoraproject.org) + on Matrix +- Type: + - `!startmeeting fedora_coreos_meeting` + - `!topic roll call` Wait for 2-4 minutes for people to check in for the roll call. -- `#chair` all the people present for the meeting -- `#topic Action items from last meeting` +- `!topic Action items from last meeting` -Find the last meeting log from -[meetbot](https://meetbot-raw.fedoraproject.org/teams/fedora_coreos_meeting) -and post the action items in the meeting for people to -update the status of. +Find the last meeting log from [meetbot](https://meetbot.fedoraproject.org/) +and post the action items in the meeting for people to update the status of. - After they are done move to each `meeting` ticket from [this tracker](https://github.com/coreos/fedora-coreos-tracker/labels/meeting) Do the following for each ticket -- `#topic` Ticket subject -- `#link` link\_to\_the\_ticket +- `!topic` Ticket subject +- `!link ` During the meeting, you can give people action items for them to complete: -- `#action ` description of what needs to be done +- `!action ` description of what needs to be done When all topics are over, go for open floor: -- `#topic Open Floor` +- `!topic Open Floor` After open floor, end the meeting. -- `#endmeeting` +- `!endmeeting` Then, when convenient: - Remove `meeting` labels from [tickets that were discussed](https://github.com/coreos/fedora-coreos-tracker/labels/meeting) - Send an email to [coreos@lists.fedoraproject.org](mailto:coreos@lists.fedoraproject.org) with the -details of the meeting from [meetbot page](https://meetbot.fedoraproject.org/sresults/?group_id=fedora_coreos_meeting&type=team). +details of the meeting from [meetbot page](https://meetbot.fedoraproject.org/). Minutes in textual format are directly available using `.txt` as URL extension. It's easiest to get the Minutes/Minutes (text)/Log URLs by copying the footer that Meetbot prints after `#endmeeting`. You can see examples in the @@ -125,6 +141,7 @@ Log: ``` +
# Voting @@ -160,3 +177,23 @@ Working days: non-holiday weekdays. Relevant holidays are the national holidays # Working Group Members and Points of Contact Please see [meeting-people.txt](https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt). + +# Metrics + +To view CountME stats you can use a tool called +[sqlitevis](https://sqliteviz.com/) to view the +CountME database and make graphs. This can easily be done with a +single URL but due to [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) +you have to run your browser in a specific mode to allow the +application to download the database and the inquiries file: + +``` +chromium-browser --disable-web-security --user-data-dir=~/chrome-disable-web-security/ +# OR +google-chrome-stable --disable-web-security --user-data-dir=~/chrome-disable-web-security/ +``` + +Now navigate to +[this](https://sqliteviz.com/app/#/load?data_url=https%3A%2F%2Fdata-analysis.fedoraproject.org%2Fcsv-reports%2Fcountme%2Ftotals-countme.db&data_format=sqlite&inquiry_url=https%3A%2F%2Fraw.githubusercontent.com%2Fcoreos%2Ffedora-coreos-tracker%2Frefs%2Fheads%2Fmain%2Fmetrics%2Ffcos-sqlitevis.json) +URL in the browser and it should autoload the database and the inquiries. This +URL was generated from the [sqlitevis docs](https://sqliteviz.com/docs/sharing/). diff --git a/docs/ci-and-builds.md b/docs/ci-and-builds.md index e06f704..aeffbf0 100644 --- a/docs/ci-and-builds.md +++ b/docs/ci-and-builds.md @@ -51,7 +51,7 @@ Examples: ## quay.io/coreos-assembler namespace -A key aspect of Fedora CoreOS as well as RHEL CoreOS is [coreos-assembler](https://github.com/coreos/coreos-assembler). As of today, we build it in quay.io and deliver it that way in the `quay.io/coreos-assembler` namespace. The list of administrators for this namespace is managed independently of anything else. If you think you need administrator access, file a ticket or ask on #fedora-coreos IRC. +A key aspect of Fedora CoreOS as well as RHEL CoreOS is [coreos-assembler](https://github.com/coreos/coreos-assembler). As of today, we build it in quay.io and deliver it that way in the `quay.io/coreos-assembler` namespace. The list of administrators for this namespace is managed independently of anything else. If you think you need administrator access, file a ticket or ask on [`#coreos:fedoraproject.org` on Matrix](https://chat.fedoraproject.org/#/room/#coreos:fedoraproject.org). ### The buildroot container: quay.io/coreos-assembler/fcos-buildroot:testing-devel diff --git a/docs/fedora-coreos-kernel-bisect.md b/docs/fedora-coreos-kernel-bisect.md index 040555b..0742cfc 100644 --- a/docs/fedora-coreos-kernel-bisect.md +++ b/docs/fedora-coreos-kernel-bisect.md @@ -37,7 +37,7 @@ Here's a summary of what those branches are used for: - `os-build` - The latest bits that track the under development yet to be release kernel. -- `fedora-5.16` +- `fedora-6.3` - Follows a particular released kernel stream. This is where things are merged before they are fed into dist-git. If you want a commit reverted this is where it will land first. @@ -54,7 +54,7 @@ environment directly in the VM. If not you'll probably want to use a container for your kernel builds. Here's how to start up a container: ``` -podman run -it --name=kbuild -v /path/to/kernel/git/:/path/to/kernel/git/ registry.fedoraproject.org/fedora:37 +podman run -it --name=kbuild -v /path/to/kernel/git/:/path/to/kernel/git/ registry.fedoraproject.org/fedora:38 ``` NOTE: try to use the same Fedora Cloud or Fedora container version as @@ -64,7 +64,7 @@ Once inside the VM or container we need to install some software to build the ke ``` sudo dnf update -y && \ -sudo dnf install -y rpm-build rsync 'dnf-command(builddep)' && \ +sudo dnf install -y make rpm-build rsync 'dnf-command(builddep)' && \ sudo dnf builddep -y kernel # reboot here if in a VM ``` @@ -76,7 +76,9 @@ which makes very large files: ``` cd /path/to/kernel/git/ -curl https://src.fedoraproject.org/rpms/kernel/raw/f37/f/kernel-x86_64-fedora.config > .config +RELEASE=f38 # or RELEASE=rawhide +curl "https://src.fedoraproject.org/rpms/kernel/raw/${RELEASE}/f/kernel-x86_64-fedora.config" > .config.fedora +cp .config.fedora .config sed -i 's/CONFIG_DEBUG_KERNEL=y/CONFIG_DEBUG_KERNEL=n/' .config ``` @@ -86,8 +88,10 @@ To build and install the kernel directly on the system (i.e. on Fedora Cloud Bas you can run the following: ``` +# Set make target. See https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec +make_target=bzImage # for x86_64 or vmlinux(ppc64le) or vmlinuz.efi(aarch64) make olddefconfig -make -j$(nproc) bzImage +make -j$(nproc) $make_target make -j$(nproc) modules sudo make modules_install sudo make install @@ -109,7 +113,12 @@ Then run the following script to build and install the kernel: cat build.sh #!/bin/bash set -eux -o pipefail -make -j$(nproc) bzImage +cp .config.fedora .config +sed -i 's/CONFIG_DEBUG_KERNEL=y/CONFIG_DEBUG_KERNEL=n/' .config +# Set make target. See https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec +make_target=bzImage # for x86_64 or vmlinux(ppc64le) or vmlinuz.efi(aarch64) +make olddefconfig +make -j$(nproc) $make_target make -j$(nproc) modules sudo make modules_install sudo make install @@ -127,6 +136,12 @@ sudo rm -vf /boot/initramfs*bisect* /boot/vmlinuz-*bisect* /boot/System.map-*bis sudo rm -rf /lib/modules/*bisect* ``` +Then you can automate with: + +``` +bash clean.sh && bash build.sh +``` + ## 2. Directly Building and Creating an RPM from the Kernel Source git repo In this scenario we're creating an RPM that can either then be package @@ -185,3 +200,30 @@ Jens Axboe linux-block@vger.kernel.org linux-kernel@vger.kernel.org ``` + +## Testing out fixes with Fedora's kernel + +Once you have a proposed fix/patch you can easily build a Fedora kernel RPM by +adding your patch to the [`linux-kernel-test.patch` file](https://docs.fedoraproject.org/en-US/quick-docs/kernel/testing-patches/#_applying_the_patch) +in the [kernel distgit repo](https://src.fedoraproject.org/rpms/kernel). + +After adding your patch you can then use `fedpkg` to build a new +kernel for your target architecture. For example: + +``` +fedpkg scratch-build --srpm --arch=x86_64 +``` + +Once the build is complete you can grab the RPMs using the `koji` CLI: + +``` +koji download-task +``` + +Placing these RPMs into the `overrides/rpm` directory and do a new COSA build +will give you a CoreOS build with the patched kernel. + +After the tested patch looks good you can then open a PR to the `fedora-X.Y` +branch in the `kernel-ark` repo. See the above +[Kernel Source git Repos](#kernel-source-git-repos) +section for more details. diff --git a/docs/fedora-coreos-systemd-bisect.md b/docs/fedora-coreos-systemd-bisect.md new file mode 100644 index 0000000..673804a --- /dev/null +++ b/docs/fedora-coreos-systemd-bisect.md @@ -0,0 +1,66 @@ + +# Systemd regressions need bisecting + +Similar to the kernel, systemd is often a core component of our +stack that has regressions that aren't easy to identify just by +inspecting a changelog. + +## Systemd Source git Repos + +There are a few kernel source git repositories to know about: + +- `https://github.com/systemd/systemd.git` + - Where the latest upstream development happens +- `https://github.com/systemd/systemd-stable.git` + - Where stable/LTS tags are handled (backports to stable branches happen here) + +There is also the [Fedora dist-git repo](https://src.fedoraproject.org/rpms/systemd). + +## Creating a Kernel Build Environment + +You can use a container to build systemd from upstream. + +``` +SHARED=/path/to/shared/directory/ +RELEASE=38 +podman run -it --name=systemdbuild -v "${SHARED}:${SHARED}" "registry.fedoraproject.org/fedora:${RELEASE}" +``` + +``` +sudo dnf update -y && \ +sudo dnf install -y make rpm-build rsync 'dnf-command(builddep)' && \ +sudo dnf builddep -y systemd +``` + +We can now make changes to the git repo (revert commits, etc) and run a few +commands to build systemd. If doing a +[`git` bisect](https://www.kernel.org/doc/html/latest/admin-guide/bug-bisect.html) +run the commands needed to start the bisect. + +## Doing the systemd build/test + +To build systemd you can run the following commands. These commands +were adapted from the notes in the +[Systemd README](https://github.com/systemd/systemd/blob/579fbe5b789cbee10546f6274c39be311e71e49c/README#L233-L247). + + +``` +meson setup build/ +``` + +And then the following can be iterated upon for each commit to test: + +``` +export DESTDIR=/path/to/shared/directory/fcos/overrides/rootfs/ +ninja -C build && ninja -C build install +``` + +NOTE: If you run into `permission denied` errors when copying the files around check for SELinux denails. + +Now you can run COSA to build/test. From the COSA directory: + +``` +cosa build && cosa kola run mytest +``` + +Now you can iterate until you find the problematic commit. diff --git a/meeting-people.txt b/meeting-people.txt index a5dc430..967e3cf 100644 --- a/meeting-people.txt +++ b/meeting-people.txt @@ -1,23 +1,25 @@ # List of people to ping before the Fedora CoreOS community meetings. # Please keep this list in alphabetical order. -tail -n +5 $0 | tr '\n' ' ' && echo -e '\nFCOS community meeting in #fedora-meeting-1' && echo "If you don't want to be pinged remove your name from this file: https://github.com/coreos/fedora-coreos-tracker/blob/main/meeting-people.txt" -exit 0 - -aaradhak -anthr76 -davdunc -dustymabe -gursewak -jaimelm -jbrooks -jcajka -jdoss -jlebon -jmarrero -lorbus -miabbott -nasirhm -ravanelli -saqali -skunkerk -walters +@aaradhak:matrix.org +@apiaseck:matrix.org +@bipinbn:fedora.im +@bri:transfem.dev +@davdunc:fedora.im +@dustymabe:matrix.org +@guidon:guidon.ems.host +@gurssing:matrix.org +@jaimelm:fedora.im +@jbrooks:matrix.org +@jbtrystram:matrix.org +@jdoss:fedora.im +@jlebon:fedora.im +@jmarrero:matrix.org +@marmijo:fedora.im +@miabbott:fedora.im +@quentin9696:matrix.org +@rapneset:matrix.org +@ravanelli:fedora.im +@tlbueno:fedora.im +@walters:fedora.im +@ydesouza:fedora.im +@yves:siegrist.io diff --git a/metadata/README.md b/metadata/README.md index 495495d..8fd7f1d 100644 --- a/metadata/README.md +++ b/metadata/README.md @@ -80,4 +80,3 @@ RPMs and our configuration into images and ostree commits. Projects: - https://github.com/coreos/coreos-assembler - - https://github.com/coreos/fedora-coreos-releng-automation/blob/main/coreos-meta-translator/trans.py diff --git a/metadata/release/sample.json b/metadata/release/sample.json index fad5ba1..23818b4 100644 --- a/metadata/release/sample.json +++ b/metadata/release/sample.json @@ -36,6 +36,17 @@ } } }, + "applehv": { + "artifacts": { + "raw.gz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-applehv.raw.gz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-applehv.raw.gz.sig", + "sha256": "a889159d661339e635372b807f0a98bb93c64aabfaf89a801b2f03491488f0ef" + } + } + } + }, "azure": { "artifacts": { "vhd.xz": { @@ -91,6 +102,28 @@ } } }, + "hetzner": { + "artifacts": { + "raw.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-hetzner.raw.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-hetzner.raw.xz.sig", + "sha256": "a889159d661339e635372b807f0a98bb93c64aabfaf89a801b2f03491488f0ef" + } + } + } + }, + "hyperv": { + "artifacts": { + "vhdx.zip": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-hyperv.vhdx.zip", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-hyperv.vhdx.zip.sig", + "sha256": "a889159d661339e635372b807f0a98bb93c64aabfaf89a801b2f03491488f0ef" + } + } + } + }, "ibmcloud": { "artifacts": { "qcow2.xz": { @@ -104,10 +137,10 @@ }, "kubevirt": { "artifacts": { - "qcow2.xz": { + "ociarchive": { "disk": { - "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-kubevirt.qcow2.xz", - "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-kubevirt.qcow2.xz.sig", + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-kubevirt.ociarchive", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-kubevirt.ociarchive.sig", "sha256": "2accf3a1385b9eb435224bb0e1595f4b3e4344c1cc05b1d2484e163c66f35bca" } } @@ -170,6 +203,28 @@ } } }, + "oraclecloud": { + "artifacts": { + "qcow2.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-oraclecloud.qcow2.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-oracecloud.qcow2.xz.sig", + "sha256": "868da197ae9179aded982ea6445d7d5e30acf8d03cdcdc32acfe2003d2c65491" + } + } + } + }, + "proxmoxve": { + "artifacts": { + "raw.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-proxmoxve.qcow2.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20190801.0/x86_64/fedora-coreos-30.20190801.0-proxmoxve.qcow2.xz.sig", + "sha256": "394cd6431b19c82a46a7215ebead15960faf9814092203456d56960a1b4d8777" + } + } + } + }, "qemu": { "artifacts": { "qcow2.xz": { diff --git a/metadata/stream/rationale.yaml b/metadata/stream/rationale.yaml index 8bcf65f..4072f34 100644 --- a/metadata/stream/rationale.yaml +++ b/metadata/stream/rationale.yaml @@ -10,8 +10,8 @@ architectures: artifacts: # Some of these will be useful for many users, such as qemu or # openstack. Some will likely only be useful for cloud operators, - # such as digitalocean or packet. Some, such as aws, are useful - # for users in special situations. + # such as digitalocean. Some, such as aws, are useful for users + # in special situations. aliyun: release: 30.1.2.3 formats: @@ -21,6 +21,14 @@ architectures: signature: https://artifacts.example.com/g0xah6aenvaaVosh.qcow2.xz.sig sha256: 149afbf4c8996fb92427ae3b0c44298fc1ce41e4649b934ca495991b7852b855 uncompressed-sha256: d02d5ac0f2a2789602e9df950c38acb15380d2799b4bdb59394e4eeabdd3a662 + applehv: + release: 30.1.2.3 + formats: + "raw.gz": + disk: + location: https://artifacts.example.com/quohgh8ei0uzaD5a.raw.gz + signature: https://artifacts.example.com/quohgh8ei0uzaD5a.raw.gz.sig + sha256: 4c8996fb92427ae41e4649b934ca4e3b0c44298fc1c149afbf95991b7852b855 aws: release: 30.1.2.3 formats: @@ -78,6 +86,22 @@ architectures: signature: https://artifacts.example.com/ais7tah1aa7Ahvei.tar.gz.sig sha256: 96fb92427ae41e4649b934ca495991b7852b85e3b0c44298fc1c149afbf4c895 uncompressed-sha256: 38acb15d02d5ac0f2a2789602e9df950c380d2799b4bdb59394e4eeabdd3a662 + hetzner: + release: 30.1.2.3 + formats: + "raw.xz": + disk: + location: https://artifacts.example.com/quohgh8ei0uzaD5a.raw.xz + signature: https://artifacts.example.com/quohgh8ei0uzaD5a.raw.xz.sig + sha256: 4c8996fb92427ae41e4649b934ca4e3b0c44298fc1c149afbf95991b7852b855 + hyperv: + release: 30.1.2.3 + formats: + "vhdx.zip": + disk: + location: https://artifacts.example.com/quohgh8ei0uzaD5a.vhdx.zip + signature: https://artifacts.example.com/quohgh8ei0uzaD5a.vhdx.zip.sig + sha256: 4c8996fb92427ae41e4649b934ca4e3b0c44298fc1c149afbf95991b7852b855 ibmcloud: release: 30.1.2.3 formats: @@ -90,12 +114,11 @@ architectures: kubevirt: release: 30.1.2.3 formats: - "qcow2.xz": + "ociarchive": disk: - location: https://artifacts.example.com/Kiejeeb6ohpu8Eel.qcow2.xz - signature: https://artifacts.example.com/Kiejeeb6ohpu8Eel.qcow2.xz.sig + location: https://artifacts.example.com/Kiejeeb6ohpu8Eel.ociarchive + signature: https://artifacts.example.com/Kiejeeb6ohpu8Eel.ociarchive.sig sha256: 2427ae41e4649b934ca495991b7852b85e3b0c44298fc1c149afbf4c8996fb95 - uncompressed-sha256: 38acb15d02d5ac0f2a2789602e9df950c380d2799b4bdb59394e4eeabdd3a662 metal: release: 30.1.2.3 formats: @@ -140,15 +163,23 @@ architectures: signature: https://artifacts.example.com/oKooheogobofai8l.qcow2.xz.sig sha256: ae41e4649b934ca495991b785e3b0c44298fc1c149afbf4c8996fb924272b855 uncompressed-sha256: 38acb15d02d5ac0f2a2789602e9df950c380d2799b4bdb59394e4eeabdd3a662 - packet: + oraclecloud: release: 30.1.2.3 formats: - "raw.xz": + "qcow2.xz": disk: - location: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz - signature: https://artifacts.example.com/Oofohng0xo2phai5.raw.xz.sig - sha256: e41e4649b934ca495991b7852b85e3b0c44298fc1c149afbf4c8996fb92427a5 - uncompressed-sha256: 38acb15d02d5ac0f2a2789602e9df950c380d2799b4bdb59394e4eeabdd3a662 + location: https://artifacts.example.com/oKooheogobofai8l.qcow2.xz + signature: https://artifacts.example.com/oKooheogobofai8l.qcow2.xz.sig + sha256: 868da197ae9179aded982ea6445d7d5e30acf8d03cdcdc32acfe2003d2c65491" + uncompressed-sha256: 75a5c30bf84a605cc9fa617e856d9523d8d4c50607837a7d33e4d81e9809891a + proxmoxve: + release: 30.1.2.3 + formats: + "qcow2.xz": + disk: + location: https://artifacts.example.com/TieshohWah0aewai/.qcow2.xz + signature: https://artifacts.example.com/TieshohWah0aewai/.qcow2.xz.sig + sha256: 394cd6431b19c82a46a7215ebead15960faf9814092203456d56960a1b4d8777 qemu: release: 30.1.2.3 formats: @@ -243,7 +274,3 @@ architectures: # currently recommended image, and its release. release: 30.1.2.3 digest-ref: exampleregistry.io/fcos/fcos@sha256:67a81539946ec0397196c145394553b8e0241acf27b14ae9de43bc56e167f773 - packet: - # Images don't have addressable versions, so an operating system - # slug is the best we can do. - image: fedora_coreos_stable diff --git a/metadata/stream/sample.json b/metadata/stream/sample.json index 58f25ae..134ebb0 100644 --- a/metadata/stream/sample.json +++ b/metadata/stream/sample.json @@ -20,6 +20,18 @@ } } }, + "applehv": { + "release": "33.20210412.3.0", + "formats": { + "raw.gz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-applehv.x86_64.raw.gz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-applehv.x86_64.raw.gz.sig", + "sha256": "728e876d87ec71de27fc1d882840e6877346423433339a2b8606fa28e57413fd" + } + } + } + }, "aws": { "release": "33.20210412.3.0", "formats": { @@ -96,6 +108,30 @@ } } }, + "hetzner": { + "release": "33.20210412.3.0", + "formats": { + "raw.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-hetzner.x86_64.raw.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-hetzner.x86_64.raw.xz.sig", + "sha256": "728e876d87ec71de27fc1d882840e6877346423433339a2b8606fa28e57413fd" + } + } + } + }, + "hyperv": { + "release": "33.20210412.3.0", + "formats": { + "vhdx.zip": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-hyperv.x86_64.vhdx.zip", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-hyperv.x86_64.vhdx.zip.sig", + "sha256": "728e876d87ec71de27fc1d882840e6877346423433339a2b8606fa28e57413fd" + } + } + } + }, "ibmcloud": { "release": "33.20210412.3.0", "formats": { @@ -112,12 +148,11 @@ "kubevirt": { "release": "33.20210412.3.0", "formats": { - "qcow2.xz": { + "ociarchive": { "disk": { - "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-kubevirt.x86_64.qcow2.xz", - "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-kubevirt.x86_64.qcow2.xz.sig", - "sha256": "6343b99ca70975bd821050f274aa1db0898fb88aae95a79f63d18a2e2a489e26", - "uncompressed-sha256": "744f25cf86927fe4780b57cd75c2d5b979e15336e4c9bd02fe4f71827d820d4c" + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-kubevirt.x86_64.ociarchive", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-kubevirt.x86_64.ociarchive.sig", + "sha256": "6343b99ca70975bd821050f274aa1db0898fb88aae95a79f63d18a2e2a489e26" } } } @@ -192,6 +227,31 @@ } } }, + "oraclecloud": { + "release": "33.20210412.3.0", + "formats": { + "qcow2.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-oraclecloud.x86_64.qcow2.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-oraclecloud.x86_64.qcow2.xz.sig", + "sha256": "868da197ae9179aded982ea6445d7d5e30acf8d03cdcdc32acfe2003d2c65491", + "uncompressed-sha256": "75a5c30bf84a605cc9fa617e856d9523d8d4c50607837a7d33e4d81e9809891a" + } + } + } + }, + "proxmoxve": { + "release": "33.20210412.3.0", + "formats": { + "qcow2.xz": { + "disk": { + "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-proxmoxve.x86_64.qcow2.xz", + "signature": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/33.20210412.3.0/x86_64/fedora-coreos-33.20210412.3.0-proxmoxve.x86_64.qcow2.xz.sig", + "sha256": "394cd6431b19c82a46a7215ebead15960faf9814092203456d56960a1b4d8777" + } + } + } + }, "qemu": { "release": "33.20210412.3.0", "formats": { diff --git a/metrics/README.md b/metrics/README.md new file mode 100644 index 0000000..24a4941 --- /dev/null +++ b/metrics/README.md @@ -0,0 +1,2 @@ + +See [README.md](../README.md#metrics). diff --git a/metrics/fcos-sqlitevis.json b/metrics/fcos-sqlitevis.json new file mode 100644 index 0000000..1c3c311 --- /dev/null +++ b/metrics/fcos-sqlitevis.json @@ -0,0 +1,711 @@ +{ + "version": 2, + "inquiries": [ + { + "id": "WUPD4gZdu-j4mFgxjHG0P", + "query": "SELECT os_variant FROM countme_totals \n WHERE weeknum = (SELECT MAX(weeknum) FROM countme_totals)\n AND os_variant REGEXP ''\n GROUP BY os_variant;", + "viewType": "chart", + "viewOptions": { + "data": [], + "layout": { + "autosize": true, + "xaxis": { + "range": [ + -1, + 6 + ], + "autorange": true + }, + "yaxis": { + "range": [ + -1, + 4 + ], + "autorange": true + } + }, + "frames": [] + }, + "name": "All OS Variants", + "createdAt": "2025-05-12T20:54:27.120Z" + }, + { + "id": "tcIRiJz5gn5ci4DJyHgqU", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, weeknum, os_variant, repo_arch, SUM(hits) FROM countme_totals \n WHERE os_variant IS 'coreos'\n AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' \n AND weeknum = (SELECT MAX(weeknum) FROM countme_totals)\n GROUP BY repo_arch;", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "pie", + "mode": "markers", + "values": null, + "valuessrc": "SUM(hits)", + "meta": { + "columnNames": { + "values": "SUM(hits)", + "labels": "repo_arch", + "text": "" + } + }, + "labels": null, + "labelssrc": "repo_arch", + "opacity": 1, + "textinfo": "label+value+percent", + "textfont": { + "size": 26, + "family": "sans-serif" + }, + "hoverinfo": "percent+label+value", + "hoverlabel": { + "align": "auto" + }, + "direction": "counterclockwise", + "rotation": 0, + "hole": 0.52, + "pull": 0, + "marker": { + "line": { + "width": 1 + } + }, + "insidetextorientation": "radial" + } + ], + "layout": { + "xaxis": { + "range": [ + -1, + 6 + ], + "autorange": true + }, + "yaxis": { + "range": [ + -1, + 4 + ], + "autorange": true + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "title": { + "text": "Fedora CoreOS Node Architecture Breakdown Week of 2025-05-04", + "font": { + "size": 25 + } + }, + "hiddenlabels": [ + "ppc64le", + "s390x" + ], + "legend": { + "x": 0.7407924239291469, + "y": 0.8257272143643333, + "font": { + "size": 20 + }, + "yanchor": "middle" + }, + "annotations": [], + "meta": [ + "2023-10-08", + "2023-10-08", + "2023-10-08", + "2023-10-08" + ], + "metasrc": "date", + "extendpiecolors": true + }, + "frames": [] + }, + "name": "FCOS Architectures", + "createdAt": "2025-05-12T20:55:12.622Z" + }, + { + "id": "wEj338NrufIRE-3UBDXPK", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, weeknum, SUM(transient_hits), SUM(static_hits), SUM(transient_hits + static_hits) FROM (\n SELECT weeknum, SUM(hits) AS transient_hits, 0 AS static_hits FROM countme_totals WHERE os_variant IS 'coreos' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age = 1 GROUP BY weeknum\n UNION\n SELECT weeknum, 0 AS transient_hits, SUM(hits) AS static_hits FROM countme_totals WHERE os_variant IS 'coreos' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age > 1 GROUP BY weeknum\n) WHERE date > '2020-01-01' GROUP BY weeknum", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "scatter", + "mode": "lines", + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(static_hits)" + } + }, + "y": null, + "ysrc": "SUM(static_hits)", + "stackgroup": 1, + "name": "Static Nodes", + "hoveron": "points" + }, + { + "type": "scatter", + "mode": "lines", + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(transient_hits)" + } + }, + "y": null, + "ysrc": "SUM(transient_hits)", + "stackgroup": 1, + "name": "Transient Nodes", + "fillcolor": "rgba(205, 96, 52, 0.5)", + "line": { + "color": "rgb(180, 38, 5)" + } + }, + { + "type": "scatter", + "mode": "lines", + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(transient_hits + static_hits)", + "text": "" + } + }, + "y": null, + "ysrc": "SUM(transient_hits + static_hits)", + "name": "Total Nodes", + "line": { + "width": 3, + "color": "rgb(95, 100, 96)" + }, + "hovertemplate": "", + "error_x": { + "_template": null, + "visible": false, + "type": "percent", + "symmetric": true, + "value": 10, + "color": "rgb(95, 100, 96)", + "thickness": 2, + "width": 4 + } + } + ], + "layout": { + "xaxis": { + "range": [ + "2021-09-19 23:38:29.3717", + "2025-05-22 01:49:05.1712" + ], + "autorange": false, + "rangeselector": { + "visible": false, + "buttons": [ + {} + ] + }, + "showspikes": false, + "rangeslider": { + "visible": false, + "yaxis": {}, + "autorange": true, + "range": [ + "2020-05-03", + "2023-11-28 23:10:02.9513" + ] + }, + "type": "date", + "tickfont": { + "size": 28 + }, + "title": { + "font": { + "size": 17 + } + } + }, + "yaxis": { + "range": [ + -1419.2166576673771, + 123989.902612791 + ], + "autorange": false, + "ticks": "", + "showspikes": false, + "showline": false, + "zeroline": true, + "type": "linear", + "tickfont": { + "size": 28 + } + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "dragmode": "pan", + "title": { + "text": "Fedora CoreOS Node Count", + "font": { + "size": 33 + } + }, + "legend": { + "font": { + "size": 28 + }, + "orientation": "h", + "x": 0.2471859552265083, + "y": 0.9623782823483056 + } + }, + "frames": [] + }, + "name": "FCOS Node Count", + "createdAt": "2025-05-12T20:55:20.874Z" + }, + { + "id": "rcnNNlpFRfhIaX27ownZa", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, upper(trim(repo_tag, 'updates-releaseed-')) as repo_tag, os_variant, SUM(hits) FROM countme_totals\n WHERE os_variant IS 'coreos'\n AND repo_tag REGEXP 'updates-released-f[3-4][0-9]'\n AND weeknum = (SELECT MAX(weeknum) FROM countme_totals)\n GROUP BY repo_tag;", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "pie", + "mode": "markers", + "values": null, + "valuessrc": "SUM(hits)", + "meta": { + "columnNames": { + "values": "SUM(hits)", + "labels": "repo_tag" + } + }, + "labels": null, + "labelssrc": "repo_tag", + "hole": 0.5, + "pull": 0, + "marker": { + "line": { + "width": 2 + } + }, + "textinfo": "label", + "textfont": { + "size": 25 + }, + "sort": false, + "direction": "clockwise", + "rotation": -90, + "legendgroup": 1, + "showlegend": true, + "hoverinfo": "percent+label+value", + "opacity": 1, + "textposition": "inside" + } + ], + "layout": { + "xaxis": { + "range": [ + -1, + 6 + ], + "autorange": true + }, + "yaxis": { + "range": [ + -1, + 4 + ], + "autorange": true + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "title": { + "text": "Fedora CoreOS Release Breakdown", + "x": 0.5, + "font": { + "size": 31 + } + }, + "showlegend": true, + "legend": { + "font": { + "family": "monospace", + "size": 22 + }, + "title": { + "text": "
", + "font": { + "size": 34 + } + }, + "y": 0.04329087951849141, + "x": 0.20084040421902638, + "yanchor": "bottom", + "orientation": "v" + }, + "hiddenlabels": [], + "hoverlabel": { + "align": "auto" + }, + "uniformtext": { + "mode": false + }, + "modebar": { + "orientation": "h" + }, + "margin": { + "pad": 0, + "r": 80 + }, + "extendpiecolors": true, + "piecolorway": [ + "#1b9e77", + "#d95f02", + "#7570b3", + "#e7298a", + "#66a61e", + "#e6ab02", + "#a6761d", + "#666666" + ] + }, + "frames": [] + }, + "name": "FCOS Release Breakdown", + "createdAt": "2025-05-12T20:55:31.761Z" + }, + { + "id": "BZlflOgPAYGoaKwjBpIe8", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, weeknum, SUM(coreos_hits), SUM(cloud_hits), SUM(server_hits) FROM (\n SELECT weeknum, 0 AS server_hits, 0 AS cloud_hits, SUM(hits) AS coreos_hits FROM countme_totals WHERE os_variant IS 'coreos' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age > 1 GROUP BY weeknum\n UNION\n SELECT weeknum, 0 AS server_hits, SUM(hits) AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'cloud' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age > 1 GROUP BY weeknum\n UNION\n SELECT weeknum, SUM(hits) AS server_hits, 0 AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'server' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age > 1 GROUP BY weeknum\n) WHERE date > '2022-01-01' GROUP BY weeknum", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "scatter", + "mode": "lines", + "stackgroup": null, + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(cloud_hits)" + } + }, + "y": null, + "ysrc": "SUM(cloud_hits)", + "name": "Cloud", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(coreos_hits)" + } + }, + "y": null, + "ysrc": "SUM(coreos_hits)", + "name": "CoreOS", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "stackgroup": null, + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(server_hits)" + } + }, + "y": null, + "ysrc": "SUM(server_hits)", + "name": "Server", + "line": { + "width": 5 + } + } + ], + "layout": { + "xaxis": { + "range": [ + "2023-05-18 21:13:31.5607", + "2025-05-04" + ], + "autorange": false, + "type": "date", + "tickfont": { + "size": 22 + } + }, + "yaxis": { + "range": [ + -4149.444444444446, + 85743.15981948335 + ], + "autorange": false, + "type": "linear", + "tickfont": { + "size": 22 + } + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "title": { + "text": "Static Node Count for Fedora Cloud/CoreOS/Server" + }, + "dragmode": "zoom", + "legend": { + "font": { + "size": 28 + }, + "orientation": "h", + "x": 0.4185161699429296, + "y": 0.988780487804878 + } + }, + "frames": [] + }, + "name": "Static Node Count By Edition", + "createdAt": "2025-05-12T20:56:19.303Z" + }, + { + "id": "r6KJ-g1sxjbqtOYuWRJoK", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, weeknum, SUM(coreos_hits), SUM(cloud_hits), SUM(server_hits) FROM (\n SELECT weeknum, 0 AS server_hits, 0 AS cloud_hits, SUM(hits) AS coreos_hits FROM countme_totals WHERE os_variant IS 'coreos' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age = 1 GROUP BY weeknum\n UNION\n SELECT weeknum, 0 AS server_hits, SUM(hits) AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'cloud' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age = 1 GROUP BY weeknum\n UNION\n SELECT weeknum, SUM(hits) AS server_hits, 0 AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'server' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' AND sys_age = 1 GROUP BY weeknum\n) WHERE date > '2022-01-01' GROUP BY weeknum", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "scatter", + "mode": "lines", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(cloud_hits)" + } + }, + "x": null, + "xsrc": "date", + "name": "Cloud", + "y": null, + "ysrc": "SUM(cloud_hits)", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(coreos_hits)" + } + }, + "x": null, + "xsrc": "date", + "y": null, + "ysrc": "SUM(coreos_hits)", + "name": "CoreOS", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "stackgroup": null, + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(server_hits)" + } + }, + "y": null, + "ysrc": "SUM(server_hits)", + "x": null, + "xsrc": "date", + "name": "Server", + "line": { + "width": 5 + } + } + ], + "layout": { + "xaxis": { + "range": [ + "2023-05-14 20:40:13.8728", + "2025-05-04" + ], + "autorange": false, + "type": "date", + "tickfont": { + "size": 22 + } + }, + "yaxis": { + "range": [ + -7660.833333333334, + 155839.15818339647 + ], + "autorange": false, + "type": "linear", + "tickfont": { + "size": 22 + } + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "title": { + "text": "Transient Node Count for Fedora Cloud/CoreOS/Server" + }, + "dragmode": "zoom", + "legend": { + "font": { + "size": 28 + }, + "orientation": "h", + "x": 0.388712745719721, + "y": 1.0030674846625767 + } + }, + "frames": [] + }, + "name": "Transient Node Count By Edition", + "createdAt": "2025-05-12T20:56:34.411Z" + }, + { + "id": "8C93FoFqg3Zpw4wcxyS2e", + "query": "SELECT date(julianday('1970-01-05')+weeknum*7 + 6) AS date, weeknum, SUM(coreos_hits), SUM(cloud_hits), SUM(server_hits) FROM (\n SELECT weeknum, 0 AS server_hits, 0 AS cloud_hits, SUM(hits) AS coreos_hits FROM countme_totals WHERE os_variant IS 'coreos' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' GROUP BY weeknum\n UNION\n SELECT weeknum, 0 AS server_hits, SUM(hits) AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'cloud' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' GROUP BY weeknum\n UNION\n SELECT weeknum, SUM(hits) AS server_hits, 0 AS cloud_hits, 0 AS coreos_hits FROM countme_totals WHERE os_variant IS 'server' AND repo_tag REGEXP 'updates-released-f[3-4][0-9]' GROUP BY weeknum\n) WHERE date > '2022-01-01' GROUP BY weeknum", + "viewType": "chart", + "viewOptions": { + "data": [ + { + "type": "scatter", + "mode": "lines", + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(cloud_hits)" + } + }, + "y": null, + "ysrc": "SUM(cloud_hits)", + "name": "Cloud", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "stackgroup": null, + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(coreos_hits)" + } + }, + "y": null, + "ysrc": "SUM(coreos_hits)", + "name": "CoreOS", + "line": { + "width": 5 + } + }, + { + "type": "scatter", + "mode": "lines", + "stackgroup": null, + "x": null, + "xsrc": "date", + "meta": { + "columnNames": { + "x": "date", + "y": "SUM(server_hits)" + } + }, + "y": null, + "ysrc": "SUM(server_hits)", + "name": "Server", + "line": { + "width": 5 + } + } + ], + "layout": { + "xaxis": { + "range": [ + "2023-05-17 21:05:12.1387", + "2025-05-04" + ], + "autorange": false, + "type": "date", + "tickfont": { + "size": 22 + } + }, + "yaxis": { + "range": [ + -10802.38888888889, + 229205.70535714284 + ], + "autorange": false, + "type": "linear", + "tickfont": { + "size": 22 + } + }, + "autosize": true, + "mapbox": { + "style": "open-street-map" + }, + "title": { + "text": "Total Node Count for Fedora Cloud/CoreOS/Server" + }, + "legend": { + "orientation": "h", + "x": 0.4578313253012048, + "y": 0.9863986313088109, + "font": { + "size": 28 + } + } + }, + "frames": [] + }, + "name": "Total Node Count for Fedora Cloud/CoreOS/Server", + "createdAt": "2025-05-12T20:56:53.783Z" + } + ] +} \ No newline at end of file diff --git a/stream-tooling.md b/stream-tooling.md index c0e1a06..f8455b8 100644 --- a/stream-tooling.md +++ b/stream-tooling.md @@ -13,8 +13,6 @@ FCOS will have multiple streams: | Development | next-devel | annex | | Mechanical | rawhide | annex | | Mechanical | branched | annex | -| Mechanical | bodhi-updates | annex | -| Mechanical | bodhi-updates-testing | annex | Development and mechanical streams are subject to change. @@ -32,8 +30,6 @@ We need a way to both (1) fix the content set for a particular stream release, a **Mechanical** streams are not curated; they're automated nightly snapshots of the underlying repos. They source their RPMs from the regular Fedora repos (using 30 here to mean `$currentrelease`): 1. **rawhide** <- f32 2. **branched** <- f31 when a branch exists, otherwise tracks **rawhide** -3. **bodhi-updates** <- f30-stable + f30-updates -4. **bodhi-updates-testing** <- f30-stable + f30-updates + f30-updates-testing **Production** streams are intended for production use. They source their RPMs from a _single_ Koji tag, `coreos-pool`, from which we create a yum repo: 1. **next** <- coreos-pool @@ -52,7 +48,7 @@ There is also a second Koji tag, `coreos-release`, for packages which have been We maintain a git repository containing the rpm-ostree treefile and lockfiles. This could be [fedora-coreos-config](https://github.com/coreos/fedora-coreos-config). We have one branch for each stream, and no main branch. -For the mechanical streams, a nightly job will run the compose from the corresponding yum repos and SCM refs. This job will output a lockfile for each CPU architecture. Those lockfiles will be committed to Git to preserve a record of the build's contents, and the builds will be pushed to the corresponding ostree refs. The {bodhi-updates, branched} lockfile will also be PR'd to the {testing-devel, next-devel} branch, the latter only during the part of the cycle where next-devel is maintained. We want to keep the development branches ready to release, so those PRs are not merged unless green. +For the mechanical streams, a nightly job will run the compose from the corresponding yum repos and SCM refs. This job will output a lockfile for each CPU architecture. Those lockfiles will be committed to Git to preserve a record of the build's contents, and the builds will be pushed to the corresponding ostree refs. The branched lockfile will also be PR'd to the {testing-devel, next-devel} branch, the latter only during the part of the cycle where next-devel is maintained. We want to keep the development branches ready to release, so those PRs are not merged unless green. The lockfiles produced from the automatic snapshot will never be hand-modified, and in the next/testing/stable branches will never be modified at all except during promotions. Instead, pins (to older NEVRAs) and updates (to newer ones) will be hand-maintained in the Git branches in a separate lockfile that overrides the autogenerated ones. These overrides will be the major distinction between the mechanical refs and the "curated" (development/production) refs. Each curated branch will have one override file, which can carry both CPU-architecture-independent and architecture-specific overrides. @@ -74,7 +70,7 @@ Update the development treefile as usual. On the next bot push, the lockfile wil To focus development effort, there will be one base treefile shared across all branches, whose canonical copy will live in the testing-devel branch. Changes will automatically be mirrored to next-devel and to the mechanical branches. To address divergence across Fedora releases, each branch will also have an overlay treefile (possibly empty): -- **testing-devel** -> automatically mirrored to bodhi-updates and bodhi-updates-testing +- **testing-devel** - **next-devel** -> automatically mirrored to branched - **rawhide**