diff --git a/docs/issues/drafts/1840-workflow-performance-alternative-linker/ISSUE.md b/docs/issues/drafts/1840-workflow-performance-alternative-linker/ISSUE.md new file mode 100644 index 000000000..70830b2fb --- /dev/null +++ b/docs/issues/drafts/1840-workflow-performance-alternative-linker/ISSUE.md @@ -0,0 +1,254 @@ +--- +doc-type: issue +issue-type: task +status: draft +priority: p1 +github-issue: null +spec-path: docs/issues/drafts/1840-workflow-performance-alternative-linker/ISSUE.md +branch: "{issue-number}-alternative-linker" +related-pr: null +last-updated-utc: 2026-06-01 00:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - Containerfile + - .cargo/config.toml + - .github/workflows/container.yaml + - .github/workflows/testing.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md +--- + + + +# Issue #[To be assigned] - Switch to a faster linker (mold or lld) to reduce link time + +## Goal + +Replace the default GNU BFD linker with a faster alternative — `mold` or `lld` +— in both the local development build and the Containerfile build stages, to +reduce the dominant per-binary link time recorded in the baseline report. + +## Background + +### The baseline finding + +The baseline profiling report +(`docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md`) +identified the build as **linker-dominated**: + +> "Individual crate compilation (frontend + codegen): ≤ 8 s per crate. +> Binary/test target linking: 35–117 s per binary — an order of magnitude +> more than any single crate compilation." + +All 20+ binary and test targets compiled by the Containerfile's +`cargo nextest archive --all-targets` show `sections: null` in the +`cargo --timings` output — the signature of a pure external linker invocation. + +The top offenders (release, warm incremental): + +| Binary / target | Link time (s) | +| ---------------------------------------------------------- | ------------- | +| `torrust-tracker` integration test | 117 | +| `torrust-tracker` bin | 117 | +| `torrust-tracker` profiling bin | 116 | +| `torrust-tracker-axum-health-check-api-server` integration | 109 | +| `torrust-tracker-core` persistence bench bin | 104 | +| … (15+ more in the 35–94 s range) | … | + +The baseline report explicitly recommends: + +> "Switching to a faster linker (e.g. `mold` or `lld`) or removing +> non-runtime binary targets from the build are the two highest-leverage +> optimisations." + +The current linker is the system default: GNU BFD via `cc` (confirmed in the +baseline measurement environment table: "system default (`cc` / BFD linker; no +`mold` or `lld`)"). + +### Local timing experiment (2026-06-01) + +A fair incremental relink benchmark was run locally (Ryzen 9 7950X, debug +profile, `--bin torrust-tracker` only, `touch src/lib.rs` to force a recompile +of the top-level crate, 2026-06-01). + +The linker was switched using `mold --run`, which intercepts `ld` via +`LD_PRELOAD` without changing `RUSTFLAGS` — so cargo's incremental cache +fingerprint is identical for both runs, ensuring only the top-level crate is +recompiled in each case. mold was confirmed active via `readelf -p .comment` +(`.comment` section showed `mold 2.40.4 (compatible with GNU ld)`). + +| Linker | Real time | User time | Sys time | Notes | +| -------------------------- | --------- | --------- | -------- | -------------------------------------- | +| BFD (default) | 54.1 s | 53.3 s | 2.3 s | `touch src/lib.rs && time cargo build` | +| mold 2.40.4 (`mold --run`) | 54.1 s | 53.0 s | 2.1 s | same RUSTFLAGS, LD_PRELOAD intercept | + +**Interpretation**: both runs are strictly equivalent (same compilation units, +same RUSTFLAGS). The results are identical — **compilation of `lib.rs` dominates +at ~52 s (user time), masking the link time difference in a single-crate +incremental rebuild**. mold's parallelism advantage only becomes visible when +the link step is a significant fraction of total build time. + +For a single incremental rebuild, the link time is approximately 2–3 s (total +54 s minus ~52 s compilation). mold compresses such a link from ~2–3 s to +sub-second, which is invisible in wall-clock terms here. + +The real benefit is in **cold builds** (like CI / Containerfile), where 20+ +binaries are linked fresh with no incremental cache. At BFD link times of 35–117 +s per binary (baseline), and mold's documented speedup of 10–31× over BFD +(MySQL: 10.84 s → 0.46 s; Clang: 42.07 s → 1.35 s; source: +[mold README](https://github.com/rui314/mold)), the container build would save +hundreds of seconds. + +> Note: the debug-profile results above represent the worst case for mold (link +> time already small). Release-profile and `--all-targets` cold builds are where +> mold delivers its full benefit. + +### Linker options considered + +The available alternatives to BFD were evaluated before choosing mold as the +primary candidate: + +| Linker | MySQL 8.3 | Clang 19 | Chromium 124 | Notes | +| ------------ | ---------- | ---------- | ------------ | ---------------------------------------------- | +| BFD (GNU ld) | 10.84 s | 42.07 s | N/A | Current default; single-threaded | +| gold (GNU) | 7.47 s | 33.13 s | 27.40 s | Linux only; deprecated upstream | +| lld (LLVM) | 1.64 s | 5.20 s | 6.10 s | Linux + macOS; ~4× faster than BFD | +| **mold** | **0.46 s** | **1.35 s** | **1.52 s** | Linux only; most parallel; ~4× faster than lld | + +Source: [mold README benchmarks](https://github.com/rui314/mold) + +**Decision: pursue mold only.** It is the clear performance winner — ~4× faster +than lld and ~23× faster than BFD. There is no performance case for lld or gold. + +The only reason to fall back to lld is **compatibility**: if mold fails to link +one of the C library dependencies (`aws-lc-sys`/BoringSSL is the known risk). +That path is covered by T8. lld is not benchmarked proactively; it is only +reached if mold is ruled out on correctness grounds. + +**gold** is not considered: it is slower than lld and deprecated upstream. + +**[wild](https://github.com/davidlattimore/wild)** (a new experimental +Rust-written linker optimized for incremental linking) is not considered: it is +too experimental for a production CI pipeline at this time. + +- **mold** (): a modern, highly parallel linker + designed as a drop-in replacement for GNU `ld` and `gold`. Available in + Ubuntu apt (`mold` package, v2.40.4 on Ubuntu 26.04). Linux-only. +- **lld** (): the LLVM project linker. Available on Linux + and macOS (`llvm-dev` or `lld` package on Ubuntu). Fallback only. + +### Scope considerations + +- **Local development**: changing `.cargo/config.toml` affects all contributors. + macOS contributors cannot use `mold` (Linux-only); they need `lld` or the + system default. Using `[target.'cfg(target_os = "linux")']` (the approach + recommended by mold's own docs) scopes the setting to Linux only and avoids + breaking macOS contributors. Example (mold in `$PATH`, GCC 12+): + + ```toml + [target.'cfg(target_os = "linux")'] + rustflags = ["-C", "link-arg=-fuse-ld=mold"] + ``` + + For older GCC or to be explicit, add `linker = "clang"` and point to the + mold executable path (`-fuse-ld=/usr/bin/mold`). + +- **Containerfile (CI)**: the Docker builder image (`chef` stage) runs on + Linux x86_64, so `mold` is the natural choice. `mold` needs to be installed + in the builder stage (`apt-get install -y mold`) and will be picked up + automatically via the `.cargo/config.toml` setting above. +- **cargo-chef cook stages**: the `dependencies` and `dependencies_debug` stages + compile external crates (no final link step for the cook stage itself — + `cargo chef cook` produces `.rlib` files, not binaries). The linker is only + invoked in the `build` and `build_debug` stages for the final binary and test + targets. The cook stages are unaffected by this change. + +## Scope + +### In scope + +- Benchmark `mold` vs BFD for the relink-only case (single binary, debug and + release profile) on the local developer machine. +- Benchmark `mold` vs BFD inside Docker (`build` and `build_debug` stages) for + the full `--all-targets` case to measure end-to-end impact on container build + time. +- If `mold` shows meaningful speedup, add it to the `chef` Docker stage and + configure it as the linker for `x86_64-unknown-linux-gnu` builds via + `.cargo/config.toml` (target-specific block to avoid breaking macOS + contributors). +- Update the baseline benchmark report with new timing numbers. + +### Out of scope + +- Changing the linker for macOS developer machines (separate concern; `lld` or + `zld` can be a follow-up if there is interest). +- Changing the linker for the `cargo test --doc` or `linter` steps (those do + not produce standalone binaries; linker swap has minimal effect). +- Evaluating `lld` unless `mold` proves unsuitable (e.g. linking errors with + specific C libraries such as `aws-lc-sys`). + +## Implementation Plan + +| Task ID | Description | Status | +| ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | +| T1 | Run relink benchmark locally: `touch src/lib.rs && time cargo build --bin torrust-tracker` vs `mold --run cargo build --bin torrust-tracker` (debug and release) | DONE | +| T2 | Run `--all-targets` benchmark locally with `mold`: `mold -run cargo build --timings --all-targets --release` and compare total wall time and per-binary times with baseline | TODO | +| T3 | Test that `mold` produces a working binary: run `cargo test --workspace` and the integration test suite with mold active | TODO | +| T4 | Verify `mold` links correctly with C dependencies (`libsqlite3-sys`, `aws-lc-sys`, `zstd-sys`, `ring`): check for linker errors or runtime failures | TODO | +| T5 | Add `mold` installation to the `chef` stage of the Containerfile: `apt-get install -y mold` | TODO | +| T6 | Add a `[target.x86_64-unknown-linux-gnu]` section to `.cargo/config.toml` pointing to `mold` as linker | TODO | +| T7 | Re-run the container cold benchmark with mold enabled and record new timings in the baseline report | TODO | +| T8 | If mold causes issues with any C library (aws-lc-sys is a known risk), evaluate `lld` as an alternative | TODO | + +## Progress Tracking + +### Checklist + +- [x] T1 — relink benchmark (local, single binary, debug) — **done**: BFD 54.1s = mold 54.1s; compile dominates; pure link time immeasurable via wall clock in incremental mode (see Background) +- [ ] T2 — `--all-targets` timings benchmark (local, mold vs BFD) +- [ ] T3 — correctness: full test suite passes with mold +- [ ] T4 — C library linking verified: `libsqlite3-sys`, `aws-lc-sys`, `zstd-sys` +- [ ] T5 — mold added to `chef` Containerfile stage +- [ ] T6 — `.cargo/config.toml` updated with `[target.x86_64-unknown-linux-gnu]` +- [ ] T7 — container cold benchmark re-run and baseline report updated +- [ ] T8 — lld evaluated as fallback if mold fails on any C library + +### Progress Log + +Append one line per meaningful update. + +- 2026-06-01 00:00 UTC - GitHub Copilot - Drafted sub-issue spec for alternative linker evaluation. Baseline data shows 35–117 s link time per binary (BFD). +- 2026-06-01 13:00 UTC - GitHub Copilot - Ran fair incremental relink benchmark using `mold --run` (LD_PRELOAD intercept, identical RUSTFLAGS). Result: BFD 54.1s = mold 54.1s — compile dominates (~52s user time) in single-crate incremental builds, masking the link time difference. Verified mold was active via `readelf -p .comment`. Updated spec with mold's official benchmarks (10–31× faster than BFD in cold builds) as the primary evidence for the container build savings. + +## Acceptance Criteria + +- [ ] AC1 — A relink benchmark comparing BFD vs mold has been run and recorded (debug and release profile, single binary and `--all-targets`). +- [ ] AC2 — `cargo test --workspace` passes with mold active (no correctness regressions). +- [ ] AC3 — C library dependencies (`aws-lc-sys`, `libsqlite3-sys`, `zstd-sys`) link correctly with mold. +- [ ] AC4 — If mold shows meaningful speedup (>20 %), it is enabled in `.cargo/config.toml` for `x86_64-unknown-linux-gnu` and in the `chef` Containerfile stage. +- [ ] AC5 — The container cold build benchmark is re-run with mold and new timings are recorded in the baseline report. + +### Acceptance Verification + +| AC ID | Status (`TODO`/`DONE`) | Evidence | +| ----- | ---------------------- | -------- | +| AC1 | TODO | | +| AC2 | TODO | | +| AC3 | TODO | | +| AC4 | TODO | | +| AC5 | TODO | | + +## Risks and Trade-offs + +- **Risk**: `mold` may not support all linker flags or section layouts expected + by `aws-lc-sys` (BoringSSL). Mitigation: T4 and T8 — verify with C library + tests before enabling globally; fall back to `lld` if needed. +- **Risk**: Changing `.cargo/config.toml` to use `mold` will break builds on + macOS (where `mold` is not available). Mitigation: use a + `[target.x86_64-unknown-linux-gnu]` section, not a global `[build]` section. +- **Trade-off**: `mold` is Linux-only; macOS contributors would not benefit from + this change locally. A separate follow-up could configure `lld` for macOS. +- **Trade-off**: Installing `mold` adds ~4 MB to the Docker builder image layer. + This is negligible relative to the build time saved. diff --git a/docs/issues/drafts/1840-workflow-performance-buildkit-cargo-cache-mounts/ISSUE.md b/docs/issues/drafts/1840-workflow-performance-buildkit-cargo-cache-mounts/ISSUE.md new file mode 100644 index 000000000..e4be0269b --- /dev/null +++ b/docs/issues/drafts/1840-workflow-performance-buildkit-cargo-cache-mounts/ISSUE.md @@ -0,0 +1,219 @@ +--- +doc-type: issue +issue-type: task +status: draft +priority: p2 +github-issue: null +spec-path: docs/issues/drafts/1840-workflow-performance-buildkit-cargo-cache-mounts/ISSUE.md +branch: "{issue-number}-buildkit-cargo-cache-mounts" +related-pr: null +last-updated-utc: 2026-06-01 00:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - Containerfile + - .github/workflows/container.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md + - docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md +--- + + + +# Issue #[To be assigned] - Pass Cargo registry/git caches into BuildKit to speed up cook stage rebuilds + +## Goal + +Add `--mount=type=cache` directives to the `cargo chef cook` RUN steps in the +Containerfile so that the Cargo registry and git caches survive across cook +layer invalidations on local developer machines. Evaluate whether the same +benefit can be extended to CI ephemeral runners. + +## Background + +### The cook stage bottleneck + +The `dependencies` and `dependencies_debug` stages (cook stages) compile all +external Rust crates and are the most expensive part of the container build. +The cook layer is invalidated — and all external crates recompiled from scratch +— whenever `Cargo.lock` changes. + +The cook RUN step has two sub-phases: + +1. **Download**: fetch crate sources from `crates.io` into the Cargo registry + (`/usr/local/cargo/registry` and `/usr/local/cargo/git`). +2. **Compile**: compile all external crates and place artifacts in + `/build/src/target`. + +### Proposed change + +Add BuildKit cache mounts to the cook RUN steps: + +```dockerfile +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ + cargo chef cook --tests --benches --examples --workspace \ + --all-targets --all-features --recipe-path /build/recipe.json +``` + +This tells BuildKit to overlay the named cache volumes over the registry and +git paths during the RUN step. On a local machine with a long-lived Docker +daemon, the volumes persist between builds. When the cook layer is invalidated +(e.g. `Cargo.lock` changes), the crates are already in the registry cache and +do not need to be re-downloaded. + +### Local benchmark: registry download time (2026-06-01) + +To quantify the download-only saving, `cargo fetch` was run against a fresh +`CARGO_HOME` (simulating an empty registry cache) and then again against the +populated registry. Machine: Ryzen 9 7950X. + +| State | Command | Time | +| ------------------------- | ----------------------------------- | ------ | +| Cold (empty registry) | `CARGO_HOME=/tmp/fresh cargo fetch` | 6.9 s | +| Warm (registry populated) | `CARGO_HOME=/tmp/fresh cargo fetch` | 0.16 s | + +Registry cache size after cold fetch: **823 MB**. + +Interpretation: registry cache mounts save approximately **7 s** per cook layer +rebuild (the download phase). The compile phase (the dominant cost in the cook +stage) is **not affected** — compiled artifacts are not included in the registry +or git cache volumes. + +### Critical limitation: ephemeral CI runners + +`--mount=type=cache` volumes are managed by the local BuildKit daemon and are +stored in the daemon's cache directory (e.g. `/var/lib/docker/buildkit/`). They +are **not** included in the BuildKit layer cache exported via +`cache-from/cache-to: type=gha`. + +The current CI workflow (`container.yaml`) uses: + +```yaml +cache-from: type=gha,scope=container- +cache-to: type=gha,scope=container-,mode=max +``` + +`type=gha` exports and restores Docker image layer blobs. It does **not** +persist `--mount=type=cache` volumes. Each GitHub Actions job starts a fresh +ephemeral runner with a new Docker daemon, so the registry cache mount is always +empty. + +Conclusion for CI: + +- If the cook layer **is** in the GHA layer cache (no `Cargo.lock` change): + the cook stage is skipped entirely; cache mounts have no effect. +- If the cook layer **is not** in the GHA layer cache (`Cargo.lock` changed): + the cook stage runs on a fresh daemon; cache mounts are empty; downloads and + compiles from scratch. + +**Registry/git cache mounts provide zero benefit to CI with GitHub Actions +ephemeral runners in the current setup.** + +The benefit is limited to local development builds where the Docker daemon is +long-lived (e.g. `docker build` run repeatedly on a developer machine). + +### Paths to CI benefit + +For the cache mounts to help in CI, one of the following would be required: + +| Option | Complexity | Notes | +| ------------------------------- | ---------- | ------------------------------------------------------------------------------------- | +| Self-hosted runner | Medium | Persistent Docker daemon; cache mounts survive across jobs | +| Depot / Namespace / similar CI | Low-Medium | Persistent BuildKit daemons as a service; cache mounts persist | +| `actions/cache` + volume export | High | Manually tar/restore the BuildKit cache mount dir between runs; fragile, non-standard | + +### Advanced variant: caching compiled artifacts + +A more aggressive approach would add a cache mount for the target directory +(`/build/src/target`) in addition to the registry/git mounts: + +```dockerfile +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ + --mount=type=cache,target=/build/src/target \ + cargo chef cook ... +``` + +If the target cache mount is populated, cargo performs **incremental +compilation** — only changed or new crates are recompiled. For a minor +`Cargo.lock` change (one or two crates updated), this could reduce the cook +rebuild from 20+ minutes to a few minutes. + +However, there is a structural incompatibility with cargo-chef's +cook-then-build layer split: + +- The cook stage's target directory (compiled artifacts) is IN the Docker layer + when no cache mount is used. Downstream stages (`FROM dependencies_debug AS +build_debug`) inherit these artifacts. +- When a `--mount=type=cache` is applied to the target path, the compiled + artifacts live in the cache volume — they are **not** part of the resulting + layer. Downstream stages see an empty target directory and must recompile + everything. + +Workarounds are possible but complex (e.g. copying artifacts out of the cache +mount before the RUN step ends, or restructuring the build to avoid the +cook/build layer split). These are tracked as a separate evaluation (see T5). + +The same CI limitation applies: target cache mounts are also ephemeral on +GitHub Actions runners. + +## Scope + +### In scope + +- Add `--mount=type=cache` for registry and git to both cook stages in the + Containerfile. +- Verify the change does not break local builds or produce different artifacts. +- Document the CI limitation clearly in the implementation notes. +- Measure the actual improvement on local builds by timing a cook layer rebuild + with and without cache mounts. +- Evaluate whether the target-dir cache mount variant is feasible (T5). + +### Out of scope + +- Switching to a self-hosted runner or a paid BuildKit service. +- Caching the target directory without a clear design that preserves the + downstream stage compatibility. +- CI cache persistence via `actions/cache` volume export (too fragile). + +## Implementation Plan + +| Task ID | Description | Status | +| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | +| T1 | Add `--mount=type=cache,target=/usr/local/cargo/registry` and `git` to both cook stages in the Containerfile | TODO | +| T2 | Run a cook layer rebuild locally (trigger by modifying `Cargo.lock` or bumping a dep version) with and without cache mounts and record wall-clock time difference | TODO | +| T3 | Verify that the resulting archives produce identical test results (`cargo nextest run` passes) with cache mounts enabled | TODO | +| T4 | Document the CI limitation (cache mounts are ephemeral on GitHub Actions) in a comment inside the Containerfile and in this spec | TODO | +| T5 | Evaluate the target-dir cache mount variant: prototype a Containerfile that uses `--mount=type=cache,target=/build/src/target` and assess whether downstream stage compatibility is solvable | TODO | +| T6 | Update the baseline benchmark report with new local timing numbers | TODO | + +## Risks and Trade-offs + +| Risk | Likelihood | Mitigation | +| ------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------- | +| Cache mount causes stale artifacts (wrong crate versions compiled) | Low | Cache is keyed by daemon lifetime; a fresh build always starts clean; `--no-cache` forces cold rebuild if needed | +| CI engineers expect CI improvement and are disappointed | Medium | Document CI limitation clearly before merging; set correct expectations in PR description | +| Target-dir cache mount breaks downstream stages | High | Keep target-dir approach in T5 (prototype-only); do not merge until downstream compatibility is solved | +| BuildKit syntax line (`# syntax=docker/dockerfile:latest`) required | Low | Already present in the Containerfile; required for cache mount support | + +## Progress Tracking + +### Checklist + +- [x] T0 — proxy benchmark: cold `cargo fetch` 6.9 s, warm 0.16 s; registry 823 MB; CI limitation documented +- [ ] T1 — registry/git cache mounts added to both cook stages +- [ ] T2 — cook layer rebuild timed with and without cache mounts +- [ ] T3 — correctness: test suite passes with cache mounts enabled +- [ ] T4 — CI limitation documented in Containerfile comment +- [ ] T5 — target-dir cache mount variant evaluated +- [ ] T6 — baseline benchmark report updated + +### Progress Log + +Append one line per meaningful update. + +| Date (UTC) | Note | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-06-01 00:00 | Spec drafted. Proxy benchmark run locally: cold registry fetch 6.9 s, warm 0.16 s, registry 823 MB. CI limitation confirmed: `type=gha` layer cache does not persist `--mount=type=cache` volumes on ephemeral runners. | diff --git a/docs/issues/drafts/1840-workflow-performance-prebuilt-base-images/ISSUE.md b/docs/issues/drafts/1840-workflow-performance-prebuilt-base-images/ISSUE.md new file mode 100644 index 000000000..98c9d5768 --- /dev/null +++ b/docs/issues/drafts/1840-workflow-performance-prebuilt-base-images/ISSUE.md @@ -0,0 +1,160 @@ +--- +doc-type: issue +issue-type: task +status: draft +priority: p3 +github-issue: null +spec-path: docs/issues/drafts/1840-workflow-performance-prebuilt-base-images/ISSUE.md +branch: "{issue-number}-prebuilt-base-images" +related-pr: null +last-updated-utc: 2026-06-01 00:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - Containerfile + - .github/workflows/container.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md +--- + + + +# Issue #[To be assigned] - Publish stable base stages as pre-built Docker Hub images + +## Goal + +Extract the rarely-changing Containerfile stages (`chef`, `tester`, `gcc`) into +versioned pre-built images published on Docker Hub, so the container build can +skip rebuilding them from scratch on every CI run. + +## Background + +The Containerfile has three base stages that change infrequently: + +- **`chef`** (`rust:trixie`): installs `cargo-binstall`, `cargo-chef`, and + `cargo-nextest`. +- **`tester`** (`rust:slim-trixie`): installs system packages (`curl`, + `sqlite3`, `time`), `cargo-binstall`, `cargo-nextest`, and initializes a + SQLite3 test database. +- **`gcc`** (`gcc:trixie`): compiles `su-exec` from source. + +These stages are stable: they only need rebuilding when the upstream Rust/GCC +base image changes or when the pinned tool versions (`cargo-chef`, +`cargo-nextest`) are updated. In a warm Docker layer cache they are already +skipped, but on cold runners (new runner allocation, cache eviction, or cache +miss) they are rebuilt from scratch, requiring apt-get downloads, cargo-binstall +bootstrap, and tool installation. + +### Expected benefit + +The expected wall-clock saving is **small**. Each stage was benchmarked locally +using `docker build --no-cache` with base images already present (i.e. simulating +a CI runner that has the upstream images cached but no intermediate layer cache). +Machine: Ryzen 9 7950X, 2026-06-01. + +| Stage | Dominant cost | Measured time (RUN/COPY steps only) | +| -------- | ------------------------------------- | ----------------------------------- | +| `gcc` | single C file compile | 1.2 s | +| `tester` | apt-get + cargo-binstall + nextest | 11 s | +| `chef` | cargo-binstall + cargo-chef + nextest | 4.5 s | + +Total build steps (RUN/COPY, base images cached): **~17 s**. + +On a truly cold runner where base images are not present, add pull time for: + +- `rust:trixie` (~1.6 GB uncompressed; ~500–600 MB compressed) +- `rust:slim-trixie` (~900 MB uncompressed; ~300 MB compressed) +- `gcc:trixie` (~1.5 GB uncompressed; ~500 MB compressed) + +At typical GitHub Actions runner network speeds (~500 Mbps), image pulls add +roughly **20–40 s**. Total worst-case cold build: **< 1 min**. + +The overall container build baseline is 35–40 min. These three stages represent +**< 2%** of total build time. The compile and link stages dominate overwhelmingly. + +By contrast, the operational cost of maintaining pre-built images is +non-trivial: + +- A separate CI workflow is needed to rebuild and publish images when any + ingredient changes (Rust version bump, tool version update, apt package + change). +- Images must be versioned and tagged precisely to avoid stale caches (e.g. + `torrust/tracker-chef:rust-trixie-chef-0.1.0-nextest-0.9.98`). +- Published images require security scanning and regular rebuilds to incorporate + upstream OS/library patches. +- Any mismatch between the pre-built image and what the Containerfile expects + is a silent correctness risk. + +### When this becomes more valuable + +The trade-off shifts in favor of pre-built images if: + +- The `chef` stage grows significantly (e.g. after adding `mold` or other + build tools — see sub-issue #9 on alternative linker). +- CI runners begin allocating fresh environments more often (longer cold-cache + periods). +- The `tester` stage requires more apt packages or longer setup steps. +- GitHub Actions introduces a way to share layer cache across workflows more + reliably, making pre-built images the natural anchor point. + +## Scope + +### In scope + +- Measure the actual cold-build time of the three base stages locally and in CI + (no layer cache) so the real baseline saving is known before deciding whether + to proceed. **Local measurement complete — see T1 in Background.** +- Evaluate what a versioning and publishing workflow would look like (trigger + policy, tagging strategy, image retention). +- Decide whether the saving justifies the maintenance cost. + +### Out of scope + +- Pre-building the `recipe`, `dependencies`, `dependencies_debug`, `build`, + `build_debug`, `test`, or `test_debug` stages — those change on every commit + and are not candidates for pre-publishing. +- Changing the base images themselves (Rust version policy is a separate + concern). +- Configuring a private registry or caching service (Docker Hub public images + are sufficient if pursued). + +## Implementation Plan + +| Task ID | Description | Status | +| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | +| T1 | Measure actual cold-build time of `chef`, `tester`, and `gcc` stages in CI (disable layer cache for those stages only) and record in baseline report | DONE | +| T2 | Define a versioning and tagging scheme for the pre-built images | TODO | +| T3 | Draft a GitHub Actions workflow that builds and publishes the base images on a push to `main`/`develop` when relevant files change | TODO | +| T4 | Update the Containerfile to `FROM` the published images instead of rebuilding from upstream | TODO | +| T5 | Validate that CI builds are still reproducible and that the image cache hit rate improves measurably | TODO | +| T6 | Document the rebuild trigger policy and tagging convention in `docs/containers.md` | TODO | + +## Risks and Trade-offs + +| Risk | Likelihood | Mitigation | +| -------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Pre-built image becomes stale after upstream patch | Medium | Automated weekly rebuild; Dependabot or Renovate alerts on base image digest change | +| Version mismatch between image and Containerfile | Medium | Pin image tags to exact tool versions in a shared variable; fail loudly on mismatch | +| Low actual saving makes maintenance unjustifiable | **Confirmed** | T1 measured locally: ~17 s total RUN/COPY (gcc: 1.2 s, tester: 11 s, chef: 4.5 s). < 2% of 35–40 min baseline. Proceed only if CI cold-cache frequency increases significantly. | +| Docker Hub rate limiting or outage | Low | Fall back to rebuilding from upstream base images (original Containerfile still works without the pre-built `FROM` lines) | + +## Progress Tracking + +### Checklist + +- [x] T1 — measure cold-build time of base stages locally: gcc 1.2 s, tester 11 s, chef 4.5 s — total ~17 s (base images cached); < 2% of 35–40 min baseline +- [ ] T2 — versioning and tagging scheme defined +- [ ] T3 — publishing workflow drafted +- [ ] T4 — Containerfile updated to FROM published images +- [ ] T5 — CI build validated; cache hit rate measured +- [ ] T6 — `docs/containers.md` updated + +### Progress Log + +Append one line per meaningful update. + +| Date (UTC) | Note | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-06-01 00:00 | Spec drafted. Low-priority idea: base stages are fast (3–7 min cold), compile dominates. Document for future re-evaluation if context changes. | +| 2026-06-01 00:00 | T1 measured locally with `docker build --no-cache` (base images cached): gcc 1.2 s, tester 11 s, chef 4.5 s — total ~17 s. Cold pull adds ~30 s for base images. Total < 1 min vs 35–40 min baseline. | diff --git a/docs/issues/drafts/1840-workflow-performance-split-external-dep-cache-layer/ISSUE.md b/docs/issues/drafts/1840-workflow-performance-split-external-dep-cache-layer/ISSUE.md new file mode 100644 index 000000000..745d6928f --- /dev/null +++ b/docs/issues/drafts/1840-workflow-performance-split-external-dep-cache-layer/ISSUE.md @@ -0,0 +1,230 @@ +--- +doc-type: issue +issue-type: task +status: draft +priority: p4 +github-issue: null +spec-path: docs/issues/drafts/1840-workflow-performance-split-external-dep-cache-layer/ISSUE.md +branch: "{issue-number}-split-external-dep-cache-layer" +related-pr: null +last-updated-utc: 2026-06-01 12:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - Containerfile + - Cargo.toml + - Cargo.lock + - .github/workflows/container.yaml + - .github/workflows/testing.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md + - docs/issues/open/1669-overhaul-packages/EPIC.md +--- + + + +# Issue #[To be assigned] - Investigate splitting cook layer to isolate external dependency cache + +## Goal + +Determine whether the `cargo-chef` cook stage can be split into two independent +Docker layers — one for external (third-party) Cargo dependencies and one for +workspace package stubs — so that external dependency compilation is cached +independently of workspace package structure changes. + +## Background + +The current [`Containerfile`](../../../../Containerfile) uses `cargo-chef` to +pre-compile all Cargo dependencies before copying real source code. The process +has two steps: + +1. `cargo chef prepare` scans every `Cargo.toml` in the workspace and produces + a `recipe.json` that captures the full dependency graph (both external crates + and workspace-internal packages) while stripping source code, replacing each + workspace member's implementation with an empty stub. +2. `cargo chef cook` compiles all external crates using those stubs. The + resulting compiled artifacts are cached as a Docker layer. + +The cook layer is invalidated whenever `recipe.json` changes. `recipe.json` +changes whenever **any** `Cargo.toml` in the workspace changes — including when: + +- A workspace package adds, removes, or upgrades an external dependency. +- A new workspace package is added or removed. +- A workspace package's feature flags or other manifest metadata are changed. + +Because third-party crate information and workspace package metadata are +entangled in a single recipe, even a pure internal change — for example, +restructuring a workspace package's Cargo.toml without adding any external +dependency — invalidates the entire cook layer. This forces a full +re-compilation of every external crate, even though the external dep versions +have not changed. + +This project has 26 workspace packages under `packages/`, plus the root crate. +These packages change frequently; they are tightly coupled to the main binary +and most application logic lives inside them. By contrast, external dependency +versions change only when a developer explicitly updates `Cargo.lock`. + +If workspace Cargo.toml changes are significantly more frequent than Cargo.lock +changes, the cook layer may be invalidated far more often than necessary, +undermining the intended caching benefit of `cargo-chef`. + +### Preliminary timing analysis + +A `cargo timings` run on the full workspace (June 2026) shows that the largest +single contributors to compilation time are C-library build scripts: + +| Crate | Cook time | +| ------------------------------ | --------- | +| `libsqlite3-sys` build scripts | ~21s | +| `aws-lc-sys` build script | ~14s | +| `zstd-sys` build script | ~11s | +| `ring` build script | ~5s | + +By contrast, workspace package stubs (the empty `src/lib.rs`/`src/main.rs` +shells that `cargo-chef` compiles during cook) are near-zero each — their +full-source compilation times (e.g. `torrust-tracker-core` at 2.4s, +`torrust-tracker-configuration` at 2.1s) are incurred in the `build` stage +**after** the source copy, not in the cook stage. + +This finding reduces the expected benefit of a split cook layer: even if the +external-dep layer is perfectly cached, the total cook time saved on a +workspace-`Cargo.toml`-only change is only the sum of workspace **stub** +compilations (likely a few seconds total), not the C build scripts (~51s+). +The C build scripts are external crates and would still execute in the inner +cook layer. + +The optimization remains worth investigating only after other higher-impact +changes (target scope narrowing, `.dockerignore` audit, cache reuse policy) +have been applied and workspace-package compilation time becomes a material +fraction of the remaining cook time. See EPIC #1669: if most workspace packages +are extracted as external crates, this issue becomes moot. + +### Relationship to EPIC #1669 + +EPIC #1669 aims to extract several generic workspace packages into standalone +repositories. Once extracted, those packages will be consumed as external crates +and their version bumps will appear in `Cargo.lock` rather than as workspace +`Cargo.toml` edits. This will naturally shift the invalidation trigger toward a +more stable baseline over time. This issue is more valuable in the short term +while the workspace is still large. + +### Distinction from existing issues + +- `1840-workflow-performance-dependency-layer-cache-reuse`: that issue covers + the CI-level cache backend (GHA cache keys, BuildKit cache mounts) and whether + cache entries are being reused across jobs and workflow runs. This issue is + about the Containerfile layer structure itself — what `cargo chef` stages are + defined and what invalidates them. + +## Scope + +### In Scope + +- Measure the frequency of cook layer invalidation in recent git history: how + often do workspace `Cargo.toml` files change without also changing `Cargo.lock`? +- Investigate whether `cargo-chef` supports generating a recipe scoped to + external dependencies only (excluding workspace members). +- Investigate alternative approaches to separating external dep compilation from + workspace stub compilation (see Known Candidate Approaches below). +- If a viable approach is found: prototype it and measure the before/after effect + on warm build times when only a workspace `Cargo.toml` is modified (no new + external deps). +- Validate that cold build time does not regress. +- If no viable approach is found: document the investigation findings and close + the issue. + +### Out of Scope + +- Changes to build targets (covered by the containerfile-target-scope issue). +- CI-level cache backend configuration (covered by dependency-layer-cache-reuse). +- Changes to `Cargo.toml` dependency versions or workspace package structure + beyond what is needed to validate the prototype. + +## Known Candidate Approaches + +| ID | Approach | Description | Feasibility Notes | +| --- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| A1 | `cargo-chef` filter flag | Use a `cargo chef prepare` option to generate an external-only recipe | Needs investigation — not documented in `cargo-chef` README as of 2026-06 | +| A2 | Post-process `recipe.json` | Strip workspace member entries from `recipe.json` after `cargo chef prepare` | Potentially feasible but fragile; `recipe.json` format is an internal detail of `cargo-chef` | +| A3 | `cargo fetch` pre-stage | Copy only `Cargo.toml`/`Cargo.lock`; run `cargo fetch --locked`; cook on top | Pre-fetches source archives but does not compile; may not preserve compiled artifact cache across layers | +| A4 | Minimal synthetic workspace | Construct a synthetic top-level `Cargo.toml` that declares only external deps; cook it first; cook the full recipe on top | Fully separates external vs internal invalidation but adds manifest maintenance overhead | + +## Implementation Plan + +Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. + +| ID | Status | Task | Notes / Expected Output | +| --- | ------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| T1 | TODO | Measure cook layer invalidation frequency in git log | Count commits in the last 6 months that changed a workspace `Cargo.toml` without also changing `Cargo.lock`. Record the ratio. | +| T2 | TODO | Investigate `cargo-chef` filter capabilities | Read `cargo-chef` source and docs; test `cargo chef prepare` options; determine if workspace-member exclusion is natively supported. | +| T3 | TODO | Evaluate candidate approaches A1–A4 | Score each approach for feasibility, complexity, and maintenance cost. Select the most promising for prototyping or conclude not feasible. | +| T4 | TODO | Prototype the chosen approach (if feasible) | Build a proof-of-concept Containerfile with a split cook stage; confirm it builds correctly locally. | +| T5 | TODO | Measure warm build time improvement | Run the warm baseline with a workspace `Cargo.toml` change (no new external dep); compare cook stage rebuild time before and after split. | +| T6 | TODO | Validate cold build time is unchanged | Run the cold baseline; confirm total build time is within measurement noise of the original baseline. | +| T7 | TODO | Document findings and update Containerfile if beneficial | If split is beneficial: update the Containerfile. If not: write a findings note and close as declined. | + +## Progress Tracking + +### Workflow Checkpoints + +- [ ] Spec drafted in `docs/issues/drafts/` +- [ ] Spec reviewed and approved by user/maintainer +- [ ] GitHub issue created and issue number added to this spec +- [ ] (Optional, recommended for complex issues) Spec-only PR merged into `develop` before implementation +- [ ] Implementation completed +- [ ] Automatic verification completed (`linter all`, relevant tests, and any pre-push checks) +- [ ] Manual verification scenarios executed and recorded (status + evidence) +- [ ] Acceptance criteria reviewed after implementation and updated with evidence +- [ ] Reviewer validated acceptance criteria and updated checkboxes +- [ ] Committer verified spec progress is up to date before commit +- [ ] Issue closed and spec moved from `docs/issues/open/` to `docs/issues/closed/` + +### Progress Log + +Append one line per meaningful update. + +- 2026-06-01 00:00 UTC - GitHub Copilot - Drafted cook layer split investigation issue from EPIC #1840 discussion - draft file created +- 2026-06-01 12:00 UTC - GitHub Copilot - Downgraded priority to p4 after cargo timings analysis: C build scripts dominate cook time; workspace stub cost is near-zero; split benefit is marginal until other bottlenecks are resolved first + +## Acceptance Criteria + +- [ ] AC1: Cook layer invalidation frequency is measured and documented (ratio of workspace-Cargo.toml-only changes vs Cargo.lock changes over the last 6 months). +- [ ] AC2: Feasibility of each candidate approach (A1–A4) is evaluated and a recommendation is documented. +- [ ] AC3: If feasible: a split cook layer is prototyped, builds correctly, and warm build time with a workspace `Cargo.toml`-only change is measured before and after. +- [ ] AC4: If feasible: cold build time does not regress compared to the baseline analysis (`#1841`). +- [ ] AC5: If not feasible or not beneficial: findings are documented and the issue is explicitly closed as declined with a rationale. +- [ ] `linter all` exits with code `0` +- [ ] All CI checks pass for any changes to `Containerfile` +- [ ] Manual verification scenarios are executed and documented (status + evidence) +- [ ] Acceptance criteria are re-reviewed after implementation and reflect actual behavior + +## Verification Plan + +Define verification before implementation starts and execute it before closing the issue. + +### Automatic Checks + +- `linter all` +- CI checks pass for any changes to `Containerfile` + +### Manual Verification Scenarios + +Status values: `TODO`, `IN_PROGRESS`, `DONE`, `FAILED`, `BLOCKED`. + +| ID | Scenario | Command/Steps | Expected Result | Status | Evidence | +| --- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------ | ---------------- | +| M1 | Measure cook invalidation frequency | `git log --oneline --follow --diff-filter=M -- '**/Cargo.toml' Cargo.lock` and classify each change by type | Ratio of workspace-Cargo.toml-only changes vs Cargo.lock changes recorded. | TODO | {analysis link} | +| M2 | Warm build with workspace `Cargo.toml` change (before) | Modify a workspace package `Cargo.toml` (add a comment or feature flag; no new dep); warm baseline run; record cook stage rebuild duration. | Cook layer fully rebuilt (baseline measurement). | TODO | {benchmark link} | +| M3 | Warm build with workspace `Cargo.toml` change (after) | Same change after implementing the split cook; warm baseline run; record cook stage rebuild duration. | External dep layer preserved; only workspace stubs layer rebuilt. Total cook time noticeably lower. | TODO | {benchmark link} | +| M4 | Cold build time unchanged | Full cold run via `./contrib/dev-tools/workflow-benchmarks/run-container-baseline.sh` | Total cold build time within measurement noise of baseline from `#1841`. | TODO | {benchmark link} | + +### Acceptance Verification + +| AC ID | Status (`TODO`/`DONE`) | Evidence | +| ----- | ---------------------- | ---------------- | +| AC1 | TODO | {analysis link} | +| AC2 | TODO | {analysis link} | +| AC3 | TODO | {benchmark link} | +| AC4 | TODO | {benchmark link} | +| AC5 | TODO | {findings link} | diff --git a/docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md b/docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md similarity index 97% rename from docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md rename to docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md index a9c8c54d9..2318ee94a 100644 --- a/docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md +++ b/docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md @@ -4,7 +4,7 @@ issue-type: task status: open priority: p2 github-issue: 1726 -spec-path: docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md +spec-path: docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md branch: 1726-reduce-build-times-sccache related-pr: null last-updated-utc: 2026-05-01 00:00 @@ -80,7 +80,7 @@ Full benchmark data and compile-hotspot analysis are in - GitHub issue: https://github.com/torrust/torrust-tracker/issues/1726 - `sccache` repository: https://github.com/mozilla/sccache - `mozilla-actions/sccache-action`: https://github.com/mozilla-actions/sccache-action -- Benchmark artifact: [`docs/issues/1726-reduce-build-times-sccache/benchmark-results.md`](./benchmark-results.md) +- Benchmark artifact: [`docs/issues/1726-1840-workflow-performance-sccache/benchmark-results.md`](./benchmark-results.md) - CI workflow: [`.github/workflows/testing.yaml`](../../../.github/workflows/testing.yaml) --- diff --git a/docs/issues/open/1726-reduce-build-times-sccache/benchmark-results.md b/docs/issues/open/1726-1840-workflow-performance-sccache/benchmark-results.md similarity index 98% rename from docs/issues/open/1726-reduce-build-times-sccache/benchmark-results.md rename to docs/issues/open/1726-1840-workflow-performance-sccache/benchmark-results.md index 21d7df7e6..7ef3ae850 100644 --- a/docs/issues/open/1726-reduce-build-times-sccache/benchmark-results.md +++ b/docs/issues/open/1726-1840-workflow-performance-sccache/benchmark-results.md @@ -3,7 +3,7 @@ semantic-links: skill-links: - create-issue related-artifacts: - - docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md + - docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md --- # Cargo Build & Test Benchmark Results @@ -47,7 +47,7 @@ Machine: local dev (clean workspace) | 1 | **5.04 s** | `tests/integration.rs` — `torrust_tracker_udp_server` (6 tests) | | 2 | **3.21 s** | `unittests src/lib.rs` — `torrust_tracker_swarm_coordination_registry` (95 tests) | | 3 | **2.08 s** | `unittests src/lib.rs` — `torrust_tracker_udp_server` (122 tests) | -| 4 | **2.05 s** | `tests/integration.rs` — `torrust_tracker_axum_health_check_api_server` (7 tests) | +| 4 | **2.05 s** | `tests/integration.rs` — `torrust_tracker_axum_health_check_api_server` (7 tests) | | 5 | **0.36 s** | `tests/integration.rs` — `torrust_tracker_axum_rest_api_server` (53 tests) | | 6 | **0.23 s** | `tests/integration.rs` — `bittorrent_tracker_core` (5 tests) | | 7 | **0.21 s** | `tests/integration.rs` — `torrust_tracker_axum_http_server` (52 tests) | @@ -73,7 +73,7 @@ can be parallelised past them. | Rank | Max single unit | Sum (all units) | # units | Crate | | ---- | --------------- | --------------- | ------- | ------------------------------------------------- | | 1 | 77.19 s | 606.43 s | 13 | `torrust-tracker` (workspace root) | -| 2 | 67.46 s | 83.09 s | 3 | `torrust-tracker-axum-health-check-api-server` | +| 2 | 67.46 s | 83.09 s | 3 | `torrust-tracker-axum-health-check-api-server` | | 3 | 62.94 s | 182.15 s | 5 | `bittorrent-tracker-core` | | 4 | 60.87 s | 96.73 s | 4 | `torrust-tracker-torrent-repository-benchmarking` | | 5 | 59.04 s | 116.97 s | 3 | `torrust-tracker-axum-rest-api-server` | @@ -91,7 +91,7 @@ can be parallelised past them. | 17 | 12.71 s | 14.19 s | 2 | `torrust-tracker-swarm-coordination-registry` | | 18 | 12.27 s | 46.54 s | 5 | `torrust-tracker-client` | | 19 | 12.08 s | 13.23 s | 2 | `torrust-tracker-metrics` | -| 20 | 9.85 s | 10.18 s | 2 | `torrust-tracker-axum-server` | +| 20 | 9.85 s | 10.18 s | 2 | `torrust-tracker-axum-server` | ### Heaviest external/C dependencies diff --git a/docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md b/docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md index 933e23e7b..4fba98d8b 100644 --- a/docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md +++ b/docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md @@ -4,7 +4,7 @@ status: planned github-issue: 1840 spec-path: docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md epic-owner: josecelano -last-updated-utc: 2026-05-27 00:00 +last-updated-utc: 2026-06-01 00:00 semantic-links: skill-links: - create-issue @@ -66,14 +66,20 @@ Ordering policy: - Subissue 1 (baseline analysis) is mandatory first. - All later subissues are provisional and may be reordered based on baseline findings. -| Order | Issue | Local Spec | Status | Notes | -| ----- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | #1841 - Baseline workflow profiling and bottleneck analysis | `docs/issues/open/1841-1840-workflow-performance-baseline-analysis/ISSUE.md` | TODO | Measure both workflows with and without local caches, document the bottleneck, and keep a reusable benchmark report for later comparisons. | -| 2 | #[To be assigned] - Narrow Containerfile build targets to tracker image needs | `docs/issues/drafts/1840-workflow-performance-containerfile-target-scope/ISSUE.md` | TODO | Current likely first optimization after baseline, but execute only if baseline confirms significant time spent compiling or linking targets not required for the final tracker image. | -| 3 | #1726 - Reduce Build Times with `sccache` | `docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md` | TODO | Existing GitHub issue; link it as a child issue after the EPIC is published. Order is provisional after baseline. | -| 4 | #[To be assigned] - Evaluate test execution policy in container image build | `docs/issues/drafts/1840-workflow-performance-container-test-gating/ISSUE.md` | TODO | Assess whether test execution inside container build is redundant, evaluate separating validation from packaging across multiple artifact types, and define safer gating plus optional debug-image paths for failing commits. | -| 5 | #[To be assigned] - Improve dependency-layer cache reuse within each workflow | `docs/issues/drafts/1840-workflow-performance-dependency-layer-cache-reuse/ISSUE.md` | TODO | Ensure dependency layers are reused reliably inside each workflow when Cargo dependencies are unchanged. Defer optional cross-workflow cache-sharing and sequencing trade-offs to follow-up once this is working. | -| 6 | #[To be assigned] - Evaluate removing duplicate container build from container workflow | `docs/issues/drafts/1840-workflow-performance-container-workflow-build-deduplication/ISSUE.md` | TODO | Assess whether PR-time container build in container workflow is redundant because testing workflow already builds an image for Docker E2E, and keep publish paths intact. | +| Order | Issue | Local Spec | Status | Notes | +| ----- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | #1841 - Baseline workflow profiling and bottleneck analysis | `docs/issues/open/1841-1840-workflow-performance-baseline-analysis/ISSUE.md` | DONE | Merged in PR #1848. Baseline report at `docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md`. | +| 2 | #1852 - Restrict recipe stage to manifest-only COPY | `docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md` | TODO | Replace `COPY . /build/src` in the `recipe` stage with per-manifest COPY lines so the cook (dependency) layers are only invalidated when `Cargo.toml` or `Cargo.lock` changes, not on every `.rs` edit. High expected impact. | +| 3 | #1851 - Audit `.dockerignore` to minimize Docker build context | `docs/issues/open/1851-1840-workflow-performance-dockerignore-audit/ISSUE.md` | TODO | Systematically exclude tracked repo paths not needed in any Containerfile stage to reduce context transfer size and reduce spurious cache invalidation of `build` and `test` stages. | +| 4 | #1853 - Narrow Containerfile build targets to tracker image needs | `docs/issues/open/1853-1840-workflow-performance-containerfile-target-scope/ISSUE.md` | TODO | Execute only if baseline confirms significant time spent compiling or linking targets not required for the final tracker image. | +| 5 | #1726 - Reduce Build Times with `sccache` | `docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md` | TODO | Existing GitHub issue; link it as a child issue after the EPIC is published. Order is provisional after baseline. | +| 6 | #1854 - Evaluate test execution policy in container image build | `docs/issues/open/1854-1840-workflow-performance-container-test-gating/ISSUE.md` | TODO | Assess whether test execution inside container build is redundant, evaluate separating validation from packaging across multiple artifact types, and define safer gating plus optional debug-image paths for failing commits. | +| 7 | #[To be assigned] - Improve dependency-layer cache reuse within each workflow | `docs/issues/drafts/1840-workflow-performance-dependency-layer-cache-reuse/ISSUE.md` | TODO | Ensure dependency layers are reused reliably inside each workflow when Cargo dependencies are unchanged. Defer optional cross-workflow cache-sharing and sequencing trade-offs to follow-up once this is working. | +| 8 | #[To be assigned] - Evaluate removing duplicate container build from container workflow | `docs/issues/drafts/1840-workflow-performance-container-workflow-build-deduplication/ISSUE.md` | TODO | Assess whether PR-time container build in container workflow is redundant because testing workflow already builds an image for Docker E2E, and keep publish paths intact. | +| 9 | #[To be assigned] - Switch to a faster linker (mold or lld) to reduce link time | `docs/issues/drafts/1840-workflow-performance-alternative-linker/ISSUE.md` | TODO | Baseline shows 35–117 s link time per binary (sections: null). Fair local relink: BFD = mold (54 s each) — compile dominates incremental builds. mold docs: 10–31× faster than BFD in cold builds (MySQL: 10.8 s → 0.46 s). 20+ binaries linked in container build. | +| 10 | #[To be assigned] - Investigate splitting cook layer to isolate external dependency cache (p4, deferred) | `docs/issues/drafts/1840-workflow-performance-split-external-dep-cache-layer/ISSUE.md` | TODO | Low priority. C build scripts dominate cook time; workspace stub cost is near-zero. Revisit once other bottlenecks are resolved and workspace shrinks via EPIC #1669. | +| 11 | #[To be assigned] - Publish stable base stages as pre-built Docker Hub images (p3, deferred) | `docs/issues/drafts/1840-workflow-performance-prebuilt-base-images/ISSUE.md` | TODO | Low priority. Base stages (`chef`, `tester`, `gcc`) are fast (3–7 min cold). Compile dominates (35+ min). Revisit if base stages grow or if CI runner cold-cache frequency increases. | +| 12 | #[To be assigned] - Pass Cargo registry/git caches into BuildKit cook stages | `docs/issues/drafts/1840-workflow-performance-buildkit-cargo-cache-mounts/ISSUE.md` | TODO | Adds `--mount=type=cache` for registry/git to cook stages. Local benefit: saves ~7 s download per cook rebuild (cold fetch 6.9 s → warm 0.16 s; registry 823 MB). CI benefit: none with ephemeral GitHub Actions runners (`type=gha` layer cache does not persist cache mount volumes). Evaluate target-dir cache mount variant as T5. | ## Delivery Strategy @@ -130,6 +136,10 @@ Append one line per meaningful update. - 2026-05-27 00:00 UTC - GitHub Copilot - Clarified that only baseline order is fixed and made later optimization order provisional - draft updated - 2026-05-27 00:00 UTC - GitHub Copilot - Created GitHub EPIC issue #1840 and moved spec to `docs/issues/open/` - draft updated - 2026-05-27 00:00 UTC - GitHub Copilot - Created baseline subissue #1841 and linked it as a GitHub child issue of #1840 - draft updated +- 2026-06-01 00:00 UTC - GitHub Copilot - Marked #1841 DONE (merged PR #1848); added sub-issues: recipe-stage-manifest-only-copy (p1), dockerignore-audit (p2), split-external-dep-cache-layer (p4 deferred); reordered table by expected impact +- 2026-06-01 00:00 UTC - GitHub Copilot - Added sub-issues: alternative-linker (p1, row 9), prebuilt-base-images (p3 deferred, row 11) +- 2026-06-01 00:00 UTC - GitHub Copilot - Added sub-issue: buildkit-cargo-cache-mounts (p2, row 12); local benchmark: cold fetch 6.9 s → warm 0.16 s; CI limitation documented +- 2026-06-01 00:00 UTC - GitHub Copilot - Promoted rows 2/3/4/6 from drafts to open: #1851 dockerignore-audit, #1852 recipe-manifest-only-copy, #1853 containerfile-target-scope, #1854 container-test-gating ## Acceptance Criteria diff --git a/docs/issues/open/1851-1840-workflow-performance-dockerignore-audit/ISSUE.md b/docs/issues/open/1851-1840-workflow-performance-dockerignore-audit/ISSUE.md new file mode 100644 index 000000000..73bba8b2f --- /dev/null +++ b/docs/issues/open/1851-1840-workflow-performance-dockerignore-audit/ISSUE.md @@ -0,0 +1,175 @@ +--- +doc-type: issue +issue-type: task +status: open +priority: p2 +github-issue: 1851 +spec-path: docs/issues/open/1851-1840-workflow-performance-dockerignore-audit/ISSUE.md +branch: "1851-workflow-performance-dockerignore-audit" +related-pr: null +last-updated-utc: 2026-05-29 00:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - .dockerignore + - .gitignore + - Containerfile + - .github/workflows/container.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md +--- + + + +# Issue #1851 - Audit .dockerignore to minimize Docker build context + +## Goal + +Ensure the Docker build context sent to BuildKit is as small as possible by +auditing `.dockerignore` against `.gitignore` and the actual container +contents, then adding any paths that are tracked by git but not needed in any +Containerfile stage. + +## Background + +Every file **not** excluded from the build context is transferred to the +BuildKit daemon before the build starts. Large contexts increase transfer time, +create unnecessary cache invalidation when unrelated files change (e.g. docs, +CI config, dev tools), and add noise to layer diffs. + +The baseline analysis (`#1841`) already identified one concrete case: the +`.tmp/` directory (AI agent hook logs + benchmark cargo isolation dirs) was +included in the build context and triggering cache misses. That entry was added +to `.dockerignore` as a quick fix. A systematic audit may reveal further +candidates. + +Additionally, the `Containerfile` stages that perform a full source copy +(`COPY . /build/src`) are particularly sensitive to context size: any file not +excluded will invalidate those layers' cache whenever it changes, even if the +change is irrelevant to the build (e.g. updating a doc or a YAML config file). + +## Scope + +### In Scope + +- Compare `.dockerignore` with `.gitignore` and identify paths present in the + repo that are not needed inside any Containerfile stage. +- Inspect the actual build context size (before and after) and the contents + transferred using `docker build --progress=plain` or `docker buildx du`. +- Optionally build a local image and inspect the filesystem at each stage to + verify no needed files are accidentally excluded. +- Add all safe exclusions to `.dockerignore` and measure the reduction in + context size and any improvement in layer cache hit rate. +- Document which files are **intentionally** kept (e.g. `share/`, `contrib/`) + and why. + +### Out of Scope + +- Restructuring the `COPY` instructions in the Containerfile to copy only + subsets of the source tree (that belongs to a separate issue). +- Changes to the build stages or caching strategy beyond `.dockerignore` edits. +- Changes to `.gitignore`. + +## Known Candidates + +Based on an initial comparison of `.dockerignore` and `.gitignore`, the +following tracked paths are not currently excluded from the Docker build context +and appear unlikely to be needed in any Containerfile stage: + +| Path | Reason likely safe to exclude | +| --------------------------------------------------------- | ---------------------------------------------- | +| `.github/` | CI config — not referenced by any stage | +| `.vscode/` | Editor config — not referenced by any stage | +| `.gitignore` | Git metadata — not referenced by any stage | +| `.git-blame-ignore` | Git metadata — not referenced by any stage | +| `docs/` | Documentation — not referenced by any stage | +| `codecov.yaml` | CI config — not referenced by any stage | +| `compose.*.yaml` | Compose files — not referenced by any stage | +| `cspell.json` / `project-words.txt` | Spell-check config — not used inside container | +| `rustfmt.toml` | Formatter config — not used inside container | +| `.markdownlint.json` / `.taplo.toml` / `.yamllint-ci.yml` | Linter config — not used inside container | +| `AGENTS.md` | Agent instructions — not used inside container | +| `README.md` / `NOTICE` / `SECURITY.md` / `LICENSE` | Project docs — not used inside container | +| `contrib/dev-tools/` | Dev tooling — not used inside container | + +> These are candidates only. Each must be confirmed safe before being added. + +## Implementation Plan + +Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. + +| ID | Status | Task | Notes / Expected Output | +| --- | ------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| T1 | TODO | Measure current build context size | Use `docker buildx build --progress=plain` or context dump to get baseline size and file list. | +| T2 | TODO | Cross-reference `.dockerignore` vs `.gitignore` | List all tracked paths absent from `.dockerignore` and classify each as needed / not needed / unsure. | +| T3 | TODO | Inspect container stage contents | Build the image locally and walk the filesystem of each stage to verify no needed file is excluded. | +| T4 | TODO | Add safe exclusions to `.dockerignore` | Add confirmed-safe paths; document intentionally included paths with inline comments. | +| T5 | TODO | Measure context size and cache behaviour after | Re-run baseline script (warm + cold) and record reduction in context transfer time and cache misses. | + +## Progress Tracking + +### Workflow Checkpoints + +- [x] Spec drafted in `docs/issues/drafts/` +- [x] Spec reviewed and approved by user/maintainer +- [x] GitHub issue created and issue number added to this spec +- [ ] (Optional, recommended for complex issues) Spec-only PR merged into `develop` before implementation +- [ ] Implementation completed +- [ ] Automatic verification completed (`linter all`, relevant tests, and any pre-push checks) +- [ ] Manual verification scenarios executed and recorded (status + evidence) +- [ ] Acceptance criteria reviewed after implementation and updated with evidence +- [ ] Reviewer validated acceptance criteria and updated checkboxes +- [ ] Committer verified spec progress is up to date before commit +- [ ] Issue closed and spec moved from `docs/issues/open/` to `docs/issues/closed/` + +### Progress Log + +Append one line per meaningful update. + +- 2026-05-29 00:00 UTC - GitHub Copilot - Drafted .dockerignore audit issue from baseline analysis findings - draft file created +- 2026-06-01 00:00 UTC - GitHub Copilot - GitHub issue #1851 created; spec moved from drafts/ to open/ + +## Acceptance Criteria + +- [ ] AC1: Current Docker build context size is measured and recorded. +- [ ] AC2: All tracked repo paths are classified as needed / excluded / intentionally kept with a rationale. +- [ ] AC3: `.dockerignore` is updated with all confirmed-safe exclusions. +- [ ] AC4: No Containerfile stage is broken by the new exclusions (all CI checks pass). +- [ ] AC5: Build context size is re-measured and the reduction is documented. +- [ ] AC6: Intentionally included paths are documented with inline comments in `.dockerignore`. +- [ ] `linter all` exits with code `0` +- [ ] All CI checks pass for changed files +- [ ] Manual verification scenarios are executed and documented (status + evidence) +- [ ] Acceptance criteria are re-reviewed after implementation and reflect actual behaviour + +## Verification Plan + +Define verification before implementation starts and execute it before closing the issue. + +### Automatic Checks + +- `linter all` +- All CI checks pass for changed `.dockerignore` and Containerfile + +### Manual Verification Scenarios + +Status values: `TODO`, `IN_PROGRESS`, `DONE`, `FAILED`, `BLOCKED`. + +| ID | Scenario | Command/Steps | Expected Result | Status | Evidence | +| --- | ---------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | ------ | ----------------- | +| M1 | Measure context before | `docker buildx build --progress=plain . 2>&1 \| grep -E 'transferring context\|sending build context'` | Baseline context size recorded. | TODO | {log/output path} | +| M2 | Verify no stage breaks | Full cold `docker build --target release .` | Build completes successfully; all stages produce expected artifacts. | TODO | {log path} | +| M3 | Measure context after | Same command as M1 after `.dockerignore` update | Context size smaller than baseline; reduction documented. | TODO | {log/output path} | +| M4 | Cache stability check | Run warm baseline twice: `run-container-baseline.sh` without `--cold` | Layer cache hit rates are stable or improved; no unexpected misses due to excluded file changes. | TODO | {benchmark link} | + +### Acceptance Verification + +| AC ID | Status (`TODO`/`DONE`) | Evidence | +| ----- | ---------------------- | -------------------- | +| AC1 | TODO | {benchmark/log link} | +| AC2 | TODO | {analysis link} | +| AC3 | TODO | {diff link} | +| AC4 | TODO | {CI run link} | +| AC5 | TODO | {benchmark/log link} | +| AC6 | TODO | {diff link} | diff --git a/docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md b/docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md new file mode 100644 index 000000000..bf8ff7f1c --- /dev/null +++ b/docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md @@ -0,0 +1,240 @@ +--- +doc-type: issue +issue-type: task +status: open +priority: p1 +github-issue: 1852 +spec-path: docs/issues/open/1852-1840-workflow-performance-recipe-stage-manifest-only-copy/ISSUE.md +branch: "1852-recipe-stage-manifest-only-copy" +related-pr: null +last-updated-utc: 2026-06-01 00:00 +semantic-links: + skill-links: + - create-issue + related-artifacts: + - Containerfile + - Cargo.toml + - Cargo.lock + - .github/workflows/container.yaml + - .github/workflows/testing.yaml + - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md + - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md +--- + + + +# Issue #1852 - Restrict recipe stage to manifest-only COPY to prevent spurious cook cache invalidation + +## Goal + +Prevent the `cargo chef cook` (dependency) layers from being invalidated on +every source code change by replacing the full-tree `COPY . /build/src` in the +`recipe` stage with a manifest-only copy of `Cargo.toml` and `Cargo.lock` files. + +## Background + +The current [`Containerfile`](../../../../Containerfile) `recipe` stage does: + +```dockerfile +FROM chef AS recipe +WORKDIR /build/src +COPY . /build/src # copies the entire source tree +RUN cargo chef prepare --recipe-path /build/recipe.json +``` + +The `cargo chef prepare` command only reads `Cargo.toml` manifests and +`Cargo.lock` to build `recipe.json`. It does not read any `.rs` source files. +This is explicitly stated in `cargo-chef`'s own CLI description: + +> "Analyze the current project to determine the **minimum subset of files +> (Cargo.lock and Cargo.toml manifests)** required to build it and cache +> dependencies" + +However, because `COPY . /build/src` copies all source files into the recipe +stage, Docker invalidates that layer's cache whenever **any tracked file +changes** — including `.rs` files, documentation, shell scripts, and anything +else in the build context. Since the recipe stage is upstream of both +`dependencies` and `dependencies_debug` cook stages, this cascades: + +```text +COPY . /build/src ← cache miss on any file change + → cargo chef prepare → recipe.json changes (or not — Docker can't tell) + → COPY --from=recipe recipe.json ← invalidated regardless + → cargo chef cook ← full external dep recompile +``` + +The cook stage recompiles everything: C build scripts (`libsqlite3-sys` ~21s, +`aws-lc-sys` ~14s, `zstd-sys` ~11s, `ring` ~5s) and hundreds of Rust crates. +On a warm run where only application code changed, this cost is paid +unnecessarily every time. + +### The fix + +Replace the full-tree copy with a manifest-only copy in the recipe stage: + +```dockerfile +FROM chef AS recipe +WORKDIR /build/src +COPY Cargo.toml Cargo.lock ./ +COPY packages/axum-health-check-api-server/Cargo.toml packages/axum-health-check-api-server/ +COPY packages/axum-http-server/Cargo.toml packages/axum-http-server/ +COPY packages/axum-rest-api-server/Cargo.toml packages/axum-rest-api-server/ +COPY packages/axum-server/Cargo.toml packages/axum-server/ +COPY packages/clock/Cargo.toml packages/clock/ +COPY packages/configuration/Cargo.toml packages/configuration/ +COPY packages/events/Cargo.toml packages/events/ +COPY packages/http-protocol/Cargo.toml packages/http-protocol/ +COPY packages/http-tracker-core/Cargo.toml packages/http-tracker-core/ +COPY packages/located-error/Cargo.toml packages/located-error/ +COPY packages/metrics/Cargo.toml packages/metrics/ +COPY packages/net-primitives/Cargo.toml packages/net-primitives/ +COPY packages/peer-id/Cargo.toml packages/peer-id/ +COPY packages/primitives/Cargo.toml packages/primitives/ +COPY packages/rest-api-client/Cargo.toml packages/rest-api-client/ +COPY packages/rest-api-core/Cargo.toml packages/rest-api-core/ +COPY packages/server-lib/Cargo.toml packages/server-lib/ +COPY packages/swarm-coordination-registry/Cargo.toml packages/swarm-coordination-registry/ +COPY packages/test-helpers/Cargo.toml packages/test-helpers/ +COPY packages/torrent-repository-benchmarking/Cargo.toml packages/torrent-repository-benchmarking/ +COPY packages/tracker-client/Cargo.toml packages/tracker-client/ +COPY packages/tracker-core/Cargo.toml packages/tracker-core/ +COPY packages/udp-protocol/Cargo.toml packages/udp-protocol/ +COPY packages/udp-server/Cargo.toml packages/udp-server/ +COPY packages/udp-tracker-core/Cargo.toml packages/udp-tracker-core/ +COPY console/tracker-client/Cargo.toml console/tracker-client/ +COPY contrib/bencode/Cargo.toml contrib/bencode/ +COPY contrib/dev-tools/analysis/workspace-coupling/Cargo.toml contrib/dev-tools/analysis/workspace-coupling/ +RUN cargo chef prepare --recipe-path /build/recipe.json +``` + +After this change, the recipe stage cache (and therefore the cook layers) is +only invalidated when `Cargo.toml` or `Cargo.lock` actually changes — not on +every `.rs` edit. For a typical PR that modifies only source code, the cook +layers remain fully cached. + +### Maintenance cost + +The manifest-only COPY list must be kept in sync with the workspace member list +in the root `Cargo.toml`. Every time a new workspace package is added or an +existing one is moved or removed, the Containerfile must be updated. The +`cargo-chef` documentation acknowledges this trade-off; it uses `COPY . .` in +its canonical example purely for simplicity and portability. This project's +workspace is relatively stable (packages are being extracted to separate repos +under EPIC #1669, reducing the list over time), so the maintenance overhead is +low and proportional to how often the workspace structure changes. + +A CI check that validates all workspace member directories have a corresponding +`COPY` line in the Containerfile can catch drift automatically. + +### Distinction from existing issues + +- `1840-workflow-performance-dockerignore-audit`: that issue reduces the build + context size (bytes transferred to the BuildKit daemon) and reduces spurious + invalidation of the `build` and `test` stages. This issue prevents spurious + invalidation of the `recipe` and `cook` stages, which is a separate and + higher-value fix: the cook stages contain the entire external dependency + compilation cost (~200–400s). +- `1840-workflow-performance-dependency-layer-cache-reuse`: that issue covers + the CI-level cache backend (GHA cache keys, BuildKit cache mounts). This + issue is about the Containerfile layer structure itself. + +## Scope + +### In Scope + +- Replace `COPY . /build/src` in the `recipe` stage with individual + `COPY /` lines for every workspace member. +- Verify that `cargo chef prepare` produces an equivalent `recipe.json` with + the manifest-only copy. +- Verify that the full build pipeline (all Containerfile targets) still works + end-to-end after the change. +- Measure warm build time before and after with a source-only change (no + `Cargo.toml` or `Cargo.lock` modification) to confirm cook layers are cached. +- Document the maintenance requirement (keeping manifest list in sync). +- Optionally: add a CI check or script to verify that every workspace member in + `Cargo.toml` has a corresponding `COPY` line in the Containerfile. + +### Out of Scope + +- Changing the `build` or `test` stage `COPY . /build/src` instructions (those + require the full source tree and cannot be restricted without a larger + redesign). +- Changes to `.dockerignore` (covered by the `dockerignore-audit` issue). +- Cross-workflow cache backend configuration. + +## Implementation Plan + +Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. + +| ID | Status | Task | Notes / Expected Output | +| --- | ------ | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| T1 | TODO | Replace full-tree COPY with manifest-only COPY in recipe stage | One `COPY /` line per workspace member, plus root `Cargo.toml` and `Cargo.lock`. | +| T2 | TODO | Verify recipe.json equivalence | Build locally; diff `recipe.json` output before and after to confirm it is identical. | +| T3 | TODO | Verify full build pipeline | Run `docker build --target release .` locally; confirm all stages succeed. | +| T4 | TODO | Measure warm build time improvement | Run warm baseline (`run-container-baseline.sh`) with a source-only change; confirm cook layers show cache hit; record time saved. | +| T5 | TODO | Document maintenance requirement in Containerfile | Add inline comment above the manifest COPY block explaining the sync requirement. | +| T6 | TODO | Optionally add CI drift check | Script or CI step that compares workspace members in `Cargo.toml` against `COPY` lines in `Containerfile` and fails on mismatch. | + +## Progress Tracking + +### Workflow Checkpoints + +- [x] Spec drafted in `docs/issues/drafts/` +- [x] Spec reviewed and approved by user/maintainer +- [x] GitHub issue created and issue number added to this spec +- [ ] (Optional, recommended for complex issues) Spec-only PR merged into `develop` before implementation +- [ ] Implementation completed +- [ ] Automatic verification completed (`linter all`, relevant tests, and any pre-push checks) +- [ ] Manual verification scenarios executed and recorded (status + evidence) +- [ ] Acceptance criteria reviewed after implementation and updated with evidence +- [ ] Reviewer validated acceptance criteria and updated checkboxes +- [ ] Committer verified spec progress is up to date before commit +- [ ] Issue closed and spec moved from `docs/issues/open/` to `docs/issues/closed/` + +### Progress Log + +Append one line per meaningful update. + +- 2026-06-01 00:00 UTC - GitHub Copilot - Drafted recipe stage manifest-only copy issue from EPIC #1840 discussion - draft file created + +## Acceptance Criteria + +- [ ] AC1: The `recipe` stage uses manifest-only COPY (no full-tree copy); every workspace member `Cargo.toml` and root `Cargo.lock` is explicitly listed. +- [ ] AC2: `recipe.json` produced by the new stage is identical to the one produced by the old full-tree copy stage (verified by diff). +- [ ] AC3: Full build pipeline (`docker build --target release .`) completes successfully with no regressions. +- [ ] AC4: Warm baseline run with a source-only change shows cook layers hitting cache; time saved is recorded. +- [ ] AC5: Containerfile contains an inline comment documenting the manifest list maintenance requirement. +- [ ] `linter all` exits with code `0` +- [ ] All CI checks pass for the changed `Containerfile` +- [ ] Manual verification scenarios are executed and documented (status + evidence) +- [ ] Acceptance criteria are re-reviewed after implementation and reflect actual behavior + +## Verification Plan + +Define verification before implementation starts and execute it before closing the issue. + +### Automatic Checks + +- `linter all` +- All CI checks pass for the changed `Containerfile` + +### Manual Verification Scenarios + +Status values: `TODO`, `IN_PROGRESS`, `DONE`, `FAILED`, `BLOCKED`. + +| ID | Scenario | Command/Steps | Expected Result | Status | Evidence | +| --- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------ | ---------------- | +| M1 | Diff recipe.json before and after | Build with old Containerfile; save `recipe.json`; build with new; diff both files. | Files are identical. | TODO | {diff output} | +| M2 | Full cold build succeeds | `docker build --target release --no-cache .` | All stages complete; release image produced. | TODO | {log path} | +| M3 | Warm build with source-only change | Edit a `.rs` file (no manifest change); run `./contrib/dev-tools/workflow-benchmarks/run-container-baseline.sh` warm run. | Cook stages show `CACHED` in BuildKit output; total warm build time significantly lower than cold. | TODO | {benchmark link} | +| M4 | Cook layer invalidated on Cargo.toml change | Edit a workspace `Cargo.toml` (add/remove a feature flag); warm run. | Cook stages are rebuilt (expected); confirm the invalidation is correct and deliberate. | TODO | {benchmark link} | + +### Acceptance Verification + +| AC ID | Status (`TODO`/`DONE`) | Evidence | +| ----- | ---------------------- | ---------------- | +| AC1 | TODO | {diff link} | +| AC2 | TODO | {diff link} | +| AC3 | TODO | {CI run link} | +| AC4 | TODO | {benchmark link} | +| AC5 | TODO | {diff link} | diff --git a/docs/issues/drafts/1840-workflow-performance-containerfile-target-scope/ISSUE.md b/docs/issues/open/1853-1840-workflow-performance-containerfile-target-scope/ISSUE.md similarity index 92% rename from docs/issues/drafts/1840-workflow-performance-containerfile-target-scope/ISSUE.md rename to docs/issues/open/1853-1840-workflow-performance-containerfile-target-scope/ISSUE.md index a83e7e8c6..8c603759f 100644 --- a/docs/issues/drafts/1840-workflow-performance-containerfile-target-scope/ISSUE.md +++ b/docs/issues/open/1853-1840-workflow-performance-containerfile-target-scope/ISSUE.md @@ -1,11 +1,11 @@ --- doc-type: issue issue-type: task -status: draft +status: open priority: p1 -github-issue: null -spec-path: docs/issues/drafts/1840-workflow-performance-containerfile-target-scope/ISSUE.md -branch: "{issue-number}-containerfile-target-scope" +github-issue: 1853 +spec-path: docs/issues/open/1853-1840-workflow-performance-containerfile-target-scope/ISSUE.md +branch: "1853-containerfile-target-scope" related-pr: null last-updated-utc: 2026-05-27 00:00 semantic-links: @@ -17,12 +17,12 @@ semantic-links: - .github/workflows/testing.yaml - docs/issues/open/1840-improve-pr-workflow-performance-epic/EPIC.md - docs/issues/open/1841-1840-workflow-performance-baseline-analysis/benchmark-results-baseline.md - - docs/issues/open/1726-reduce-build-times-sccache/ISSUE.md + - docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md --- -# Issue #[To be assigned] - Narrow Containerfile build targets to tracker image needs +# Issue #1853 - Narrow Containerfile build targets to tracker image needs ## Goal @@ -69,9 +69,9 @@ Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. ### Workflow Checkpoints -- [ ] Spec drafted in `docs/issues/drafts/` -- [ ] Spec reviewed and approved by user/maintainer -- [ ] GitHub issue created and issue number added to this spec +- [x] Spec drafted in `docs/issues/drafts/` +- [x] Spec reviewed and approved by user/maintainer +- [x] GitHub issue created and issue number added to this spec - [ ] (Optional, recommended for complex issues) Spec-only PR merged into `develop` before implementation - [ ] Implementation completed - [ ] Automatic verification completed (`linter all`, relevant tests, and any pre-push checks) @@ -86,6 +86,7 @@ Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. Append one line per meaningful update. - 2026-05-27 00:00 UTC - GitHub Copilot - Drafted Containerfile target-scope optimization issue from EPIC discussion - draft file created +- 2026-06-01 00:00 UTC - GitHub Copilot - GitHub issue #1853 created; spec moved from drafts/ to open/ ## Acceptance Criteria diff --git a/docs/issues/drafts/1840-workflow-performance-container-test-gating/ISSUE.md b/docs/issues/open/1854-1840-workflow-performance-container-test-gating/ISSUE.md similarity index 95% rename from docs/issues/drafts/1840-workflow-performance-container-test-gating/ISSUE.md rename to docs/issues/open/1854-1840-workflow-performance-container-test-gating/ISSUE.md index 395fbe97f..cab57d06c 100644 --- a/docs/issues/drafts/1840-workflow-performance-container-test-gating/ISSUE.md +++ b/docs/issues/open/1854-1840-workflow-performance-container-test-gating/ISSUE.md @@ -1,11 +1,11 @@ --- doc-type: issue issue-type: task -status: draft +status: open priority: p1 -github-issue: null -spec-path: docs/issues/drafts/1840-workflow-performance-container-test-gating/ISSUE.md -branch: "{issue-number}-container-test-gating" +github-issue: 1854 +spec-path: docs/issues/open/1854-1840-workflow-performance-container-test-gating/ISSUE.md +branch: "1854-container-test-gating" related-pr: null last-updated-utc: 2026-05-27 00:00 semantic-links: @@ -21,7 +21,7 @@ semantic-links: -# Issue #[To be assigned] - Evaluate test execution policy in container image build +# Issue #1854 - Evaluate test execution policy in container image build ## Goal @@ -74,9 +74,9 @@ Status values: `TODO`, `IN_PROGRESS`, `BLOCKED`, `DONE`. ### Workflow Checkpoints -- [ ] Spec drafted in `docs/issues/drafts/` -- [ ] Spec reviewed and approved by user/maintainer -- [ ] GitHub issue created and issue number added to this spec +- [x] Spec drafted in `docs/issues/drafts/` +- [x] Spec reviewed and approved by user/maintainer +- [x] GitHub issue created and issue number added to this spec - [ ] (Optional, recommended for complex issues) Spec-only PR merged into `develop` before implementation - [ ] Implementation completed - [ ] Automatic verification completed (`linter all`, relevant tests, and any pre-push checks) @@ -92,6 +92,7 @@ Append one line per meaningful update. - 2026-05-27 00:00 UTC - GitHub Copilot - Drafted issue to evaluate container-build test execution policy and alternatives - draft file created - 2026-05-27 00:00 UTC - GitHub Copilot - Expanded the issue to evaluate separation of validation from packaging targets - draft updated +- 2026-06-01 00:00 UTC - GitHub Copilot - GitHub issue #1854 created; spec moved from drafts/ to open/ ## Acceptance Criteria diff --git a/project-words.txt b/project-words.txt index 3301e1bf7..93d30e731 100644 --- a/project-words.txt +++ b/project-words.txt @@ -39,6 +39,7 @@ Bragilevsky bufs buildid Buildx +BuildKit byteorder callgrind CALLSITE @@ -167,6 +168,7 @@ Lphant lscr LVJDMDAwMDAwMDAwMDAwMDAwMDE matchmakes +Mbps Mebibytes metainfo middlewares @@ -239,6 +241,7 @@ randomised Rasterbar realpath reannounce +readelf recognised recompiles referer