Skip to content

Add script for scan-based summary backfill - #7444

Merged
FestiveKyle merged 2 commits into
masterfrom
summaries/backfill-new-formats
Jul 21, 2026
Merged

Add script for scan-based summary backfill#7444
FestiveKyle merged 2 commits into
masterfrom
summaries/backfill-new-formats

Conversation

@FestiveKyle

@FestiveKyle FestiveKyle commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

The summaries job only recently started producing the psd/pgs/all scopes so there are years of missing history. This adds a one-off backfill that reconstructs those summaries for every past day.

  • New scan_summaries.py, run as a K8s job, rebuilds chart + org summaries directly from the raw dns/web scans (not the denormalized domain.status), so history can be reconstructed for any date.
  • Writes to shadow collections chartSummaries_rebuild / organizationSummaries_rebuild, prod collections are untouched. (--target prod exists but isn't used here.)
  • Carries each domain's status forward per scan-type, matching the live job's domain.status semantics.

After ensuring the new summaries data aligns with the previously generated data, we can switch to use --target prod to move to the prod collections. At that point we should also merge the logic of scan_summaries.py with summaries.py.

@FestiveKyle
FestiveKyle marked this pull request as ready for review July 20, 2026 17:53
@FestiveKyle
FestiveKyle requested a review from lcampbell2 July 20, 2026 17:53
@FestiveKyle
FestiveKyle merged commit edc2b4f into master Jul 21, 2026
38 checks passed
@FestiveKyle
FestiveKyle deleted the summaries/backfill-new-formats branch July 21, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants