Skip to content

fix: generate idnits2 rfc-status blob for 5-digit RFC numbers - #11621

Merged
jennifer-richards merged 1 commit into
ietf-tools:mainfrom
rjsparks:idnits2_rfc10k
Aug 21, 2026
Merged

fix: generate idnits2 rfc-status blob for 5-digit RFC numbers#11621
jennifer-richards merged 1 commit into
ietf-tools:mainfrom
rjsparks:idnits2_rfc10k

Conversation

@rjsparks

Copy link
Copy Markdown
Member

generate_idnits2_rfc_status() allocated a fixed 10000-element array and indexed it by RFC number, so it raised IndexError for any RFC above 10000. The task has been failing on every run since 2026-06-16, and because ietf/doc/tasks.py computes the blob outside its try block, the exception escapes before anything is written. The served file has been frozen at 9998 characters since then (content-length 10154), stale for all RFCs rather than only 5-digit ones.

This commit sizes the array from the highest rfc_number instead. The most recent versions of idnits2 (through 2.17.1) will correctly consume this larger array without modification.

It also stops the generator crashing on RFC rows it doesn't expect by excluding RFCs with a null rfc_number (int(None) raises TypeError) and falling back to 'U' for an unrecognised std_level_id (symbols[None] raises KeyError). Document.std_level is nullable, and a single such row would take down the whole task.

To allow existing idnits clients at version 2.17.1 and below to keep operating, override RFC 16 to 'O'. This deliberately contradicts both the datatracker and the RFC Editor, which record RFC 16 as updated rather than obsoleted. idnits2 validates its download of this file by matching the first 64 characters against a literal pattern asserting 'O' at position 16. The reason is lost, but it was likely the result of manual curation at tools.ietf.org long ago. Without the override, existing idnits2 clients discard the file as corrupt, fall back to whatever stale copy they have, and silently perform no RFC status checks at all. This is independent of the crash and predates it.

Note that the generator uses a floor of 6312. This is required because the RFC 16, RFC 200 and RFC 6312 workarounds write those offsets unconditionally, so the array must reach 6312 regardless of the data; without the floor the generator raises IndexError for any dataset whose highest RFC is below that. It also keeps output identical to the previous behaviour, where the fixed 10000-element array always had those offsets in range. Making the workaround writes conditional instead would remove the need for the floor, but that was not done here.

Verification:

  • Against the production snapshot, positions 1..9993 and the first line are byte-identical to the pre-change algorithm; the blob extends from 9999 to 10031. rfc10001='B', rfc10008='P', rfc10031='P' match their std_level_id values.
  • idnits2's own download validation (grep -qsE against the first line) now passes, where it fails against the file production serves today.
  • idnits2's lookup path resolves 5-digit statuses correctly against the
    generated file: rfc10001 -> Best Current Practice, rfc10031 ->
    Proposed Standard, rfc10032 -> past end of blob.
  • ietf.doc.tests (122 tests) and ietf.doc.tests_tasks ietf.doc.tests_downref (15 tests) pass.

This PR produced primarily by Claude.

generate_idnits2_rfc_status() allocated a fixed 10000-element array
and indexed it by RFC number, so it raised IndexError for any RFC
above 10000. The task has been failing on every run since 2026-06-16,
and because ietf/doc/tasks.py computes the blob outside its try block,
the exception escapes before anything is written. The served file has
been frozen at 9998 characters since then (content-length 10154),
stale for all RFCs rather than only 5-digit ones.

This commit sizes the array from the highest rfc_number instead. The
most recent versions of idnits2 (through 2.17.1) will correctly
consume this larger array without modification.

It also stops the generator crashing on RFC rows it doesn't expect by
excluding RFCs with a null rfc_number (int(None) raises TypeError) and
falling back to 'U' for an unrecognised std_level_id (symbols[None]
raises KeyError). Document.std_level is nullable, and a single such
row would take down the whole task.

To allow existing idnits clients at version 2.17.1 and below to keep
operating, override RFC 16 to 'O'. This deliberately contradicts both
the datatracker and the RFC Editor, which record RFC 16 as updated
rather than obsoleted. idnits2 validates its download of this file by
matching the first 64 characters against a literal pattern asserting
'O' at position 16. The reason is lost, but it was likely the result
of manual curation at tools.ietf.org long ago. Without the override,
existing idnits2 clients discard the file as corrupt, fall back to
whatever stale copy they have, and silently perform no RFC status
checks at all. This is independent of the crash and predates it.

Note that the generator uses a floor of 6312. This is required because
the RFC 16, RFC 200 and RFC 6312 workarounds write those offsets
unconditionally, so the array must reach 6312 regardless of the data;
without the floor the generator raises IndexError for any dataset
whose highest RFC is below that. It also keeps output identical to the
previous behaviour, where the fixed 10000-element array always had
those offsets in range. Making the workaround writes conditional
instead would remove the need for the floor, but that was not done
here.

Verification:

- Against the production snapshot, positions 1..9993 and the first line
  are byte-identical to the pre-change algorithm; the blob extends from
  9999 to 10031. rfc10001='B', rfc10008='P', rfc10031='P' match their
  std_level_id values.
- idnits2's own download validation (grep -qsE against the first line)
  now passes, where it fails against the file production serves today.
- idnits2's lookup path resolves 5-digit statuses correctly against the
  generated file: rfc10001 -> Best Current Practice, rfc10031 ->
  Proposed Standard, rfc10032 -> past end of blob.
- ietf.doc.tests (122 tests) and ietf.doc.tests_tasks
  ietf.doc.tests_downref (15 tests) pass.

This commit produced primarily by Claude.
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.72%. Comparing base (e756948) to head (5df4bf9).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #11621   +/-   ##
=======================================
  Coverage   88.71%   88.72%           
=======================================
  Files         335      335           
  Lines       45256    45258    +2     
=======================================
+ Hits        40151    40153    +2     
  Misses       5105     5105           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jennifer-richards
jennifer-richards merged commit 5a88e9e into ietf-tools:main Aug 21, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants