fix: Exclude replaced documents from IESG discusses

[microamp]

Fixes #7179.

I'm retaining @pselkirk's commit from the PR, #7258.

I've updated the existing test, ietf.iesg.tests.IESGTests.test_feed, to ensure that replaced documents from IESG discusses are correctly filtered.

ietf/manage.py test ietf.iesg.tests.IESGTests.test_feed --settings=settings_test

...
Ran 7 tests in 18.513s

OK

[microamp]

Interesting observation:

possible_docs = possible_docs.exclude(states__type="draft", states__slug="repl")

adds a rather complex conditional to the existing query.

WHERE (
  -- the existing logic below
  (
    ("doc_state"."slug" IN ('lc', 'writeupw', 'goaheadw', 'iesg-eva', 'defer') AND "doc_state"."type_id" = 'draft-iesg')
    OR
    ("doc_state"."slug" IN ('intrev', 'iesgrev') AND "doc_state"."type_id" = 'charter')
    OR
    ("doc_state"."slug" IN ('iesgeval', 'defer') AND "doc_state"."type_id" IN ('statchg', 'conflrev'))
  )
  AND
  "name_ballotpositionname"."blocking"
  -- and with the new exclude filter below
  AND
  NOT (
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" U1 INNER JOIN "doc_state" U2 ON (U1."state_id" = U2."id")
      WHERE (U2."slug" = 'repl' AND U1."document_id" = ("doc_document"."id"))
      LIMIT 1
    )
    AND
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" U1 INNER JOIN "doc_state" U2 ON (U1."state_id" = U2."id")
      WHERE (U2."type_id" = 'draft' AND U1."document_id" = ("doc_document"."id"))
      LIMIT 1
    )
  )
)

I tried adding it to the main filter statement instead, but it generates a even more verbose one.

WHERE (
  -- the existing logic here
  AND
  NOT (
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" U1 INNER JOIN "doc_state" U2 ON (U1."state_id" = U2."id")
      WHERE (U2."slug" = 'repl' AND U1."id" = ("doc_document_states"."id") AND "doc_document_states"."document_id" = ("doc_document"."id"))
      LIMIT 1
    )
    AND
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" U1 INNER JOIN "doc_state" U2 ON (U1."state_id" = U2."id")
      WHERE (U2."type_id" = 'draft' AND U1."id" = ("doc_document_states"."id") AND "doc_document_states"."document_id" = ("doc_document"."id"))
      LIMIT 1
    )
  )
)

Let me know if I'm reading it wrong.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.79%. Comparing base (b00dfd3) to head (57fdba4).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7712      +/-   ##
==========================================
- Coverage   88.79%   88.79%   -0.01%     
==========================================
  Files         296      296              
  Lines       41319    41324       +5     
==========================================
+ Hits        36688    36692       +4     
- Misses       4631     4632       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rjsparks]

You aren't reading it wrong, but it points out that we are getting lucky with a pattern we are using to filter things with.

[rjsparks]

after discussion with Jennifer - we should change the exclude to work against a specific State (the form here works with .filter but not with .exclude.

[jennifer-richards]

The exclude() query is due to behavior documented in the second "Note" box here: https://docs.djangoproject.com/en/5.0/topics/db/queries/#spanning-multi-valued-relationships

The upshot is that filter(states__type="draft", states__slug="repl") will do what we want by combining the type and slug conditions with an AND. For exclude(), you have to add a second query and it should look like exclude(states__in=State.objects.filter(type="draft", slug="repl")

[microamp]

Replacing the existing exclude statement with

possible_docs = possible_docs.exclude(states__in=State.objects.filter(type="draft", slug="repl"))

generates a more compact query, but still has the NOT (EXISTS (... LIMIT 1)) pattern to it.

WHERE (
  -- the existing logic here
  AND
  NOT (
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" V1 WHERE (
        V1."state_id" IN (
          SELECT U0."id" FROM "doc_state" U0 WHERE (U0."slug" = 'replaced' AND U0."type_id" = 'draft')
        )
        AND
        V1."document_id" = ("doc_document"."id")
      )
      LIMIT 1
    )
  )
)

[microamp]

Here's the entire query after the change.

SELECT DISTINCT "doc_document"."id", "doc_document"."time", "doc_document"."type_id", "doc_document"."title", "doc_document"."stream_id", "doc_document"."group_id", "doc_document"."abstract", "doc_document"."rev", "doc_document"."pages", "doc_document"."words", "doc_document"."intended_std_level_id", "doc_document"."std_level_id", "doc_document"."ad_id", "doc_document"."shepherd_id", "doc_document"."expires", "doc_document"."notify", "doc_document"."external_url", "doc_document"."uploaded_filename", "doc_document"."note", "doc_document"."internal_comments", "doc_document"."rfc_number", "doc_document"."name", "name_streamname"."slug", "name_streamname"."name", "name_streamname"."desc", "name_streamname"."used", "name_streamname"."order", "group_group"."id", "group_group"."time", "group_group"."name", "group_group"."state_id", "group_group"."type_id", "group_group"."parent_id", "group_group"."description", "group_group"."list_email", "group_group"."list_subscribe", "group_group"."list_archive", "group_group"."comments", "group_group"."meeting_seen_as_area", "group_group"."used_roles", "group_group"."uses_milestone_dates", "group_group"."acronym", "group_group"."charter_id", "person_person"."id", "person_person"."user_id", "person_person"."time", "person_person"."name", "person_person"."plain", "person_person"."ascii", "person_person"."ascii_short", "person_person"."pronouns_selectable", "person_person"."pronouns_freetext", "person_person"."biography", "person_person"."photo", "person_person"."photo_thumb", "person_person"."name_from_draft"
FROM "doc_document"
INNER JOIN "doc_document_states" ON ("doc_document"."id" = "doc_document_states"."document_id")
INNER JOIN "doc_state" ON ("doc_document_states"."state_id" = "doc_state"."id")
INNER JOIN "doc_docevent" ON ("doc_document"."id" = "doc_docevent"."doc_id")
INNER JOIN "doc_ballotpositiondocevent" ON ("doc_docevent"."id" = "doc_ballotpositiondocevent"."docevent_ptr_id")
INNER JOIN "name_ballotpositionname" ON ("doc_ballotpositiondocevent"."pos_id" = "name_ballotpositionname"."slug")
LEFT OUTER JOIN "name_streamname" ON ("doc_document"."stream_id" = "name_streamname"."slug")
LEFT OUTER JOIN "group_group" ON ("doc_document"."group_id" = "group_group"."id")
LEFT OUTER JOIN "person_person" ON ("doc_document"."ad_id" = "person_person"."id")
WHERE (
  (
    ("doc_state"."slug" IN ('lc', 'writeupw', 'goaheadw', 'iesg-eva', 'defer') AND "doc_state"."type_id" = 'draft-iesg')
    OR
    ("doc_state"."slug" IN ('intrev', 'iesgrev') AND "doc_state"."type_id" = 'charter')
    OR
    ("doc_state"."slug" IN ('iesgeval', 'defer') AND "doc_state"."type_id" IN ('statchg', 'conflrev'))
  )
  AND
  "name_ballotpositionname"."blocking"
  AND
  NOT (
    EXISTS(
      SELECT 1 AS "a" FROM "doc_document_states" V1
      WHERE (
        V1."state_id" IN (
          SELECT U0."id" FROM "doc_state" U0 WHERE (U0."slug" = 'replaced' AND U0."type_id" = 'draft')
        )
        AND
        V1."document_id" = ("doc_document"."id")
      )
      LIMIT 1
    )
  )
)