Skip to content

test: increase number of tries to find tz not near midnight#4756

Merged
rjsparks merged 1 commit into
ietf-tools:mainfrom
painless-security:jennifer/more-tries
Nov 15, 2022
Merged

test: increase number of tries to find tz not near midnight#4756
rjsparks merged 1 commit into
ietf-tools:mainfrom
painless-security:jennifer/more-tries

Conversation

@jennifer-richards

Copy link
Copy Markdown
Member

This addresses a failure of the automated tests that occurred when testing the timezone_not_near_midnight() helper method.

The test operates by mocking the list of timezones available to 3 that should be rejected because they are close to midnight and 1 that should be selected. The test is repeated 5 times to limit the chance that the method could pass the test by picking one of its options at random.

It turns out that the infinte-loop prevention in timezone_not_near_midnight(), which allowed up to 20 guess-and-check attempts, leads to an expected failure of this test about once every 50 runs. That comes from 1 - (1-(3/4)**20)**5 ~ 0.015 as the probability of at least one of the 5 tests failing

This PR increases the number of attempts to 50 which should give a test failure every 10^5 tries or so.

None of this should affect use of the method itself, since about 11 of 12 real time zones will be outside the 2-hour rejection window. It's unlikely ever to need more than a few tries.

@rjsparks

Copy link
Copy Markdown
Member

Lets talk through again what the random selection (rather than a deterministic selection of known based on well-known timezone names and the offset of current) is buying us. Is it only the chance of early discovery of the effect of an unusual (or broken in the library) timezone that some user might be in (off by {n}h30m or the like)? Some comments on the benefit are probably useful to leave in the test.

@rjsparks
rjsparks merged commit 36bbc9b into ietf-tools:main Nov 15, 2022
@jennifer-richards
jennifer-richards deleted the jennifer/more-tries branch November 15, 2022 16:39
@jennifer-richards

Copy link
Copy Markdown
Member Author

I think the main thing is that it avoids baking in dependencies like we're seeing now, where running tests in the hours between UTC midnight and PST8PDT midnight cause test failures that we don't find until later because our pull requests are mostly generated in western hemisphere working hours.

We might do better to have it generate a synthetic time zone with a random UTC offset in the window we want. That would avoid the testing-the-library misfeature.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Nov 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants