ci: Switch to using geckodriver by larseggert · Pull Request #6541 · ietf-tools/datatracker

larseggert · 2023-10-25T17:23:27Z

Because that works also on amd64.

codecov · 2023-10-25T18:56:15Z

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (4500642) 88.82% compared to head (90646ff) 88.83%.
Report is 10 commits behind head on main.

Files	Patch %	Lines
ietf/utils/jstest.py	50.00%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6541   +/-   ##
=======================================
  Coverage   88.82%   88.83%           
=======================================
  Files         285      285           
  Lines       40230    40224    -6     
=======================================
- Hits        35734    35732    -2     
+ Misses       4496     4492    -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…racker into ci-selenium-geckodriver

jennifer-richards

Looks ok to me (and tests run, though with a couple failures, on my m2 mac)

jennifer-richards

Withdrew my approval - noticed that the geckodriver tests didn't actually run here. I think it pulls the standard base image, so doesn't have geckodriver installed. Testing locally, there are some changes needed.

The inputs are hidden and managed by JS, so click the visible elements instead.

Waiting for the button to be clickable does not work because the modal is still fading in, so does not actually close. Would be better to check for it responding, but I haven't found the right way to do that yet.

jennifer-richards · 2023-10-26T21:09:36Z

Those commits fix the selenium tests under ietf.meeting.tests_js for me (apologies for the sleep() in the second one, but I haven't found a better way)

jennifer-richards · 2023-10-26T22:18:26Z

~~Still a failure in ietf.doc.tests_js.EditAuthorsTests.test_add_author_forms~~

With the last commit, all tests pass for me.

Sending '\n' does not seem to work as it did before, so click the option instead. Also reverted some fixme hacks that seem to be obe.

Selenium tests didn't run

rjsparks

Please move this to draft until it passes in both CI and on both mac and windows developer's platforms

larseggert · 2023-10-27T15:31:40Z

You need to rebuild the base docker image locally first to have geckodriver available in the dev image.

jennifer-richards · 2023-10-27T15:53:26Z

You need to rebuild the base docker image locally first to have geckodriver available in the dev image.

That's what I've done locally (mac). Can you check whether the tests I touched work on your system? (I'm a little worried that I may have undone workarounds that are platform dependent in my purported fixes)

jennifer-richards · 2023-10-30T19:13:52Z

For me (m2 mac, geckodriver 0.33.0 (a80e5fd61076 2023-04-02 18:31 +0000)):

dev ➜ /workspace $ ietf/manage.py test --settings=settings_test --skip-coverage ietf.meeting.tests_js.InterimTests
     Datatracker 11.0.0-dev test suite, 30 October 2023 12:05:35 PDT:
     Python 3.9.18 (main, Oct 12 2023, 02:21:43)  [GCC 10.2.1 20210110].
     Django 4.2.6, settings 'settings_test'
     Changing TEMPLATES[0]['OPTIONS']['string_if_invalid'] to '' during testing
     Changing INTERNAL_IPS to '[]' during testing.
     Running an SMTP test server on 127.0.0.1:2026 to catch outgoing email.
     Saving factory-boy random state to .factoryboy_random_state
     Validating all HTML generated during the tests
Found 12 test(s).
     Creating test database...
     Loading global test fixtures: names, ietf.utils.test_data.make_immutable_base_data, nomcom_templates, proceedings_templates
System check identified no issues (3 silenced).
............
----------------------------------------------------------------------
Ran 12 tests in 90.344s

OK

I previously ran the whole suite (without the --skip-coverage) without failure as well. Maybe it's a time-of-day or date-related thing? Or something to do with orbstack vs docker? (seems unlikely)

larseggert · 2023-11-04T13:35:11Z

Tests passed now...

jennifer-richards · 2023-11-04T15:18:13Z

Tests are still passing for me on my M2. Still have the bootstrapping problem for running the selenium tests on github CI, though.

NGPixel · 2023-11-08T03:25:30Z

geckodriver added to the base image.

rjsparks · 2023-11-12T22:45:57Z

What can we do to get insight into why geckodriver is crashing in the test container?

NGPixel · 2023-11-13T01:37:56Z

@rjsparks Use the tests debug script:

Clone the PR locally (gh pr checkout 6541)
cd to dev/tests
run sh debug.sh
after init, run:

./ietf/manage.py check
./ietf/manage.py migrate --fake-initial
./ietf/manage.py test -v2 --validate-html-harder --settings=settings_test

While this will run in the same environment setup as GHA, including the container being limited to 2 cores and 7GB of RAM, you'll be running in arm64, while GHA is using x64. So if you don't get the errors, a x64 environment would be needed to properly reproduce.

I was not able to reproduce the crash on my side...

larseggert · 2023-11-13T07:49:20Z

What's in geckodriver.log when it crashes?

NGPixel · 2023-11-13T10:07:49Z

@larseggert https://github.com/ietf-tools/datatracker/suites/18134236487/artifacts/1045634129

The recurring error being:

1699868894816	mozrunner::runner	INFO	Running command: MOZ_CRASHREPORTER="1" MOZ_CRASHREPORTER_NO_REPORT="1" MOZ_CRASHREPORTER_SHUTDOWN="1" MOZ_NO_REMOTE="1" "/github/home/.cache/selenium/firefox/linux64/119.0.1/firefox" "--marionette" "--headless" "--remote-debugging-port" "51209" "--remote-allow-hosts" "localhost" "-no-remote" "-profile" "/tmp/rust_mozprofileOPgBo4"
Running Firefox as root in a regular user's session is not supported.  ($HOME is /github/home which is owned by uid 1001.)

NGPixel · 2023-11-13T11:02:42Z

Adding HOME=/root before running the tests appears to be working.

This is a long standing firefox specific bug that they can't be bothered to fix.

larseggert changed the title ~~ci: Switch to using geckdriver~~ ci: Switch to using geckodriver Oct 25, 2023

larseggert added 5 commits October 25, 2023 10:48

Switch to using geckodriver

ce08a57

Switch to selenium 4

00765eb

Undo

36e82f6

Remove comment

53a071a

Fixes

75c9868

larseggert force-pushed the ci-selenium-geckodriver branch from 6de85b5 to 75c9868 Compare October 25, 2023 18:25

larseggert added 2 commits October 25, 2023 11:27

Restore non-standard line endings to minimize diff

371571b

Undo

b9f1652

Merge branch 'main' into ci-selenium-geckodriver

d7765a5

larseggert marked this pull request as ready for review October 25, 2023 22:24

larseggert added 2 commits October 25, 2023 15:25

Remove comment

c4df0bf

Merge branch 'ci-selenium-geckodriver' of github.com:larseggert/datat…

2d5c56e

…racker into ci-selenium-geckodriver

rjsparks requested review from NGPixel and jennifer-richards October 25, 2023 22:32

jennifer-richards previously approved these changes Oct 26, 2023

View reviewed changes

jennifer-richards self-requested a review October 26, 2023 20:18

jennifer-richards reviewed Oct 26, 2023

View reviewed changes

jennifer-richards added 2 commits October 26, 2023 17:45

test: Fix test_upcoming_view_time_zone_selection

21e54c0

The inputs are hidden and managed by JS, so click the visible elements instead.

test: Clumsy fix to test_upcoming_materials_modal

228e32a

Waiting for the button to be clickable does not work because the modal is still fading in, so does not actually close. Would be better to check for it responding, but I haven't found the right way to do that yet.

test: Fix test_add_author_forms

70e2e2c

Sending '\n' does not seem to work as it did before, so click the option instead. Also reverted some fixme hacks that seem to be obe.

rjsparks requested changes Oct 27, 2023

View reviewed changes

larseggert marked this pull request as draft October 28, 2023 00:29

Merge branch 'main' into ci-selenium-geckodriver

b5e6b4d

larseggert added 2 commits October 31, 2023 15:29

Merge branch 'main' into ci-selenium-geckodriver

62d8836

Merge branch 'main' into ci-selenium-geckodriver

1364678

larseggert marked this pull request as ready for review November 4, 2023 13:35

larseggert requested review from jennifer-richards and rjsparks November 4, 2023 13:35

NGPixel added 2 commits November 7, 2023 22:21

Merge branch 'main' into ci-selenium-geckodriver

0f18f1f

ci: Update base.Dockerfile

06a576f

NGPixel and others added 3 commits November 8, 2023 14:29

Merge branch 'main' into ci-selenium-geckodriver

e2c6deb

Merge branch 'main' into ci-selenium-geckodriver

63775f2

Merge branch 'main' into ci-selenium-geckodriver

7fd4d88

test: add resource limits to dev/tests/debug.sh env

2234a00

ci: add upload of geckodriver.log on failure

e24abdc

NGPixel added 3 commits November 13, 2023 05:14

ci: run tests as user 1001

92ba7bc

ci: run app-create-dirs as sudo

22f6673

ci: set home folder to root to run tests

fd7b088

NGPixel approved these changes Nov 21, 2023

View reviewed changes

Merge branch 'main' into ci-selenium-geckodriver

90646ff

rjsparks approved these changes Nov 21, 2023

View reviewed changes

rjsparks merged commit 3d44825 into ietf-tools:main Nov 21, 2023

github-actions Bot locked as resolved and limited conversation to collaborators Nov 25, 2023

larseggert deleted the ci-selenium-geckodriver branch December 18, 2023 08:16

Uh oh!

Conversation

larseggert commented Oct 25, 2023

Uh oh!

codecov Bot commented Oct 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jennifer-richards left a comment

Choose a reason for hiding this comment

Uh oh!

jennifer-richards left a comment

Choose a reason for hiding this comment

Uh oh!

jennifer-richards commented Oct 26, 2023

Uh oh!

jennifer-richards commented Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjsparks left a comment

Choose a reason for hiding this comment

Uh oh!

larseggert commented Oct 27, 2023

Uh oh!

jennifer-richards commented Oct 27, 2023

Uh oh!

jennifer-richards commented Oct 30, 2023

Uh oh!

larseggert commented Nov 4, 2023

Uh oh!

jennifer-richards commented Nov 4, 2023

Uh oh!

NGPixel commented Nov 8, 2023

Uh oh!

rjsparks commented Nov 12, 2023

Uh oh!

NGPixel commented Nov 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larseggert commented Nov 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NGPixel commented Nov 13, 2023

Uh oh!

NGPixel commented Nov 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Oct 25, 2023 •

edited

Loading

jennifer-richards commented Oct 26, 2023 •

edited

Loading

NGPixel commented Nov 13, 2023 •

edited

Loading

larseggert commented Nov 13, 2023 •

edited

Loading