Have integration tests run with both scheduler and job queue #6787

jonathan-eq · 2023-12-12T09:30:54Z

Issue
Resolves #6756

Add pytest scheduler custom mark and fixture, indicating that an integration test will be ran with both old job queue and new scheduler.
The commit in this PR also moves all integration tests (atleast those marked with integration_test) to a new integration test directory.

tests/conftest.py

berland · 2023-12-13T11:29:43Z

Recommend to do pre-commit install in your local project folder, and it will be harder to miss on style issues :)

berland · 2023-12-13T11:36:33Z

The commit in this PR also moves all integration tests (atleast those marked with integration_test) to a new integration test directory.

Should the moving around of tests be in a separate commit?

jonathan-eq · 2023-12-13T11:39:52Z

I only moved parts of the files, but that is by far the main part of the commit. I think I would reword the commit message instead

berland · 2023-12-13T13:21:01Z

For the tests that use the pytest snapshot feature, you need to git mv the snapshot files as well.

pyproject.toml

berland · 2023-12-14T07:53:04Z

tests/conftest.py

@@ -245,6 +245,20 @@ def excepthook(cls, exc, tb):
    monkeypatch.setattr(sys, "excepthook", excepthook)


+@pytest.fixture(params=[False, True])
+def try_queue_and_scheduler(request, monkeypatch):


For some reason, the test that is being decorated with this fixture also needs to specify the monkeypatch fixture. Don't know why (otherwise it will silently ignore the request to enable scheduler).

berland · 2023-12-14T12:28:32Z

tests/integration_tests/analysis/test_es_update.py

 @pytest.mark.integration_test
 def test_that_surfaces_retain_their_order_when_loaded_and_saved_by_ert(
-    copy_case
+    copy_case, try_queue_and_scheduler


monkeypatch needed here?

I forgot to commit the changes...

berland · 2023-12-14T12:59:01Z

Still two snapshots missing?

xjules · 2023-12-15T08:02:31Z

There are snapshots to be updated @jonathan-eq, but very good work nevertheless!

jonathan-eq · 2023-12-15T08:04:21Z

Yes, it seems they updated ES-MDA so I will wait until they update the snapshots

codecov-commenter · 2023-12-19T12:03:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (1221607) 83.86% compared to head (a98c8be) 83.89%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6787      +/-   ##
==========================================
+ Coverage   83.86%   83.89%   +0.03%     
==========================================
  Files         365      365              
  Lines       21440    21440              
  Branches      948      948              
==========================================
+ Hits        17980    17988       +8     
+ Misses       3166     3158       -8     
  Partials      294      294

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

berland

LGTM

This commit moves all integration tests (the ones marked with pytest.mark.integration_tests atleast) to a new directory tests/integration_tests.

This commits adds the pytest.mark.scheduler mark and scheduler fixture to some of the integration tests, so that they will be ran with both the scheduler and job queue.

jonathan-eq commented Dec 12, 2023

View reviewed changes

tests/conftest.py Outdated Show resolved Hide resolved

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch 9 times, most recently from 83f0daf to 40a8c94 Compare December 13, 2023 11:16

jonathan-eq changed the title ~~Add pytest scheduler custom mark~~ Have integration tests run with both scheduler and job queue Dec 13, 2023

jonathan-eq self-assigned this Dec 13, 2023

jonathan-eq marked this pull request as ready for review December 13, 2023 11:18

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch 2 times, most recently from d965583 to 73d6c6d Compare December 13, 2023 11:28

berland mentioned this pull request Dec 13, 2023

Flaky test_ensemble_legacy with scheduler #6800

Closed

berland reviewed Dec 14, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

berland reviewed Dec 14, 2023

View reviewed changes

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch 4 times, most recently from b93ade0 to b0ef19e Compare December 14, 2023 08:34

berland mentioned this pull request Dec 14, 2023

Test ensemble_evaluator with new scheduler #6803

Merged

8 tasks

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch 3 times, most recently from ab0b359 to dff5199 Compare December 14, 2023 11:16

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch from dff5199 to 1660d9c Compare December 14, 2023 12:04

berland reviewed Dec 14, 2023

View reviewed changes

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch from 1660d9c to 2f19c04 Compare December 14, 2023 12:41

jonathan-eq mentioned this pull request Dec 19, 2023

Mark status tests for testing with Scheduler #6819

Merged

8 tasks

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch from c006aab to 3c8e24b Compare December 19, 2023 11:50

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch from 3c8e24b to 966a609 Compare December 19, 2023 12:29

berland approved these changes Dec 20, 2023

View reviewed changes

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch 5 times, most recently from d047b62 to a98c8be Compare December 20, 2023 13:51

jonathan-eq added 3 commits December 20, 2023 15:07

Move integration tests to separate directory

726ce03

This commit moves all integration tests (the ones marked with pytest.mark.integration_tests atleast) to a new directory tests/integration_tests.

Have integration tests run with both queue and scheduler

f1f6919

This commits adds the pytest.mark.scheduler mark and scheduler fixture to some of the integration tests, so that they will be ran with both the scheduler and job queue.

Move pytest snapshots down one level

83134a0

jonathan-eq force-pushed the move-integration-tests-for-new-queue branch from a98c8be to 83134a0 Compare December 20, 2023 14:08

jonathan-eq enabled auto-merge (squash) December 20, 2023 14:09

jonathan-eq added the release-notes:skip If there should be no mention of this in release notes label Dec 20, 2023

jonathan-eq merged commit a810eb0 into equinor:main Dec 20, 2023
43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Have integration tests run with both scheduler and job queue #6787

Have integration tests run with both scheduler and job queue #6787

jonathan-eq commented Dec 12, 2023 •

edited

Loading

berland commented Dec 13, 2023

berland commented Dec 13, 2023

jonathan-eq commented Dec 13, 2023

berland commented Dec 13, 2023

berland Dec 14, 2023

berland Dec 14, 2023

jonathan-eq Dec 14, 2023

berland commented Dec 14, 2023

xjules commented Dec 15, 2023 •

edited

Loading

jonathan-eq commented Dec 15, 2023

codecov-commenter commented Dec 19, 2023 •

edited

Loading

berland left a comment

Have integration tests run with both scheduler and job queue #6787

Have integration tests run with both scheduler and job queue #6787

Conversation

jonathan-eq commented Dec 12, 2023 • edited Loading

berland commented Dec 13, 2023

berland commented Dec 13, 2023

jonathan-eq commented Dec 13, 2023

berland commented Dec 13, 2023

berland Dec 14, 2023

Choose a reason for hiding this comment

berland Dec 14, 2023

Choose a reason for hiding this comment

jonathan-eq Dec 14, 2023

Choose a reason for hiding this comment

berland commented Dec 14, 2023

xjules commented Dec 15, 2023 • edited Loading

jonathan-eq commented Dec 15, 2023

codecov-commenter commented Dec 19, 2023 • edited Loading

Codecov Report

berland left a comment

Choose a reason for hiding this comment

jonathan-eq commented Dec 12, 2023 •

edited

Loading

xjules commented Dec 15, 2023 •

edited

Loading

codecov-commenter commented Dec 19, 2023 •

edited

Loading