Overhaul test suite with fixtures #2675
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kudos, SonarCloud Quality Gate passed!
|
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
1d4791a to
b8a3582
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
b8a3582 to
da5c940
Compare
|
This was referenced Sep 12, 2024
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
da5c940 to
0edf1f0
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## test/fix-skip-issue #2675 +/- ##
========================================================
- Coverage 60.47% 38.14% -22.33%
========================================================
Files 603 603
Lines 43771 43777 +6
Branches 48 48
========================================================
- Hits 26469 16700 -9769
- Misses 17290 27065 +9775
Partials 12 12 ☔ View full report in Codecov by Sentry. |
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
0edf1f0 to
57c8096
Compare
|
These items have been factored out as separate PRs and this PR has been rebased on the latest master:
|
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
57c8096 to
7e7ea25
Compare
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
7e7ea25 to
94d9e4e
Compare
|
Extracted everything to do with skip logic issues to #3015 and rebased this on that. |
lunkwill42
force-pushed
the
test/fix-skip-issue
branch
2 times, most recently
from
bf85f9f to
936aa34
Compare
This creates a PostgreSQL fixture that can be depended on by tests that require a running PostgreSQL server with the NAV database to be present. The fixture will work with externally set up PostgreSQL servers, like in the case of GitHub Actions, while it can also provision a PostgreSQL server with the NAV schema using Docker if run locally.
This ensures any tests that rely on the various pre-existing db-related fixtures will have a running PostgreSQL server available.
This ensures the postgres fixture is started for all tests that inherit from DjangoTransactionTestCase
Change TestCase implementations that should really be transactional test cases (which in turn ensures they have PostgreSQL available through the postgres fixture, and that transactions are properly rolled back after the tests have run)
Declares a dependency on the postgresql fixture to ensure the NAV database is actually available for these tests
This ensures that any test that depends on a logged in web client also automatically depends on having the NAV database running.
This ensure the snmpsim fixture can run both on Linux and MacOS, by replacing Linux specific parts (like looking at /proc/net/udp). This slightly changes the logic of verifying whether the snmpsimd process is up and running before yielding the fixture value: Instead of waiting for something to appear to listen to port 1024, this verifies the presence of the SNMP agent by performing an actual SNMP query.
These imports made the entire test module depend on the PostgreSQL database actually running. Moving the imports to be local to the tests (that depend directly or indirectly) on the postgresql fixture ensures that the imports only take place after PostgreSQL is actually running.
The admin_navlet fixture is dynamic/parametrized, which is why its implementation is a bit weird. Unfortunately, the tests that depend on it currently fail under the new regime of the postgresql fixture. Adding this note so we know where to go back and fix it.
While pytest can accomplish a lot of exciting things, it cannot use fixtures as input to test parametrization. While can make a test depend on a fixture for getting a postgresql instance, the test discovery phase that generates the tests cannot. I.e. we cannot get our test parameters from a database unless the database was already up and running when the test discovery phase started. Sad, but true,. This changes the navlet test to iterate over the available admin navlets itself.
These tests would only work incidentally; i.e. if they ran after another test that depended on the postgresql fixture
The postgresql fixture is fine to depend on if you just need a database to be present. But if you need to write data during the test, you should probably use the db fixture.
For now, at least. If the Open-LDAP libraries aren't available locally, the test suite will fail because python-ldap will not build.
lunkwill42
force-pushed
the
test/fixture-experiments
branch
from
94d9e4e to
ee65d2c
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has an overarching goal of making the test suite easier to run locally, rather then requiring devs to run all tests inside a very specifically defined Docker container (which is NOT how the test suite is run on GitHub actions).
Goals:
conftest.py, such as gunicorn, PostgreSQL, Xvfb, while expecting that these are available in the local environment, these services should be defined as test fixtures. Any test that depends on PostgreSQL should declare a dependency to this fixture, in order for it to be able to use PostgreSQL.Some of the changes herein would probably stand to be factored out into separate PRs, as there are many fixes to the existing test suite that are not dependent on the fixture changes made.