Improve testing, coverage and benchmarking

[originalsouth]

User Story

This a meta ticket meant as a discussion tracker with the aim to improve testing, coverage and benchmarking of KAT.

The scope of this ticket ought to be refined, but let us have a start here with some ideas.

Acceptance Criteria:

• Measure and improve coverage of unit tests as part of the pipeline
• Allow KAT states to be cross-platform reproducible (serialization)
  • Introduce reproducible system tests (without using the GUI)
  • Allow bug reporters to commit a "state dump"
• Measure performance of code
  • Benchmark key functions
  • Stress test the system
  • Measure and improve performance per commit/branch as part of the pipeline

[underdarknl]

I'd love to move all normalizer tests to their respective normalizer folders, and make them into a form of input->output files where possible so we could use 1 single python runner to run all normalizer tests. This would remove a lot of code, and would make it easier for devs to create new tests.

[Donnype]

Some thoughts I have to improve testing, coverage and benchmarking.

Unit tests

We can improve both the coverage of the code and standardize our tooling/methods. My thoughts include:

• Use pytest and fixtures everywhere so we can remove the TestCase leftovers.
• Use coverage (perhaps xdist) tools around pytest in every module.
• Set a coverage target: make it a priority for modules to have at least X% coverage and schedule time to improve lacking modules.
• Try to remove as much monkeypatching as possible. In general I see mocker.patch as a red flag that code is not very modular. Also it's just a pain to debug if for some reason your class doesn't get patched.
• Have Github report on coverage on PR's through a coverage action

Integration tests

Some setup has been done, but we can improve our integration testing a lot still.

• Have a set of generic tests for each service where they talk to all relevant related services. Currently a few modules already have some version of a make itest available. We should standardize these make targets across modules.

Robot Framework

I think we should remove the Robot tests from Octopoes, and replace them with pytest if necessary.

End-to-end testing

Since we have the built containers, we can look into how we want to tests functionality across all services.

• We should be able to create a working kat instance in the CI that we can run robot tests on.
• There are several flows in the QA template that can be covered through end-to-end tests using robot in that case.

QA and Staging

Since we have the test server available, let's start using it more often!

• We can consider having a CI process that deploys pull request branches to the test server. We already build container images for services that changes, so it's probably not too much work to deploy those. This could speed up the QA process, but we should think about CI efficiency as well here. ("How do I prevent a deploy for this trivial hotfix PR?")
• We should review the QA steps in the documentation again and improve them, especially the process of doing QA on a release branch. We could structurally use the test server for this as well.
• We should test upgrade plans as well. This means setting up a reasonable involved install and performing an update, migration etc. on this to see if the changes are backward compatible on old data.

Benchmarking

I made a setup for a benchmark tests in Rocky outputting pstat files, but:

• We should think about when we want to run benchmarks. Each PR? In a release branch?
• We should also consider which flows, functions etc. we want to benchmark.
• We can create a Github action that reports on these benchmarks. (We should look into if there are other output formats we need to be using in that case.)

But we should be careful not making this too noisy and annoying, else we'll just turn it off after a week or two.

[originalsouth]

Perhaps we can use https://www.selenium.dev/ for system tests.

[madelondohmen]

Ideas for unit testing

Making the unit tests more generic and create a template with instructions, so it will be easier for the community to contribute to unit testing.

Step 1:

• Create an easy and understandable template for unit testing
• This template can then be copied every time someone is writing unit tests.
• It contains variables, which can be filled in or left empty. (Note: it has to be super clear which variables have to be filled in for which unit test.)
• The user also needs to fill in the expected result in JSON.
• At the end of the file, another file that executes the unit tests will be called (with all the variables and the expected output as input) (see point 2).

Step 2:

• Create a general file that executes the logic in the unit test
• This file checks the given input (the variables) and executes the test.
• It then checks if the expected output is the same as the real output.

Notes

I think it will be hard to make al the unit tests generic, because the logic is different. For the unit tests of the reports this can be a good solution, because they all follow the same flow and contain the same logic. But the other tests contain different logic and I can't think of an easy solution for that.

Solution 1

Do step 1 and step 2 for all different types of unit tests (i.e. reports, rascals, normalizers, etc.) because they all contain different logic.

Solution 2

Skip both steps (leave the unit tests as they are) and create documentation for each type of unit test, explaining how unit tests should be written.

[ammar92]

Improve Rocky tests:

• Replace mocked HTTP calls with pytest-httpx (depends on Replace requests with httpx #2576)
  • This modification aims to reduce the amount of mocking code, making the tests more comprehensible and minimizing redundancy

[madelondohmen]

What about using SonarCloud?
https://www.sonarsource.com/products/sonarcloud/

[dekkers]

As far as I understood sonarcloud is going to get rolled out within iRealisatie in April and we will also be able to make use of it.

[dekkers]

Triggered when reviewing javascript code in #2624: Test our javascript code #2663