Automatic Testing
In the QED.jl eco-system, we use Continuous integration (CIs) for running automatic tests. Each time, if a pull request is opened or a change is committed to an open pull request, the CI pipeline is triggered and starts a test script. The result of the tests will be reported in the pull request. In QED.jl, we distinguish between two kinds of tests
- Unit tests: tests which check the functionality of the modified (sub) package. Those tests can either run standalone or use the functionality of other (third-party) packages.
- Integration tests: tests which check the interoperatebility of the modified package with all sub-packages. For example, if package
Ais dependent on packageBand you change packageB, the integration test will check, if packageAstill works with the new version of packageB.
The CI will first execute the unit tests. Then, if all unit tests are passed, the integration tests will be started.
Our CI uses the GitLab CI because it allows us to use CI resources provided by HIFIS, where we can strong CPU runners and special runners, like Nvidia and AMD GPUs.
Unit Test for CI Users
The unit tests are automatically triggered, if you open a pull request, which target the main or dev branch. Before the tests start, the CI setup the test environment. Therefore it takes the Project.toml of a project and add the latest development version of each QED dependency. Other dependencies will be regular resolved by the Julia Package manager. Afterwards the unit tests will be executed.
You can also modify, which version of a QED dependency should be used. For example you need to test your code with a function, which is not merged in the development branch yet. Therefore you need to add specific line to your commit message with the following format:
CI_UNIT_PKG_URL_<dep_name>: https://github.com/<user>/<dep_name>#<commit_hash>You find the <dep_name> name in the Project.toml of the dependent project. For an example let's assume the name of the dependent package is depLibrary. The url of the fork is https://github.com/user/depLibrary.jl and the required feature has the commit sha 45a753b.
This commit extends function foo with a new
-function argument.
-
-The function argument is required to control
-the new functionality.
-
-If you pass a 0, it has a special meaning.
-
-CI_UNIT_PKG_URL_depLibrary: https://github.com/user/depLibrary.jl#45a753bIt is also possible to set a custom URL for more than one package. Simple add an additional line with the shape of CI_UNIT_PKG_URL_<dep_name>: https://github.com/<user>/<dep_name>#<commit_hash> to the commit message.
You don't need to add a new commit to set custom URLs. You can modify the commit message with git commit --amend and force push to the branch. This also starts the CI pipeline again.
Integration Test for CI Users
The integration tests are automatically triggered, if you open a pull request, which target the main or dev branch, and the unit tests passed. The integration tests itself are in an extra stage of the CI.
If the tests pass successfully, you don't need to do something. If the fails, i.e. the change breaks the interoperation with another package in the eco-system, the pull request will suspend, and one has two options to proceed:
- One can solve the problem locally, by changing the code of the modified (sub) package. The workflow is the same, like fixing unit tests.
- One need to modify the depending package, which failed in the integration test. In the following, we describe how to provide the necessary changes to the downstream package and makes the CI pass the integration tests, which will resume the pull request.
For better understanding, the package currently modified by the pull request is called orig, and the package, which depends on it is referred to as dep. First, one should fork the package dep and checkout a new feature branch on this fork. The fix of the integration issue for dep is now developed on the feature branch. Once finished, the changes to dep are push to GitHub, and pull request on dep is opend to check in the changes. By default, the unit test for dep should fail, because the CI in dep needs to use the modified version of orig. The solution for this problem is explained in the section Unit Test for CI Users. Using this, one should develop the fix on the feature branch until the CI of dep passes all unit tests. In this case, original pull request in the upstream package orig can be resumed. Therefore, one needs to tell the CI of orig that the integration tests should use fixed version package dep, which is still on the feature branch in a pull request on dep. In order to proceed, the CI on orig needs the information, where the fix for dep is lying. This is given to the CI of orig in a commit message on the origin branch of the pull request on orig, one just needs to add a new line with the following format to the commit message:
CI_INTG_PKG_URL_<dep_name>: https://github.com/<user>/<dep_name>#<commit_hash>You find the names of the environment variables in the dict package_infos in the jobGenerator/src/jobGenerator.jl. For an example let's assume the name of the dep package is dep1.jl, user1 forked the package and the commit hash of the fix for package dep1.jl is 45a723b. Then, an example message could look like this:
This commit extends function foo with a new
-function argument.
-
-The function argument is required to control
-the new functionality.
-
-If you pass a 0, it has a special meaning.
-
-CI_INTG_PKG_URL_DEP1JL: https://github.com/user1/dep1.jl#45a723bIt is also possible to set a custom URL for more than one package, which depends on orig. Simple add an additional line with the shape of CI_INTG_PKG_URL_<dep_name>: https://github.com/<user>/<dep_name>#<commit_hash> to the commit message.
You don't need to add a new commit to set custom URLs. You can modify the commit message with git commit --amend and force push to the branch. This also starts the CI pipeline again.
Integration Test for CI Develops
This section explains how the integration tests are created and executed. It is not mandatory to read the section if you only want to use the CI. The following figure shows the stages of the CI pipeline:
All of the following stages are executed in the CI pipeline of the sub-package, where the code is modified. This means also, that GitLab CI automatically checks out the repository with the changes of the pull request and provides it in the CI job. We name the package orig for easier understanding of the documentation.
- Stage: Integration Tests: Execute the unit tests of
origviaPkg.test(). - Stage: Generate integration Tests: Download the
jobGenerator.jlscript from theQED.jlpackage viagit clone. The script is executed with the name of the subpackageorig. The script traverses the dependency tree of theQED.jland it is searching for sub-packages which has a dependency of packageorig. For each package with the dependency of packageorig, the generator generates a job yaml. By default, it uses the upstream repository and development branch of the package tested in the integration test. The repository and commit can be changed via the environment parameter in the git commit message. ThejobGenerator.jluses thePackage.tomlof theQED.jlpackage because all sub-packages are direct and indirect dependencies of it. - Stage: Run Integration Tests: This stage uses the generated job yaml to create and run new test jobs. It uses the GitLab CI child pipeline mechanism.
- Stage: Integration Tests of Sub package N: Each job clones the repository of the subpackage. After the clone, it uses the Julia function
Pkg.develop(path="$CI_package_DIR")to replace the dependency to the packageorigwith the modified version of the Pull Request and execute the tests of the subpackage viaPkg.test().
If a sub package triggers an integration test, the main package QED.jl is passive. It does not get any notification or trigger any script. The repository is simply cloned.
Features of the packages
- calculate area of a cylinder
- calculate site area of a cylinder
- calculate volume of a cylinder