-
Notifications
You must be signed in to change notification settings - Fork 13
Getting Started with tests‐sos
The tests-sos repository is a collection of tests that originated from the Sandia OpenSHMEM (SOS) project. It contains unit tests, shmemx (extension) tests, application examples, performance microbenchmarks, and specification examples, each residing in its own sub-directory within tests-sos/test.
A primary goal of the OpenSHMEM community is to ensure compliance with the latest API specification. To support this goal, the "tests-sos" repository, which contains test suites useful for verifying any OpenSHMEM-compliant implementation, has been transferred to the official openshmem-org GitHub organization. As of May 2024, this repository is now incorporated as a Git submodule within the SOS project and can be similarly integrated into other OpenSHMEM implementations. This change facilitates consistent testing and validation across different OpenSHMEM projects.
Users of tests-sos can rely on the exit code to indicate whether any particular test has passed (exit code is 0) or failed (exit code is non-zero). For the SOS repo, this is how the Github workflow/action determines which "rows" in the testing matrix passed. You can check out the entire workflow script here: https://github.com/Sandia-OpenSHMEM/SOS/blob/main/.github/workflows/ci.yml
If you're unfamiliar, a prerequisite to understanding that file/script is some details in the Github docs, e.g.: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions
That said, using Github is not at all required for these tests, for example SOS used Travis CI for a long time and also had success with Jenkins. But right now SOS relies almost entirely on Github Actions for CI/CD.
In other words, what you see within the SOS "Actions" tab represents our primary strategy for validating SOS. However, the SOS development team also manually run make check on various systems of interest and check measurements from test/performance/shmem_perf_suite for performance regressions.
Please refer the SOS Developer Documentation page for guidelines and procedures regarding pull requests (PRs) and code review. We follow those same guidelines with tests-sos; however, it is acceptable to merge tests-sos PRs with only 1 approving review, but 2 approvals are preferred for PRs with non-trivial changes.