diff --git a/contributing-docs/testing/system_tests.rst b/contributing-docs/testing/system_tests.rst index a8a1a9fc85660..f9add8b377180 100644 --- a/contributing-docs/testing/system_tests.rst +++ b/contributing-docs/testing/system_tests.rst @@ -20,16 +20,40 @@ Airflow System Tests System tests need to communicate with external services/systems that are available if you have appropriate credentials configured for your tests. -The system tests derive from the ``tests.test_utils.system_test_class.SystemTests`` class. They should also -be marked with ``@pytest.marker.system(SYSTEM)`` where ``system`` designates the system -to be tested (for example, ``google.cloud``). These tests are skipped by default. -You can execute the system tests by providing the ``--system SYSTEM`` flag to ``pytest``. You can -specify several --system flags if you want to execute tests for several systems. +.. contents:: :local: + +Purpose of System Tests +----------------------- + +Airflow system tests are pretty special because they serve three purposes: + +* they are runnable tests that communicate with external services +* they are example_dags that users can just copy&paste and use as starting points for their own DAGs +* the excerpts from these system tests are used to generate documentation + +Old System Tests +---------------- + +The system tests derive from the ``tests.test_utils.system_test_class.SystemTests`` class. + +Old versions of System tests should also be marked with ``@pytest.marker.system(SYSTEM)`` where ``system`` +designates the system to be tested (for example, ``google.cloud``). These tests are skipped by default. + +For older version of the tests, you can execute the system tests by providing the +``--system SYSTEM`` flag to ``pytest``. You can specify several --system flags if you want to execute +tests for several systems. The system tests execute a specified example DAG file that runs the DAG end-to-end. -.. contents:: :local: +New System Tests +---------------- + +The new system tests follows +[AIP-47 New Design of System Tests](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests) +and those system tests do not require separate ``pytest`` flag to be executed, they also don't need a separate +class to run - all the code is kept in the system test class that is also an executable DAG, it is discoverable +by ``pytest`` and it is executable as Python script. Environment for System Tests ---------------------------- @@ -74,13 +98,8 @@ run Google Cloud system tests. RANDOM_POSTFIX=$(cat "${RANDOM_FILE}") - -To execute system tests, specify the ``--system SYSTEM`` -flag where ``SYSTEM`` is a system to run the system tests for. It can be repeated. - - Forwarding Authentication from the Host ----------------------------------------------------- +--------------------------------------- For system tests, you can also forward authentication from the host to your Breeze container. You can specify the ``--forward-credentials`` flag when starting Breeze. Then, it will also forward the most commonly used @@ -94,75 +113,11 @@ Currently forwarded credentials are: * credentials stored in ``${HOME}/.docker`` for ``docker`` client * credentials stored in ``${HOME}/.snowsql`` for ``snowsql`` - SnowSQL (Snowflake CLI client) -Adding a New System Test --------------------------- - -We are working on automating system tests execution (AIP-4) but for now, system tests are skipped when -tests are run in our CI system. But to enable the test automation, we encourage you to add system -tests whenever an operator/hook/sensor is added/modified in a given system. - -* To add your own system tests, derive them from the - ``tests.test_utils.system_tests_class.SystemTest`` class and mark with the - ``@pytest.mark.system(SYSTEM_NAME)`` marker. The system name should follow the path defined in - the ``providers`` package (for example, the system tests from ``tests.providers.google.cloud`` - package should be marked with ``@pytest.mark.system("google.cloud")``. - -* If your system tests need some credential files to be available for an - authentication with external systems, make sure to keep these credentials in the - ``files/airflow-breeze-config/keys`` directory. Mark your tests with - ``@pytest.mark.credential_file()`` so that they are skipped if such a credential file is not there. - The tests should read the right credentials and authenticate them on their own. The credentials are read - in Breeze from the ``/files`` directory. The local "files" folder is mounted to the "/files" folder in Breeze. - -* If your system tests are long-running ones (i.e., require more than 20-30 minutes - to complete), mark them with the ```@pytest.markers.long_running`` marker. - Such tests are skipped by default unless you specify the ``--long-running`` flag to pytest. - -* The system test itself (python class) does not have any logic. Such a test runs - the DAG specified by its ID. This DAG should contain the actual DAG logic - to execute. Make sure to define the DAG in ``providers//example_dags``. These example DAGs - are also used to take some snippets of code out of them when documentation is generated. So, having these - DAGs runnable is a great way to make sure the documentation is describing a working example. Inside - your test class/test method, simply use ``self.run_dag(,)`` to run the DAG. Then, - the system class will take care about running the DAG. Note that the DAG_FOLDER should be - a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers//example_dags``. - - -A simple example of a system test is available in: - -``tests/providers/google/cloud/operators/test_compute_system.py``. - -It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py``. - - -The typical system test session -------------------------------- - -Here is the typical session that you need to do to run system tests: - -1. Enter breeze - -.. code-block:: bash - - breeze down - breeze --python 3.8 --db-reset --forward-credentials - -This will: - -* stop the whole environment (i.e. recreates metadata database from the scratch) -* run Breeze with: - - * python 3.8 version - * resetting the Airflow database - * forward your local credentials to Breeze - -3. Run the tests: - -.. code-block:: bash +Running the System Tests +------------------------ - pytest -o faulthandler_timeout=2400 \ - --system=google tests/providers/google/cloud/operators/test_compute_system.py +Running the tests and developing tests in `Test documentation <../../tests/system/README.md>`__ ------ +------ For other kinds of tests look at `Testing document <../09_testing.rst>`__