diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 96f80220770cc..7869f254a6397 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -24,15 +24,15 @@ Contributor's Quick Start Note to Starters ################ -Airflow is quite a complex project, and setting up a working environment, but we made it rather simple if -you follow the guide. +Airflow is a complex project, but setting up a working environment is quite simple +if you follow the guide. There are three ways you can run the Airflow dev env: 1. With a Docker Containers and Docker Compose (on your local machine). This environment is managed with `Breeze <../dev/breeze/doc/README.rst>`_ tool written in Python that makes the environment - management, yeah you guessed it - a breeze. -2. With a local virtual environment (on your local machine). + management, yeah you guessed it - a breeze +2. With a local virtual environment (on your local machine) 3. With a remote, managed environment (via remote development environment) Before deciding which method to choose, there are a couple of factors to consider: @@ -77,14 +77,14 @@ Docker Community Edition "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null -2. Install Docker Engine, containerd, and Docker Compose Plugin. +2. Install Docker Engine, containerd, and Docker Compose Plugin .. code-block:: bash sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -3. Creating group for docker and adding current user to it. +3. Creating group for docker and adding current user to it .. code-block:: bash @@ -105,13 +105,13 @@ If you use Colima as your container runtimes engine, please follow the next step 1. `Install buildx manually `_ and follow its instructions -2. Link the Colima socket to the default socket path. Note that this may break other Docker servers. +2. Link the Colima socket to the default socket path. Note that this may break other Docker servers .. code-block:: bash sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock -3. Change docker context to use default: +3. Change docker context to use default .. code-block:: bash @@ -147,7 +147,7 @@ Setting up virtual-env as your build and integration frontend, and we already use ``hatchling`` build backend for Airflow. You can read more about Hatch and it's use in Airflow in `Local virtualenv <07_local_virtualenv.rst>`_. See [PEP-517](https://peps.python.org/pep-0517/#terminology-and-goals) for explanation of what the - frontend and backend meaning is. + frontend and backend meaning is 2. After creating, you need to install a few more required packages for Airflow. The below command adds basic system-level dependencies on Debian/Ubuntu-like system. You will have to adapt it to install similar packages @@ -170,12 +170,12 @@ like system, this command will install all necessary dependencies that should be software-properties-common sqlite3 sudo unixodbc unixodbc-dev 3. With Hatch you can enter the virtual environment with ``hatch shell`` command, check - `Local virtualenvs <./07_local_virtualenv.rst#using-hatch>`__ for more details: + `Local virtualenvs <./07_local_virtualenv.rst#using-hatch>`__ for more details Forking and cloning Project --------------------------- -1. Goto |airflow_github| and fork the project. +1. Goto |airflow_github| and fork the project .. |airflow_github| raw:: html @@ -188,7 +188,7 @@ Forking and cloning Project alt="Forking Apache Airflow project"> -2. Goto your github account's fork of airflow click on ``Code`` you will find the link to your repo. +2. Goto your github account's fork of airflow click on ``Code`` you will find the link to your repo .. raw:: html @@ -199,7 +199,7 @@ Forking and cloning Project 3. Follow `Cloning a repository `_ to clone the repo locally (you can also do it in your IDE - see the `Using your IDE`_ - chapter below. + chapter below Note: For windows based machines, on cloning, the Git line endings may be different from unix based systems and might lead to unexpected behaviour on running breeze tooling. Manually setting a property will mitigate this issue. @@ -224,11 +224,11 @@ Setting up Breeze 1. Install ``pipx`` (>=1.2.1) - follow the instructions in `Install pipx `_ It is important to install version of pipx >= 1.2.1 to workaround ``packaging`` breaking change introduced - in September 2023. + in September 2023 2. Run ``pipx install -e ./dev/breeze`` in your checked-out repository. Make sure to follow any instructions printed by ``pipx`` during the installation - this is needed to make sure that ``breeze`` command is - available in your PATH. + available in your PATH .. warning:: @@ -271,11 +271,11 @@ Setting up Breeze 5. When you enter Breeze environment you should see prompt similar to ``root@e4756f6ac886:/opt/airflow#``. This means that you are inside the Breeze container and ready to run most of the development tasks. You can leave - the environment with ``exit`` and re-enter it with just ``breeze`` command. + the environment with ``exit`` and re-enter it with just ``breeze`` command 6. Once you enter breeze environment, create airflow tables and users from the breeze CLI. ``airflow db reset`` is required to execute at least once for Airflow Breeze to get the database/tables created. If you run - tests, however - the test database will be initialized automatically for you. + tests, however - the test database will be initialized automatically for you .. code-block:: bash @@ -293,7 +293,7 @@ Setting up Breeze 7. Exiting Breeze environment. After successfully finishing above command will leave you in container, type ``exit`` to exit the container. The database created before will remain and servers will be - running though, until you stop breeze environment completely. + running though, until you stop breeze environment completely .. code-block:: bash @@ -301,7 +301,7 @@ Setting up Breeze root@b76fcb399bb6:/opt/airflow# exit 8. You can stop the environment (which means deleting the databases and database servers running in the - background) via ``breeze down`` command. + background) via ``breeze down`` command .. code-block:: bash @@ -314,7 +314,7 @@ Using Breeze 1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run( In this case python and backend will be picked up from last execution ``breeze --python 3.9 --backend postgres``) It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left - and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate. + and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate .. code-block:: bash @@ -387,7 +387,7 @@ Using Breeze 2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin`` - and password ``admin``. + and password ``admin`` .. |http://127.0.0.1:28080| raw:: html @@ -402,7 +402,7 @@ Using Breeze 3. Setup a PostgreSQL database in your database management tool of choice (e.g. DBeaver, DataGrip) with host ``127.0.0.1``, port ``25433``, - user ``postgres``, password ``airflow``, and default schema ``airflow``. + user ``postgres``, password ``airflow``, and default schema ``airflow`` .. raw:: html @@ -602,7 +602,7 @@ on macOS, install via Installing airflow in the local venv ------------------------------------ -1. It may require some packages to be installed; watch the output of the command to see which ones are missing. +1. It may require some packages to be installed; watch the output of the command to see which ones are missing .. code-block:: bash @@ -610,7 +610,7 @@ Installing airflow in the local venv ./scripts/tools/initialize_virtualenv.py -2. Add following line to ~/.bashrc in order to call breeze command from anywhere. +2. Add following line to ~/.bashrc in order to call breeze command from anywhere .. code-block:: bash @@ -674,7 +674,7 @@ All Tests are inside ./tests directory. breeze --backend postgres --postgres-version 15 --python 3.9 --db-reset testing tests --test-type All --integration mongo -- For more information on Testing visit : |09_testing.rst| +- For more information on Testing visit |09_testing.rst| .. |09_testing.rst| raw:: html @@ -734,7 +734,7 @@ Raising Pull Request alt="Goto fork and select branches"> -2. Click on ``New pull request`` button on branch from which you want to raise a pull request. +2. Click on ``New pull request`` button on branch from which you want to raise a pull request .. raw:: html @@ -743,7 +743,7 @@ Raising Pull Request alt="Accessing local airflow"> -3. Add title and description as per Contributing guidelines and click on ``Create pull request``. +3. Add title and description as per Contributing guidelines and click on ``Create pull request`` .. raw:: html diff --git a/contributing-docs/09_testing.rst b/contributing-docs/09_testing.rst index 8500ded3d5b76..895fd1b964f09 100644 --- a/contributing-docs/09_testing.rst +++ b/contributing-docs/09_testing.rst @@ -18,9 +18,14 @@ Airflow Test Infrastructure =========================== +Airflow features a comprehensive testing infrastructure encompassing multiple testing methodologies designed to +ensure reliability and functionality across different deployment scenarios and integrations. The testing framework +includes: + * `Unit tests `__ are Python tests that do not require any additional integrations. Unit tests are available both in the `Breeze environment <../dev/breeze/doc/README.rst>`__ - and `local virtualenv <07_local_virtualenv.rst>`__. More about unit tests. + and `local virtualenv <07_local_virtualenv.rst>`__. Note that in order for a pull request to be reviewed, + **it should have a unit test**, unless a test is not required i.e. for documentation changes. * `Integration tests `__ are available in the `Breeze environment <../dev/breeze/doc/README.rst>`__ that is also used for Airflow CI tests. @@ -39,7 +44,7 @@ Airflow Test Infrastructure * `System tests `__ are automatic tests that use external systems like Google Cloud and AWS. These tests are intended for an end-to-end DAG execution. -You can also run other kind of tests when you are developing airflow packages: +You can also run other kinds of tests when you are developing airflow packages: * `Testing packages `__ is a document that describes how to manually build and test pre-release candidate packages of airflow and providers. diff --git a/contributing-docs/10_working_with_git.rst b/contributing-docs/10_working_with_git.rst index a3b0f4ac40bde..2d64ba24929f6 100644 --- a/contributing-docs/10_working_with_git.rst +++ b/contributing-docs/10_working_with_git.rst @@ -112,21 +112,20 @@ Here is how rebase looks in practice (you can find a summary below these detaile 1. You first need to add the Apache project remote to your git repository. This is only necessary once, so if it's not the first time you are following this tutorial you can skip this step. In this example, -we will be adding the remote -as "apache" so you can refer to it easily: +we will be adding the remote as "apache" so you can refer to it easily * If you use ssh: ``git remote add apache git@github.com:apache/airflow.git`` * If you use https: ``git remote add apache https://github.com/apache/airflow.git`` 2. You then need to make sure that you have the latest main fetched from the ``apache`` repository. You can do this - via: + via ``git fetch apache`` (to fetch apache remote) ``git fetch --all`` (to fetch all remotes) 3. Assuming that your feature is in a branch in your repository called ``my-branch`` you can easily check - what is the base commit you should rebase from by: + what is the base commit you should rebase from via ``git merge-base my-branch apache/main`` @@ -147,11 +146,11 @@ as "apache" so you can refer to it easily: Will "transplant" all commits after the commit with the HASH. -4. Providing that you weren't already working on your branch, check out your feature branch locally via: +4. Providing that you weren't already working on your branch, check out your feature branch locally via ``git checkout my-branch`` -5. Commit your code change. +5. Commit your code change ``git add .`` @@ -165,7 +164,7 @@ as "apache" so you can refer to it easily: You can add the ``--global`` flag to avoid setting it for every cloned repo. -6. Rebase: +6. Rebase ``git rebase HASH --onto apache/main`` @@ -175,12 +174,12 @@ as "apache" so you can refer to it easily: 7. If you have no conflicts - that's cool. You rebased. You can now run ``git push --force-with-lease`` to push your changes to your repository. That should trigger the build in our CI if you have a - Pull Request (PR) opened already. + Pull Request (PR) opened already 8. While rebasing you might have conflicts. Read carefully what git tells you when it prints information about the conflicts. You need to solve the conflicts manually. This is sometimes the most difficult part and requires deliberately correcting your code and looking at what has changed since you developed your - changes. + changes There are various tools that can help you with this. You can use: @@ -191,7 +190,7 @@ as "apache" so you can refer to it easily: you have a very intuitive and helpful merge tool. For more information, see `Resolve conflicts `_. -9. After you've solved your conflict run: +9. After you've solved your conflict run ``git rebase --continue`` diff --git a/contributing-docs/16_contribution_workflow.rst b/contributing-docs/16_contribution_workflow.rst index f4d2d77c9581b..8c941f8afc7a4 100644 --- a/contributing-docs/16_contribution_workflow.rst +++ b/contributing-docs/16_contribution_workflow.rst @@ -68,9 +68,9 @@ on your local machine, you might choose Local Virtualenv, or dockerized Breeze e also have support for popular remote development environments: GitHub Codespaces and GitPodify. You can see the differences between the various environments in `Development environments <06_development_environments.rst>`__. -The local env instructions can be found in full in the `Local virtualenv <07_local_virtualenv.rst>`_ file. +The local environment instructions can be found in full in the `Local virtualenv <07_local_virtualenv.rst>`_ file. -The Breeze Docker Compose env is to maintain a consistent and common development environment so that you +The Breeze Docker Compose environment is to maintain a consistent and common development environment so that you can replicate CI failures locally and work on solving them locally rather by pushing to CI. The Breeze instructions can be found in full in the `Breeze documentation <../dev/breeze/doc/README.rst>`_ file. @@ -121,8 +121,8 @@ to make them immediately visible in the environment. ./scripts/tools/initialize_virtualenv.py -6. Open your IDE (for example, PyCharm) and select the virtualenv you created - as the project's default virtualenv in your IDE. +6. Open your IDE (for example, PyCharm) and select the virtual environment you created + as the project's default virtual environment in your IDE. Step 3: Connect with People ---------------------------