diff --git a/_freeze/blogs/11-fiddly-bits-of-pytest/execute-results/html.json b/_freeze/blogs/11-fiddly-bits-of-pytest/execute-results/html.json index 7885011..f1548bf 100644 --- a/_freeze/blogs/11-fiddly-bits-of-pytest/execute-results/html.json +++ b/_freeze/blogs/11-fiddly-bits-of-pytest/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "c7af69b7f9e8f6d77a61f5556854b601", + "hash": "214bfe625b93386ce9685cf55699db3a", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Pytest Fixtures in Plain English\nauthor: Rich Leyshon\ndate: April 07 2024\ndescription: Plain English Discussion of Pytest Fixtures.\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - fixtures\n - pytest-in-plain-english\nimage: 'https://images.pixexid.com/sculptural-simplicity-monochrome-background-highlighting-the-beauty-of-minimali-jmhkipzb.jpeg?h=699&q=70'\nimage-alt: 'Sculptural simplicity, monochrome background highlighting the beauty of minimalist sculptures by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)'\ntoc: true\n---\n\n
\n \n
Creative commons license by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses using test data as fixtures\nwith `pytest` and is the first in a series of blogs called\n[pytest in plain English](/../index.html#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\npytest documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend the audience's understanding of the more\nintricate features of `pytest` by describing their utility with simple code\nexamples. \n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\noptimise their test code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[fixtures branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/fixtures)\nof the example code repo.\n\n## What are fixtures?\n\nData. Well, data provided specifically for testing purposes. This is the\nessential definition for a fixture. One could argue the case that fixtures are\nmore than this. Fixtures could be environment variables, class instances,\nconnection to a server or whatever dependencies your code needs to run.\n\nI would agree that fixtures are not just data. But that all fixtures return\ndata of some sort, regardless of the system under test.\n\n## When would you use fixtures?\n\nIt's a bad idea to commit data to a git repository, right? Agreed. Though\nfixtures are rarely 'real' data. The data used for testing purposes should be\nminimal and are usually synthetic. \n\n**Minimal fixtures** conform to the schema of the actual data that the system\nrequires. These fixtures will be as small as possible while capturing all known\nimportant cases. Keeping the data small maintains a performant test suite and\navoids problems associated with large files and git version control.\n\nIf you have ever encountered a problem in a system that was caused by a\nproblematic record in the data, the aspect of that record that broke your\nsystem should absolutely make it into the next version of your minimal test\nfixture. Writing a test that checks that the codebase can handle such problem\nrecords is known as 'regression testing' - safeguarding against old bugs\nresurfacing when code is refactored or new features are implemented. This\nscenario commonly occurs when a developer unwittingly violates Chesterton's\nPrinciple.\n\n\n\n\nMany thanks to my colleague Mat for pointing me towards this useful analogy. A\nconsiderate developer would probably include a comment in their code about a\nspecific problem that they've handled (like erecting a sign next to\nChesterton's fence). An experienced developer would do the same, and also write\na regression test to ensure the problem doesn't re-emerge in the future\n(monitoring the fence with CCTV...). Discovering these problem cases and\nemploying defensive strategies avoids future pain for yourself and colleagues.\n\nAs you can imagine, covering all the important cases while keeping the fixture\nminimal is a compromise. At the outset of the work, it may not be obvious what\nproblematic cases may arise. Packages such as\n[`hypothesis`](https://hypothesis.readthedocs.io/en/latest/) allow you to\ngenerate awkward cases. Non-utf-8 strings anyone? Hypothesis can generate\nthese test cases for you, along with many more interesting edge-cases -\n*ăѣ𝔠ծềſģȟᎥ𝒋ǩľḿꞑȯ𝘱𝑞𝗋𝘴ȶ𝞄𝜈ψ𝒙𝘆𝚣* (Non-utf8 strings often cause problems for web apps). \n\n**Non-disclosive fixtures** are those that do not expose personally\nidentifiable or commercially-sensitive information. If you are working with\nthis sort of data, it is necessary to produce toy test fixtures that mimic the\nschema of the real data. Names and addresses can be de-identified to random\nalphanumeric strings. Location data can be adjusted with noise. The use of\ndummy variables or categories can mitigate the risk of disclosure by\ndifferencing.\n\nBy adequately anonymising data and testing problem cases, the programmer\nexhibits upholds duties under the General Data Protection Regulation:\n\n> accurately store, process, retain and erase personally-identifiable information.\n\nIn cases where the system integrates with data available in the public domain,\nit is may be permissible to include a small sample of the data as a test\nfixture. Ensure the license that the data is distributed under is compatible\nwith your code's license. If the license is compatible, I recommend including a\nreference to the fixture, its source and license within a LICENSE.note file.\nThis practice is enforced by Comprehensive R Archive Network. You can read more\nabout this in the\n[R Packages documentation](https://r-pkgs.org/license.html#sec-how-to-include).\n\n## Scoping fixtures\n\n`pytest` fixtures have different scopes, meaning that they will be prepared\ndifferently dependent on the scope you specify. The available scopes are as\nfollows: \n\n| Scope Name | Teardown after each |\n| ----------- | ------------------- |\n| function | test function |\n| class | test class |\n| module | test module |\n| package | package under test |\n| session | `pytest` session |\n\nNote that the default scope for any fixtures that you define will be\n'function'. A function-scoped fixture will be set up for every test function\nthat requires it. Once the function has executed, the fixture will then be\ntorn down and all changes to this fixture will be lost. This default behaviour\nencourages isolation in your test suite. Meaning that the tests have no\ndependencies upon each other. The test functions could be run in any order\nwithout affecting the results of the test. Function-scoped fixtures are the\nshortest-lived fixtures. Moving down the table above, the persistence of the\nfixtures increases. Changes to a session-scoped fixture persist for the entire\ntest execution duration, only being torn down once `pytest` has executed all\ntests.\n\n### Scoping for performance\n***\n\n> performance vs isolation\n\nBy definition, a unit test is completely isolated, meaning that it will have no\ndependencies other than the code it needs to test. However, there may be a few\ncases where this would be less desirable. Slow test suites may introduce\nexcessive friction to the software development process. Persistent fixtures can\nbe used to improve the performance of a test suite. \n\nFor example, here we define some expensive class:\n\n```{.python filename=expensive.py}\n\"\"\"A module containing an expensive class definition.\"\"\"\nimport time\nfrom typing import Union\n\n\nclass ExpensiveDoodah:\n \"\"\"A toy class that represents some costly operation.\n\n This class will sleep for the specified number of seconds on instantiation.\n\n Parameters\n ----------\n sleep_time : Union[int, float]\n Number of seconds to wait on init.\n\n \"\"\"\n def __init__(self, sleep_time: Union[int, float] = 2):\n print(f\"Sleeping for {sleep_time} seconds\")\n time.sleep(sleep_time)\n return None\n\n```\n\nThis class will be used to demonstrate the effect of scoping with some costly\noperation. This example could represent reading in a bulky xlsx file or\nquerying a large database.\n\nTo serve `ExpensiveDoodah` to our tests, I will define a function-scoped\nfixture. To do this, we use a `pytest` fixture decorator to return the class\ninstance with a specified sleep time of 2 seconds.\n\n```{.python filename=test_expensive.py}\nimport pytest\n\nfrom example_pkg.expensive import ExpensiveDoodah\n\n\n@pytest.fixture(scope=\"function\")\ndef module_doodah():\n \"\"\"Function-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n```\nNow to test `ExpensiveDoodah` we extend our test module to include a test class\nwith 3 separate test functions. The assertions will all be the same for this\nsimple example - that `ExpensiveDoodah` executes without raising any error\nconditions. Notice we must pass the name of the fixture in each test function's\nsignature.\n\n```{.python filename=test_expensive.py}\n\"\"\"Tests for expensive.py using function-scoped fixture.\"\"\"\nfrom contextlib import nullcontext as does_not_raise\nimport pytest\n\nfrom example_pkg.expensive import ExpensiveDoodah\n\n\n@pytest.fixture(scope=\"function\")\ndef doodah_fixture():\n \"\"\"Function-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n\nclass TestA:\n \"\"\"A test class.\"\"\"\n\n def test_1(self, doodah_fixture):\n \"\"\"Test 1.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n def test_2(self, doodah_fixture):\n \"\"\"Test 2.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n def test_3(self, doodah_fixture):\n \"\"\"Test 3.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n```\n\nThe result of running this test module can be seen below:\n\n```\ncollected 3 items\n\n./tests/test_expensive_function_scoped.py ... [100%]\n\n============================ 3 passed in 6.04s ================================\n\n```\n\nNotice that the test module took just over 6 seconds to execute because the\nfunction-scoped fixture was set up once for each test function.\n\nIf instead we had defined `doodah_fixture` with a different scope, it would\nreduce the time for the test suite to complete by approximately two thirds.\nThis is the sort of benefit that can be gained from considerate use of `pytest`\nfixtures.\n\n```{.python filename=test_expensive.py}\n@pytest.fixture(scope=\"module\")\ndef doodah_fixture():\n \"\"\"Module-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n```\n\n```\ncollected 3 items\n\n./tests/test_expensive_function_scoped.py ... [100%]\n\n============================ 3 passed in 2.02s ================================\n\n```\n\nThe scoping feature of `pytest` fixtures can be used to optimise a test-suite\nand avoid lengthy delays while waiting for your test suites to execute.\nHowever, any changes to the fixture contents will persist until the fixture is\nnext torn down. Keeping track of the states of differently-scoped fixtures in a\ncomplex test suite can be tricky and reduces segmentation overall. Bear this in\nmind when considering which scope best suits your needs.\n\n### Scope persistence\n***\n\n> function < class < module < package < session\n\nUsing scopes other than 'function' can be useful for end-to-end testing.\nPerhaps you have a complex analytical pipeline and need to check that the\nvarious components work well **together**, rather than in isolation as with a\nunit test. This sort of test can be extremely useful for developers in a rush.\nYou can test that the so called 'promise' of the codebase is as expected, even\nthough the implementation may change.\n\nThe analogy here would be that the success criteria of a SatNav is that it gets\nyou to your desired destination whatever the suggested route you selected.\nChecking that you used the fastest or most fuel efficient option is probably a\ngood idea. But if you don't have time, you'll just have to take the hit if you\nencounter a toll road. Though it's still worth checking that the postcode you\nhastily input to the satnav is the correct one.\n\n\n
\nPerhaps your success criteria is that you need to write a DataFrame to file. \nA great end-to-end test would check that the DataFrame produced has the\nexpected number of rows, or even has rows! Of course it's also a good idea to\ncheck the DataFrame conforms to the expected table schema, too: number of\ncolumns, names of columns, order and data types. This sort of check is often\noverlooked in favour of pressing on with development. If you've ever\nencountered a situation where you've updated a codebase and later realised you\nnow have empty tables (I certainly have), this sort of test would be really\nhandy, immediately alerting you to this fact and helping you efficiently locate\nthe source of the bug.\n\n#### Define Data\n\nIn this part, I will explore the scoping of fixtures with DataFrames. Again,\nI'll use a toy example to demonstrate scope behaviour. Being a child of the\n'90s (mostly), I'll use a scenario from my childhood. Scooby Doo is still a\nthing, right?\n\n**Enter: The Mystery Machine**\n\n\"Scooby\n\n\nThe scenario: The passengers of the Mystery Machine van all have the munchies.\nThey stop at a 'drive thru' to get some takeaway. We have a table with a record\nfor each character. We have columns with data about the characters' names,\ntheir favourite food, whether they have 'the munchies', and the contents of\ntheir stomach.\n\n::: {#7ed99bb4 .cell execution_count=1}\n``` {.python .cell-code}\nimport pandas as pd\nmystery_machine = pd.DataFrame(\n {\n \"name\": [\"Daphne\", \"Fred\", \"Scooby Doo\", \"Shaggy\", \"Velma\"],\n \"fave_food\": [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ],\n \"has_munchies\": [True] * 5, # everyone's hungry\n \"stomach_contents\": [\"empty\"] * 5, # all have empty stomachs\n }\n )\nmystery_machine\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
namefave_foodhas_munchiesstomach_contents
0DaphnecarrotsTrueempty
1FredbeansTrueempty
2Scooby Dooscooby snacksTrueempty
3ShaggyburgersTrueempty
4Velmahot dogsTrueempty
\n
\n```\n:::\n:::\n\n\nTo use this simple DataFrame as a fixture, I could go ahead and define it with\n`@pytest.fixture()` directly within a test file. But if I would like to share\nit across several test modules\n([as implemented later](#fixtures-across-multiple-test-modules)), then there\nare 2 options:\n\n1. Write the DataFrame to disk as csv (or whatever format you prefer) and save\nin a `./tests/data/` folder. At the start of your test modules you can read the\ndata from disk and use it for testing. In this approach you'll likely define\nthe data as a test fixture in each of the test modules that need to work with\nit.\n2. Define the fixtures within a special python file called `conftest.py`, which\nmust be located at the root of your project. This file is used to configure\nyour tests. `pytest` will look in this file for any required fixture\ndefinitions when executing your test suite. If it finds a fixture with the same\nname as that required by a test, the fixture code may be run. \n\n:::{.callout-caution}\nWait! Did you just say '**may** be run'?\n:::\n\nDepending on the scope of your fixture, `pytest` may not need to execute the\ncode for each test. For example, let's say we're working with a session-scoped\nfixture. This type of fixture will persist for the duration of the entire test\nsuite execution. Imagine test number 1 and 10 both require this test fixture.\nThe fixture definition only gets executed the first time a test requires it.\nThis test fixture will be set up as test 1 executes and will persist until\ntear down occurs at the end of the `pytest` session. Test 10 will therefore use\nthe same instance of this fixture as test 1 used, meaning any changes to the\nfixture may be carried forward.\n\n#### Define fixtures\n\nFor our example, we will create a `conftest.py` file and define some fixtures\nwith differing scopes.\n\n```{.python filename=conftest.py}\n\"\"\"Demonstrate scoping fixtures.\"\"\"\nimport pandas as pd\nimport pytest\n\n\n@pytest.fixture(scope=\"session\")\ndef _mystery_machine():\n \"\"\"Session-scoped fixture returning pandas DataFrame.\"\"\"\n return pd.DataFrame(\n {\n \"name\": [\"Daphne\", \"Fred\", \"Scooby Doo\", \"Shaggy\", \"Velma\"],\n \"fave_food\": [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ],\n \"has_munchies\": [True] * 5,\n \"stomach_contents\": [\"empty\"] * 5,\n }\n )\n\n\n@pytest.fixture(scope=\"session\")\ndef _mm_session_scoped(_mystery_machine):\n \"\"\"Session-scoped fixture returning the _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"module\")\ndef _mm_module_scoped(_mystery_machine):\n \"\"\"Module-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"class\")\ndef _mm_class_scoped(_mystery_machine):\n \"\"\"Class-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"function\")\ndef _mm_function_scoped(_mystery_machine):\n \"\"\"Function-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n```\n\nFixtures can reference each other, if they're scoped correctly. More on this in\n[the next section](#scopemismatch-error). This is useful for my toy example as\nI intend the source functions to update the DataFrames directly, if I wasn't\ncareful about deep copying the fixtures, my functions would update the original\n`_mystery_machine` fixture's table. Those changes would then be subsequently\npassed to the other fixtures, meaning I couldn't clearly demonstrate how the\ndifferent scopes persist.\n\n#### Define the source functions\n\nNow, let's create a function that will feed characters their favourite food if\nthey have the munchies. \n\n```{.python filename=feed_characters.py}\n\"\"\"Helping learners understand how to work with pytest fixtures.\"\"\"\nimport pandas as pd\n\n\ndef serve_food(df: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Serve characters their desired food.\n\n Iterates over a df, feeding characters if they have 'the munchies' with\n their fave_food. If the character is not Scooby Doo or Shaggy, then update\n their has_munchies status to False. The input df is modified inplace.\n\n Parameters\n ----------\n df : pd.DataFrame\n A DataFrame with the following columns: \"name\": str, \"fave_food\": str,\n \"has_munchies\": bool, \"stomach_contents\": str.\n\n Returns\n -------\n pd.DataFrame\n Updated DataFrame with new statuses for stomach_contents and\n has_munchies.\n\n \"\"\"\n for ind, row in df.iterrows():\n if row[\"has_munchies\"]:\n # if character is hungry then feed them\n food = row[\"fave_food\"]\n character = row[\"name\"]\n print(f\"Feeding {food} to {character}.\")\n df.loc[ind, [\"stomach_contents\"]] = food\n if character not in [\"Scooby Doo\", \"Shaggy\"]:\n # Scooby & Shaggy are always hungry\n df.loc[ind, \"has_munchies\"] = False\n else:\n # if not hungry then do not adjust\n pass\n return df\n\n```\n\nNote that it is commonplace to copy a pandas DataFrame so that any operations\ncarried out by the function are confined to the function's scope. To\ndemonstrate changes to the fixtures I will instead choose to edit the DataFrame\ninplace.\n\n#### Fixtures Within a Single Test Module\n\nNow to write some tests. To use the fixtures we defined earlier, we simply \ndeclare that a test function requires the fixture. `pytest` will notice this \ndependency on collection, check the fixture scope and execute the fixture code \nif appropriate. The following test `test_scopes_before_action` checks that the\n`mystery_machine` fixtures all have the expected `has_munchies` column values\nat the outset of the test module, i.e. everybody is hungry before our source\nfunction takes some action. This type of test doesn't check behaviour of any\nsource code and therefore would be unnecessary for quality assurance purposes.\nBut I include it here to demonstrate the simple use of fixtures and prove to\nthe reader the state of the DataFrame fixtures prior to any source code\nintervention.\n\n:::{.callout-tip collapse=\"true\"}\n\n##### **Testing `pandas` DataFrames**\n\nYou may notice that the assert statements in the tests below requires pulling \ncolumn values out and casting to lists. The `pandas` package has its own\ntesting module that is super useful for testing all aspects of DataFrames.\nCheck out the\n[`pandas` testing documentation](https://pandas.pydata.org/docs/reference/testing.html)\nfor more on how to write robust tests for pandas DataFrames and Series.\n\n:::\n\n```{.python filename=\"test_feed_characters.py\"}\n\"\"\"Testing pandas operations with test fixtures.\"\"\"\nfrom example_pkg.feed_characters import serve_food\n\n\ndef test_scopes_before_action(\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n):\n \"\"\"Assert that all characters have the munchies at the outset.\"\"\"\n assert list(_mm_session_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The session-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_module_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The module-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_class_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The class-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_function_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The function-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n\n```\n\nNow to test the `serve_food()` function operates as expected. We can define\na test class that will house all tests for `serve_food()`. Within that class\nlet's define our first test that simply checks that the value of the\n`has_munchies` column has been updated as we would expect after using the\n`serve_food()` function.\n\n```{.python filename=\"test_feed_characters.py\"}\nclass TestServeFood:\n \"\"\"Tests that serve_food() updates the 'has_munchies' column.\"\"\"\n\n def test_serve_food_updates_df(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test serve_food updates the has_munchies columns as expected.\n\n This function will update each fixture in the same way, providing each\n character with their favourite_food and updating the contents of their\n stomach. The column we will assert against will be has_munchies, which\n should be updated to False after feeding in all cases except for Scooby\n Doo and Shaggy, who always have the munchies.\n \"\"\"\n # first lets check that the session-scoped dataframe gets updates\n assert list(serve_food(_mm_session_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the session-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # next check the same for the module-scoped fixture\n assert list(serve_food(_mm_module_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the module-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # Next check class-scoped fixture updates\n assert list(serve_food(_mm_class_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the class-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # Finally check the function-scoped df does the same...\n assert list(\n serve_food(_mm_function_scoped)[\"has_munchies\"].values\n ) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the function-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n\n```\n\nNotice that the test makes exactly the same assertion for every differently\nscoped fixture? In every instance, we have fed the characters in the mystery\nmachine DataFrame and therefore everyone's `has_munchies` status (apart from\nScooby Doo and Shaggy's) gets updated to `False`.\n\n:::{.callout-tip collapse=\"true\"}\n\n##### **Parametrized Tests**\n\nWriting the test out this way makes things explicit and easy to follow.\nHowever, you could make this test smaller by using a neat feature of the\n`pytest` package called **parametrized** tests. This is basically like applying\nconditions to your tests in a `for` loop. Perhaps you have a bunch of\nconditions to check, multiple DataFrames or whatever. These can be\nprogrammatically served with\n[parametrized tests](https://docs.pytest.org/en/7.1.x/how-to/parametrize.html).\nWhile outside of the scope of this article, I intend to write a blog on this in\nthe future.\n:::\n\nNext, we can add to the test class, including a new test that checks the state\nof the fixtures. At this point, we will start to see some differences due to\nscoping. The new `test_expected_states_within_same_class()` will assert that\nthe changes to the fixtures brought about in the previous test\n`test_serve_food_updates_df()` will persist, except for the the case of\n`_mm_function_scoped` which will go through teardown at the end of every test\nfunction.\n\n```{.python filename=\"test_feed_characters.py\"}\nclass TestServeFood:\n \"\"\"Tests that serve_food() updates the 'has_munchies' column.\"\"\"\n # ... (test_serve_food_updates_df)\n\n def test_expected_states_within_same_class(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test to ensure fixture states are as expected.\"\"\"\n # Firstly, session-scoped changes should persist, only Scooby Doo &\n # Shaggy should still have the munchies...\n assert list(_mm_session_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the session-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Secondly, module-scoped changes should persist, as was the case for\n # the session-scope test above\n assert list(_mm_module_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the module-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Next, class-scoped changes should persist just the same\n assert list(_mm_class_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the class-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Finally, demonstrate that function-scoped fixture starts from scratch\n # Therefore all characters should have the munchies all over again.\n assert (\n list(_mm_function_scoped[\"has_munchies\"].values) == [True] * 5\n ), (\n \"The function_scoped df 'has_munchies' column is not as expected.\",\n )\n\n```\n\nIn the above test, we assert that the function-scoped fixture values have\nthe original fixture's values. The function-scoped fixture goes through set-up\nagain as `test_expected_states_within_same_class` is executed, ensuring a\n'fresh', unchanged version of the fixture DataFrame is provided. \n\nWithin the same test module, we can add some other test class and make\nassertions about the fixtures. This new test will check whether the\n`stomach_contents` column of the module and class-scoped fixtures have been\nupdated. Recall that the characters start out with `\"empty\"` stomach contents.\n\n```{.python filename=\"test_feed_characters.py\"}\n\n# ... (TestServeFood)\n # (test_serve_food_updates_df)\n # (test_expected_states_within_same_class) ...\n\nclass TestSomeOtherTestClass:\n \"\"\"Demonstrate persistence of changes to class-scoped fixture.\"\"\"\n\n def test_whether_changes_to_stomach_contents_persist(\n self, _mm_class_scoped, _mm_module_scoped\n ):\n \"\"\"Check the stomach_contents column.\"\"\"\n assert list(_mm_module_scoped[\"stomach_contents\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], \"Changes to module-scoped fixture have not propagated as expected.\"\n assert (\n list(_mm_class_scoped[\"stomach_contents\"].values) == [\"empty\"] * 5\n ), \"Values in class-scoped fixture are not as expected\"\n```\n\nIn this example, it is demonstrated that changes to the class-scoped fixture \nhave been discarded. As `test_whether_changes_to_stomach_contents_persist()`\nexists within a new class called `TestSomeOtherTestClass`, the code for\n`_mm_class_scoped` has been executed again, providing the original DataFrame\nvalues.\n\n##### **Balancing Isolation & Persistence**\n***\n\nWhile the persistence of fixtures may be useful for end to end tests, this\napproach reduces isolation in the test suite. Be aware that this may introduce\na bit of friction to your `pytest` development process. For example, it can be\ncommonplace to develop a new test and to check that it passes by invoking\n`pytest` with the keyword `-k` flag to run that single test\n(or subset of tests) only. This approach is useful if you have a costly test\nsuite and you just want to examine changes in a single unit.\n\nAt the current state of the test module, executing the entire test module by\nrunning `pytest ./tests/test_feed_characters.py` will pass. However, running\n`pytest -k \"TestSomeOtherTestClass\"` will fail. This is because the assertions\nin `TestSomeOtherTestClass` rely on code being executed within the preceding\ntest class. Tests in `TestSomeOtherTestClass` rely on changes elsewhere in your\ntest suite and by definition are no longer unit tests. For those developers who\nwork with [pytest-randomly](https://pypi.org/project/pytest-randomly/) to help\nsniff out poorly-isolated tests, this approach could cause a bit of a headache.\n\nA good compromise would be to ensure that the use of fixture scopes other than\n`function` are isolated and clearly documented within a test suite. Thoughtful\ngrouping of integration tests within test modules or classes can limit grief \nfor collaborating developers. Even better would be to\n[mark tests](https://docs.pytest.org/en/latest/how-to/mark.html) according to\ntheir scoped dependencies. This approach allows tests to be grouped and\nexecuted separately, though the implementation of this is beyond the scope of\nthis article. \n\n#### Fixtures Across Multiple Test Modules\n\nFinally in this section, we will explore fixture behaviour across more than one\ntest module. Below I define a new source module with a function used to update\nthe `mystery_machine` DataFrame. This function will update the `fave_food`\ncolumn for a character if it has already eaten. This is meant to represent a\ncharacter's preference for a dessert following a main course. Once more, this\nfunction will not deep copy the input DataFrame but will allow inplace\nadjustment.\n\n\"Delicious\n\n```{.python filename=\"update_food.py\"}\n\"\"\"Helping learners understand how to work with pytest fixtures.\"\"\"\nimport pandas as pd\n\n\ndef fancy_dessert(\n df: pd.DataFrame,\n fave_desserts: dict = {\n \"Daphne\": \"brownie\",\n \"Fred\": \"ice cream\",\n \"Scooby Doo\": \"apple crumble\",\n \"Shaggy\": \"pudding\",\n \"Velma\": \"banana bread\",\n },\n) -> pd.DataFrame:\n \"\"\"Update a characters favourite_food to a dessert if they have eaten.\n\n Iterates over a df, updating the fave_food value for a character if the\n stomach_contents are not 'empty'.\n\n Parameters\n ----------\n df : pd.DataFrame\n A dataframe with the following columns: \"name\": str, \"fave_food\": str,\n \"has_munchies\": bool, \"stomach_contents\": str.\n fave_desserts : dict, optional\n A mapping of \"name\" to a replacement favourite_food, by default\n { \"Daphne\": \"brownie\", \"Fred\": \"ice cream\",\n \"Scooby Doo\": \"apple crumble\", \"Shaggy\": \"pudding\",\n \"Velma\": \"banana bread\", }\n\n Returns\n -------\n pd.DataFrame\n Dataframe with updated fave_food values.\n\n \"\"\"\n for ind, row in df.iterrows():\n if row[\"stomach_contents\"] != \"empty\":\n # character has eaten, now they should prefer a dessert\n character = row[\"name\"]\n dessert = fave_desserts[character]\n print(f\"{character} now wants {dessert}.\")\n df.loc[ind, \"fave_food\"] = dessert\n else:\n # if not eaten, do not adjust\n pass\n return df\n\n```\nNote that the condition required for `fancy_dessert()` to take action is that\nthe contents of the character's `stomach_contents` should be not equal to\n\"empty\". Now to test this new src module, we create a new test module. We will\nrun assertions of the `fave_food` columns against the differently-scoped\nfixtures. \n\n```{.python filename=\"test_update_food.py\"}\n\"\"\"Testing pandas operations with test fixtures.\"\"\"\nfrom example_pkg.update_food import fancy_dessert\n\n\nclass TestFancyDessert:\n \"\"\"Tests for fancy_dessert().\"\"\"\n\n def test_fancy_dessert_updates_fixtures_as_expected(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test fancy_dessert() changes favourite_food values to dessert.\n\n These assertions depend on the current state of the scoped fixtures. If\n changes performed in\n test_feed_characters::TestServeFood::test_serve_food_updates_df()\n persist, then characters will not have empty stomach_contents,\n resulting in a switch of their favourite_food to dessert.\n \"\"\"\n # first, check update_food() with the session-scoped fixture.\n assert list(fancy_dessert(_mm_session_scoped)[\"fave_food\"].values) == [\n \"brownie\",\n \"ice cream\",\n \"apple crumble\",\n \"pudding\",\n \"banana bread\",\n ], (\n \"The changes to the session-scoped df 'stomach_contents' column\",\n \" have not persisted as expected.\",\n )\n # next, check update_food() with the module-scoped fixture.\n assert list(fancy_dessert(_mm_module_scoped)[\"fave_food\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], (\n \"The module-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n # now, check update_food() with the class-scoped fixture. Note that we\n # are now making assertions about changes from a different class.\n assert list(fancy_dessert(_mm_class_scoped)[\"fave_food\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], (\n \"The class-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n # Finally, check update_food() with the function-scoped fixture. As\n # in TestServeFood::test_expected_states_within_same_class(), the\n # function-scoped fixture starts from scratch.\n assert list(\n fancy_dessert(_mm_function_scoped)[\"fave_food\"].values\n ) == [\"carrots\", \"beans\", \"scooby snacks\", \"burgers\", \"hot dogs\"], (\n \"The function-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n\n```\nNote that the only fixture expected to have been adjusted by `update_food()` is\n`_mm_session_scoped`. When running the `pytest` command, changes from executing\nthe first test module `test_feed_characters.py` propagate for this fixture\nonly. All other fixture scopes used will go through teardown and then setup\nonce more on execution of the second test module.\n\nThis arrangement is highly dependent on the order of which the test modules are\ncollected. `pytest` collects tests in alphabetical ordering by default, and as\nsuch `test_update_food.py` can be expected to be executed after\n`test_feed_characters.py`. This test module is highly dependent upon the order\nof the `pytest` execution. This makes the tests less portable and means that\nrunning the test module with `pytest tests/test_update_food.py` in isolation\nwould fail. I would once more suggest using\n[`pytest` marks](https://docs.pytest.org/en/latest/how-to/mark.html) to group\nthese types of tests and execute them separately to the rest of the test suite.\n\n## `ScopeMismatch` Error\n\nWhen working with `pytest` fixtures, occasionally you will encounter a\n`ScopeMismatch` exception. This may happen when attempting to use certain\n`pytest` plug-ins or perhaps if trying to use temporary directory fixtures like\n`tmp_path` with fixtures that are scoped differently to function-scope.\nOccasionally, you may encounter this exception when attempting to reference\nyour own fixture in other fixtures, as was done with the\n[`mystery_machine` fixture above](#define-fixtures). \n\nThe reason for `ScopeMismatch` is straightforward. Fixture scopes have a\nhierarchy, based on their persistence:\n\n> function < class < module < package < session\n\nFixtures with a greater scope in the hierarchy are not permitted to reference\nthose lower in the hierarchy. The way I remember this rule is that:\n\n> Fixtures must only reference equal or greater scopes.\n\nIt is unclear why this rule has been implemented other than to reduce\ncomplexity (which is reason enough in my book). There was talk about\nimplementing `scope=\"any\"` some time ago, but it looks like this idea was\nabandoned. To reproduce the error:\n\n```{.python filename=\"test_bad_scoping.py\"}\n\"\"\"Demomstrate ScopeMismatch error.\"\"\"\n\nimport pytest\n\n@pytest.fixture(scope=\"function\")\ndef _fix_a():\n return 1\n\n@pytest.fixture(scope=\"class\")\ndef _fix_b(_fix_a):\n return _fix_a + _fix_a\n\n\ndef test__fix_b_return_val(_fix_b):\n assert _fix_b == 2\n\n```\nExecuting this test module results in:\n```\n================================= ERRORS ======================================\n________________ ERROR at setup of test__fix_b_return_val _____________________\nScopeMismatch: You tried to access the function scoped fixture _fix_a with a\nclass scoped request object, involved factories:\ntests/test_bad_scoping.py:9: def _fix_b(_fix_a)\ntests/test_bad_scoping.py:5: def _fix_a()\n========================== short test summary info ============================\nERROR tests/test_bad_scoping.py::test__fix_b_return_val - Failed:\nScopeMismatch: You tried to access the function scoped fixture _fix_a with a\nclass scoped request object, involved factories:\n=========================== 1 error in 0.01s ==================================\n```\n\nThis error can be avoided by adjusting the fixture scopes to adhere to the\nhierarchy rule, so updating `_fix_a` to use a class scope or greater would\nresult in a passing test.\n\n## Summary\n\nHopefully by now you feel comfortable in when and how to use fixtures for\n`pytest`. We've covered quite a bit, including:\n\n* What fixtures are\n* Use-cases\n* Where to store them\n* How to reference them\n* How to scope them\n* How changes to fixtures persist or not\n* Handling scope errors \n\nIf you spot an error with this article, or have suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Clara\n* Dan C\n* Dan S\n* Edward\n* Ethan\n* Henry\n* Ian\n* Iva\n* Jay\n* Mark\n* Martin R\n* Martin W\n* Mat\n* Sergio\n\n

fin!

\n\n", + "markdown": "---\ntitle: Pytest Fixtures in Plain English\nauthor: Rich Leyshon\ndate: April 07 2024\ndescription: Plain English Discussion of Pytest Fixtures.\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - fixtures\n - pytest-in-plain-english\nimage: 'https://images.pixexid.com/sculptural-simplicity-monochrome-background-highlighting-the-beauty-of-minimali-jmhkipzb.jpeg?h=699&q=70'\nimage-alt: 'Sculptural simplicity, monochrome background highlighting the beauty of minimalist sculptures by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)'\ntoc: true\n---\n\n
\n \n
Creative commons license by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses using test data as fixtures\nwith `pytest` and is the first in a series of blogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\npytest documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend the audience's understanding of the more\nintricate features of `pytest` by describing their utility with simple code\nexamples. \n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\noptimise their test code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[fixtures branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/fixtures)\nof the example code repo.\n\n## What are fixtures?\n\nData. Well, data provided specifically for testing purposes. This is the\nessential definition for a fixture. One could argue the case that fixtures are\nmore than this. Fixtures could be environment variables, class instances,\nconnection to a server or whatever dependencies your code needs to run.\n\nI would agree that fixtures are not just data. But that all fixtures return\ndata of some sort, regardless of the system under test.\n\n## When would you use fixtures?\n\nIt's a bad idea to commit data to a git repository, right? Agreed. Though\nfixtures are rarely 'real' data. The data used for testing purposes should be\nminimal and are usually synthetic. \n\n**Minimal fixtures** conform to the schema of the actual data that the system\nrequires. These fixtures will be as small as possible while capturing all known\nimportant cases. Keeping the data small maintains a performant test suite and\navoids problems associated with large files and git version control.\n\nIf you have ever encountered a problem in a system that was caused by a\nproblematic record in the data, the aspect of that record that broke your\nsystem should absolutely make it into the next version of your minimal test\nfixture. Writing a test that checks that the codebase can handle such problem\nrecords is known as 'regression testing' - safeguarding against old bugs\nresurfacing when code is refactored or new features are implemented. This\nscenario commonly occurs when a developer unwittingly violates Chesterton's\nPrinciple.\n\n\n\n\nMany thanks to my colleague Mat for pointing me towards this useful analogy. A\nconsiderate developer would probably include a comment in their code about a\nspecific problem that they've handled (like erecting a sign next to\nChesterton's fence). An experienced developer would do the same, and also write\na regression test to ensure the problem doesn't re-emerge in the future\n(monitoring the fence with CCTV...). Discovering these problem cases and\nemploying defensive strategies avoids future pain for yourself and colleagues.\n\nAs you can imagine, covering all the important cases while keeping the fixture\nminimal is a compromise. At the outset of the work, it may not be obvious what\nproblematic cases may arise. Packages such as\n[`hypothesis`](https://hypothesis.readthedocs.io/en/latest/) allow you to\ngenerate awkward cases. Non-utf-8 strings anyone? Hypothesis can generate\nthese test cases for you, along with many more interesting edge-cases -\n*ăѣ𝔠ծềſģȟᎥ𝒋ǩľḿꞑȯ𝘱𝑞𝗋𝘴ȶ𝞄𝜈ψ𝒙𝘆𝚣* (Non-utf8 strings often cause problems for web apps). \n\n**Non-disclosive fixtures** are those that do not expose personally\nidentifiable or commercially-sensitive information. If you are working with\nthis sort of data, it is necessary to produce toy test fixtures that mimic the\nschema of the real data. Names and addresses can be de-identified to random\nalphanumeric strings. Location data can be adjusted with noise. The use of\ndummy variables or categories can mitigate the risk of disclosure by\ndifferencing.\n\nBy adequately anonymising data and testing problem cases, the programmer\nexhibits upholds duties under the General Data Protection Regulation:\n\n> accurately store, process, retain and erase personally-identifiable information.\n\nIn cases where the system integrates with data available in the public domain,\nit is may be permissible to include a small sample of the data as a test\nfixture. Ensure the license that the data is distributed under is compatible\nwith your code's license. If the license is compatible, I recommend including a\nreference to the fixture, its source and license within a LICENSE.note file.\nThis practice is enforced by Comprehensive R Archive Network. You can read more\nabout this in the\n[R Packages documentation](https://r-pkgs.org/license.html#sec-how-to-include).\n\n## Scoping fixtures\n\n`pytest` fixtures have different scopes, meaning that they will be prepared\ndifferently dependent on the scope you specify. The available scopes are as\nfollows: \n\n| Scope Name | Teardown after each |\n| ----------- | ------------------- |\n| function | test function |\n| class | test class |\n| module | test module |\n| package | package under test |\n| session | `pytest` session |\n\nNote that the default scope for any fixtures that you define will be\n'function'. A function-scoped fixture will be set up for every test function\nthat requires it. Once the function has executed, the fixture will then be\ntorn down and all changes to this fixture will be lost. This default behaviour\nencourages isolation in your test suite. Meaning that the tests have no\ndependencies upon each other. The test functions could be run in any order\nwithout affecting the results of the test. Function-scoped fixtures are the\nshortest-lived fixtures. Moving down the table above, the persistence of the\nfixtures increases. Changes to a session-scoped fixture persist for the entire\ntest execution duration, only being torn down once `pytest` has executed all\ntests.\n\n### Scoping for performance\n***\n\n> performance vs isolation\n\nBy definition, a unit test is completely isolated, meaning that it will have no\ndependencies other than the code it needs to test. However, there may be a few\ncases where this would be less desirable. Slow test suites may introduce\nexcessive friction to the software development process. Persistent fixtures can\nbe used to improve the performance of a test suite. \n\nFor example, here we define some expensive class:\n\n```{.python filename=expensive.py}\n\"\"\"A module containing an expensive class definition.\"\"\"\nimport time\nfrom typing import Union\n\n\nclass ExpensiveDoodah:\n \"\"\"A toy class that represents some costly operation.\n\n This class will sleep for the specified number of seconds on instantiation.\n\n Parameters\n ----------\n sleep_time : Union[int, float]\n Number of seconds to wait on init.\n\n \"\"\"\n def __init__(self, sleep_time: Union[int, float] = 2):\n print(f\"Sleeping for {sleep_time} seconds\")\n time.sleep(sleep_time)\n return None\n\n```\n\nThis class will be used to demonstrate the effect of scoping with some costly\noperation. This example could represent reading in a bulky xlsx file or\nquerying a large database.\n\nTo serve `ExpensiveDoodah` to our tests, I will define a function-scoped\nfixture. To do this, we use a `pytest` fixture decorator to return the class\ninstance with a specified sleep time of 2 seconds.\n\n```{.python filename=test_expensive.py}\nimport pytest\n\nfrom example_pkg.expensive import ExpensiveDoodah\n\n\n@pytest.fixture(scope=\"function\")\ndef module_doodah():\n \"\"\"Function-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n```\nNow to test `ExpensiveDoodah` we extend our test module to include a test class\nwith 3 separate test functions. The assertions will all be the same for this\nsimple example - that `ExpensiveDoodah` executes without raising any error\nconditions. Notice we must pass the name of the fixture in each test function's\nsignature.\n\n```{.python filename=test_expensive.py}\n\"\"\"Tests for expensive.py using function-scoped fixture.\"\"\"\nfrom contextlib import nullcontext as does_not_raise\nimport pytest\n\nfrom example_pkg.expensive import ExpensiveDoodah\n\n\n@pytest.fixture(scope=\"function\")\ndef doodah_fixture():\n \"\"\"Function-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n\nclass TestA:\n \"\"\"A test class.\"\"\"\n\n def test_1(self, doodah_fixture):\n \"\"\"Test 1.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n def test_2(self, doodah_fixture):\n \"\"\"Test 2.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n def test_3(self, doodah_fixture):\n \"\"\"Test 3.\"\"\"\n with does_not_raise():\n doodah_fixture\n\n```\n\nThe result of running this test module can be seen below:\n\n```\ncollected 3 items\n\n./tests/test_expensive_function_scoped.py ... [100%]\n\n============================ 3 passed in 6.04s ================================\n\n```\n\nNotice that the test module took just over 6 seconds to execute because the\nfunction-scoped fixture was set up once for each test function.\n\nIf instead we had defined `doodah_fixture` with a different scope, it would\nreduce the time for the test suite to complete by approximately two thirds.\nThis is the sort of benefit that can be gained from considerate use of `pytest`\nfixtures.\n\n```{.python filename=test_expensive.py}\n@pytest.fixture(scope=\"module\")\ndef doodah_fixture():\n \"\"\"Module-scoped ExpensiveDoodah.\"\"\"\n return ExpensiveDoodah(2)\n\n```\n\n```\ncollected 3 items\n\n./tests/test_expensive_function_scoped.py ... [100%]\n\n============================ 3 passed in 2.02s ================================\n\n```\n\nThe scoping feature of `pytest` fixtures can be used to optimise a test-suite\nand avoid lengthy delays while waiting for your test suites to execute.\nHowever, any changes to the fixture contents will persist until the fixture is\nnext torn down. Keeping track of the states of differently-scoped fixtures in a\ncomplex test suite can be tricky and reduces segmentation overall. Bear this in\nmind when considering which scope best suits your needs.\n\n### Scope persistence\n***\n\n> function < class < module < package < session\n\nUsing scopes other than 'function' can be useful for end-to-end testing.\nPerhaps you have a complex analytical pipeline and need to check that the\nvarious components work well **together**, rather than in isolation as with a\nunit test. This sort of test can be extremely useful for developers in a rush.\nYou can test that the so called 'promise' of the codebase is as expected, even\nthough the implementation may change.\n\nThe analogy here would be that the success criteria of a SatNav is that it gets\nyou to your desired destination whatever the suggested route you selected.\nChecking that you used the fastest or most fuel efficient option is probably a\ngood idea. But if you don't have time, you'll just have to take the hit if you\nencounter a toll road. Though it's still worth checking that the postcode you\nhastily input to the satnav is the correct one.\n\n\n
\nPerhaps your success criteria is that you need to write a DataFrame to file. \nA great end-to-end test would check that the DataFrame produced has the\nexpected number of rows, or even has rows! Of course it's also a good idea to\ncheck the DataFrame conforms to the expected table schema, too: number of\ncolumns, names of columns, order and data types. This sort of check is often\noverlooked in favour of pressing on with development. If you've ever\nencountered a situation where you've updated a codebase and later realised you\nnow have empty tables (I certainly have), this sort of test would be really\nhandy, immediately alerting you to this fact and helping you efficiently locate\nthe source of the bug.\n\n#### Define Data\n\nIn this part, I will explore the scoping of fixtures with DataFrames. Again,\nI'll use a toy example to demonstrate scope behaviour. Being a child of the\n'90s (mostly), I'll use a scenario from my childhood. Scooby Doo is still a\nthing, right?\n\n**Enter: The Mystery Machine**\n\n\"Scooby\n\n\nThe scenario: The passengers of the Mystery Machine van all have the munchies.\nThey stop at a 'drive thru' to get some takeaway. We have a table with a record\nfor each character. We have columns with data about the characters' names,\ntheir favourite food, whether they have 'the munchies', and the contents of\ntheir stomach.\n\n::: {#1e860ea0 .cell execution_count=1}\n``` {.python .cell-code}\nimport pandas as pd\nmystery_machine = pd.DataFrame(\n {\n \"name\": [\"Daphne\", \"Fred\", \"Scooby Doo\", \"Shaggy\", \"Velma\"],\n \"fave_food\": [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ],\n \"has_munchies\": [True] * 5, # everyone's hungry\n \"stomach_contents\": [\"empty\"] * 5, # all have empty stomachs\n }\n )\nmystery_machine\n```\n\n::: {.cell-output .cell-output-display execution_count=1}\n```{=html}\n
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
namefave_foodhas_munchiesstomach_contents
0DaphnecarrotsTrueempty
1FredbeansTrueempty
2Scooby Dooscooby snacksTrueempty
3ShaggyburgersTrueempty
4Velmahot dogsTrueempty
\n
\n```\n:::\n:::\n\n\nTo use this simple DataFrame as a fixture, I could go ahead and define it with\n`@pytest.fixture()` directly within a test file. But if I would like to share\nit across several test modules\n([as implemented later](#fixtures-across-multiple-test-modules)), then there\nare 2 options:\n\n1. Write the DataFrame to disk as csv (or whatever format you prefer) and save\nin a `./tests/data/` folder. At the start of your test modules you can read the\ndata from disk and use it for testing. In this approach you'll likely define\nthe data as a test fixture in each of the test modules that need to work with\nit.\n2. Define the fixtures within a special python file called `conftest.py`, which\nmust be located at the root of your project. This file is used to configure\nyour tests. `pytest` will look in this file for any required fixture\ndefinitions when executing your test suite. If it finds a fixture with the same\nname as that required by a test, the fixture code may be run. \n\n:::{.callout-caution}\nWait! Did you just say '**may** be run'?\n:::\n\nDepending on the scope of your fixture, `pytest` may not need to execute the\ncode for each test. For example, let's say we're working with a session-scoped\nfixture. This type of fixture will persist for the duration of the entire test\nsuite execution. Imagine test number 1 and 10 both require this test fixture.\nThe fixture definition only gets executed the first time a test requires it.\nThis test fixture will be set up as test 1 executes and will persist until\ntear down occurs at the end of the `pytest` session. Test 10 will therefore use\nthe same instance of this fixture as test 1 used, meaning any changes to the\nfixture may be carried forward.\n\n#### Define fixtures\n\nFor our example, we will create a `conftest.py` file and define some fixtures\nwith differing scopes.\n\n```{.python filename=conftest.py}\n\"\"\"Demonstrate scoping fixtures.\"\"\"\nimport pandas as pd\nimport pytest\n\n\n@pytest.fixture(scope=\"session\")\ndef _mystery_machine():\n \"\"\"Session-scoped fixture returning pandas DataFrame.\"\"\"\n return pd.DataFrame(\n {\n \"name\": [\"Daphne\", \"Fred\", \"Scooby Doo\", \"Shaggy\", \"Velma\"],\n \"fave_food\": [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ],\n \"has_munchies\": [True] * 5,\n \"stomach_contents\": [\"empty\"] * 5,\n }\n )\n\n\n@pytest.fixture(scope=\"session\")\ndef _mm_session_scoped(_mystery_machine):\n \"\"\"Session-scoped fixture returning the _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"module\")\ndef _mm_module_scoped(_mystery_machine):\n \"\"\"Module-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"class\")\ndef _mm_class_scoped(_mystery_machine):\n \"\"\"Class-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n\n@pytest.fixture(scope=\"function\")\ndef _mm_function_scoped(_mystery_machine):\n \"\"\"Function-scoped _mystery_machine DataFrame.\"\"\"\n return _mystery_machine.copy(deep=True)\n\n```\n\nFixtures can reference each other, if they're scoped correctly. More on this in\n[the next section](#scopemismatch-error). This is useful for my toy example as\nI intend the source functions to update the DataFrames directly, if I wasn't\ncareful about deep copying the fixtures, my functions would update the original\n`_mystery_machine` fixture's table. Those changes would then be subsequently\npassed to the other fixtures, meaning I couldn't clearly demonstrate how the\ndifferent scopes persist.\n\n#### Define the source functions\n\nNow, let's create a function that will feed characters their favourite food if\nthey have the munchies. \n\n```{.python filename=feed_characters.py}\n\"\"\"Helping learners understand how to work with pytest fixtures.\"\"\"\nimport pandas as pd\n\n\ndef serve_food(df: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Serve characters their desired food.\n\n Iterates over a df, feeding characters if they have 'the munchies' with\n their fave_food. If the character is not Scooby Doo or Shaggy, then update\n their has_munchies status to False. The input df is modified inplace.\n\n Parameters\n ----------\n df : pd.DataFrame\n A DataFrame with the following columns: \"name\": str, \"fave_food\": str,\n \"has_munchies\": bool, \"stomach_contents\": str.\n\n Returns\n -------\n pd.DataFrame\n Updated DataFrame with new statuses for stomach_contents and\n has_munchies.\n\n \"\"\"\n for ind, row in df.iterrows():\n if row[\"has_munchies\"]:\n # if character is hungry then feed them\n food = row[\"fave_food\"]\n character = row[\"name\"]\n print(f\"Feeding {food} to {character}.\")\n df.loc[ind, [\"stomach_contents\"]] = food\n if character not in [\"Scooby Doo\", \"Shaggy\"]:\n # Scooby & Shaggy are always hungry\n df.loc[ind, \"has_munchies\"] = False\n else:\n # if not hungry then do not adjust\n pass\n return df\n\n```\n\nNote that it is commonplace to copy a pandas DataFrame so that any operations\ncarried out by the function are confined to the function's scope. To\ndemonstrate changes to the fixtures I will instead choose to edit the DataFrame\ninplace.\n\n#### Fixtures Within a Single Test Module\n\nNow to write some tests. To use the fixtures we defined earlier, we simply \ndeclare that a test function requires the fixture. `pytest` will notice this \ndependency on collection, check the fixture scope and execute the fixture code \nif appropriate. The following test `test_scopes_before_action` checks that the\n`mystery_machine` fixtures all have the expected `has_munchies` column values\nat the outset of the test module, i.e. everybody is hungry before our source\nfunction takes some action. This type of test doesn't check behaviour of any\nsource code and therefore would be unnecessary for quality assurance purposes.\nBut I include it here to demonstrate the simple use of fixtures and prove to\nthe reader the state of the DataFrame fixtures prior to any source code\nintervention.\n\n:::{.callout-tip collapse=\"true\"}\n\n##### **Testing `pandas` DataFrames**\n\nYou may notice that the assert statements in the tests below requires pulling \ncolumn values out and casting to lists. The `pandas` package has its own\ntesting module that is super useful for testing all aspects of DataFrames.\nCheck out the\n[`pandas` testing documentation](https://pandas.pydata.org/docs/reference/testing.html)\nfor more on how to write robust tests for pandas DataFrames and Series.\n\n:::\n\n```{.python filename=\"test_feed_characters.py\"}\n\"\"\"Testing pandas operations with test fixtures.\"\"\"\nfrom example_pkg.feed_characters import serve_food\n\n\ndef test_scopes_before_action(\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n):\n \"\"\"Assert that all characters have the munchies at the outset.\"\"\"\n assert list(_mm_session_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The session-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_module_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The module-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_class_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The class-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n assert list(_mm_function_scoped[\"has_munchies\"].values) == [True] * 5, (\n \"The function-scoped DataFrame 'has_munchies' column was not as \",\n \"expected before any action was taken.\",\n )\n\n```\n\nNow to test the `serve_food()` function operates as expected. We can define\na test class that will house all tests for `serve_food()`. Within that class\nlet's define our first test that simply checks that the value of the\n`has_munchies` column has been updated as we would expect after using the\n`serve_food()` function.\n\n```{.python filename=\"test_feed_characters.py\"}\nclass TestServeFood:\n \"\"\"Tests that serve_food() updates the 'has_munchies' column.\"\"\"\n\n def test_serve_food_updates_df(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test serve_food updates the has_munchies columns as expected.\n\n This function will update each fixture in the same way, providing each\n character with their favourite_food and updating the contents of their\n stomach. The column we will assert against will be has_munchies, which\n should be updated to False after feeding in all cases except for Scooby\n Doo and Shaggy, who always have the munchies.\n \"\"\"\n # first lets check that the session-scoped dataframe gets updates\n assert list(serve_food(_mm_session_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the session-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # next check the same for the module-scoped fixture\n assert list(serve_food(_mm_module_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the module-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # Next check class-scoped fixture updates\n assert list(serve_food(_mm_class_scoped)[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the class-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n # Finally check the function-scoped df does the same...\n assert list(\n serve_food(_mm_function_scoped)[\"has_munchies\"].values\n ) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The `serve_food()` has not updated the function-scoped df\",\n \" 'has_munchies' column as expected.\",\n )\n\n```\n\nNotice that the test makes exactly the same assertion for every differently\nscoped fixture? In every instance, we have fed the characters in the mystery\nmachine DataFrame and therefore everyone's `has_munchies` status (apart from\nScooby Doo and Shaggy's) gets updated to `False`.\n\n:::{.callout-tip collapse=\"true\"}\n\n##### **Parametrized Tests**\n\nWriting the test out this way makes things explicit and easy to follow.\nHowever, you could make this test smaller by using a neat feature of the\n`pytest` package called **parametrized** tests. This is basically like applying\nconditions to your tests in a `for` loop. Perhaps you have a bunch of\nconditions to check, multiple DataFrames or whatever. These can be\nprogrammatically served with\n[parametrized tests](https://docs.pytest.org/en/7.1.x/how-to/parametrize.html).\nWhile outside of the scope of this article, I intend to write a blog on this in\nthe future.\n:::\n\nNext, we can add to the test class, including a new test that checks the state\nof the fixtures. At this point, we will start to see some differences due to\nscoping. The new `test_expected_states_within_same_class()` will assert that\nthe changes to the fixtures brought about in the previous test\n`test_serve_food_updates_df()` will persist, except for the the case of\n`_mm_function_scoped` which will go through teardown at the end of every test\nfunction.\n\n```{.python filename=\"test_feed_characters.py\"}\nclass TestServeFood:\n \"\"\"Tests that serve_food() updates the 'has_munchies' column.\"\"\"\n # ... (test_serve_food_updates_df)\n\n def test_expected_states_within_same_class(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test to ensure fixture states are as expected.\"\"\"\n # Firstly, session-scoped changes should persist, only Scooby Doo &\n # Shaggy should still have the munchies...\n assert list(_mm_session_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the session-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Secondly, module-scoped changes should persist, as was the case for\n # the session-scope test above\n assert list(_mm_module_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the module-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Next, class-scoped changes should persist just the same\n assert list(_mm_class_scoped[\"has_munchies\"].values) == [\n False,\n False,\n True,\n True,\n False,\n ], (\n \"The changes to the class-scoped df 'has_munchies' column have\",\n \" not persisted as expected.\",\n )\n # Finally, demonstrate that function-scoped fixture starts from scratch\n # Therefore all characters should have the munchies all over again.\n assert (\n list(_mm_function_scoped[\"has_munchies\"].values) == [True] * 5\n ), (\n \"The function_scoped df 'has_munchies' column is not as expected.\",\n )\n\n```\n\nIn the above test, we assert that the function-scoped fixture values have\nthe original fixture's values. The function-scoped fixture goes through set-up\nagain as `test_expected_states_within_same_class` is executed, ensuring a\n'fresh', unchanged version of the fixture DataFrame is provided. \n\nWithin the same test module, we can add some other test class and make\nassertions about the fixtures. This new test will check whether the\n`stomach_contents` column of the module and class-scoped fixtures have been\nupdated. Recall that the characters start out with `\"empty\"` stomach contents.\n\n```{.python filename=\"test_feed_characters.py\"}\n\n# ... (TestServeFood)\n # (test_serve_food_updates_df)\n # (test_expected_states_within_same_class) ...\n\nclass TestSomeOtherTestClass:\n \"\"\"Demonstrate persistence of changes to class-scoped fixture.\"\"\"\n\n def test_whether_changes_to_stomach_contents_persist(\n self, _mm_class_scoped, _mm_module_scoped\n ):\n \"\"\"Check the stomach_contents column.\"\"\"\n assert list(_mm_module_scoped[\"stomach_contents\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], \"Changes to module-scoped fixture have not propagated as expected.\"\n assert (\n list(_mm_class_scoped[\"stomach_contents\"].values) == [\"empty\"] * 5\n ), \"Values in class-scoped fixture are not as expected\"\n```\n\nIn this example, it is demonstrated that changes to the class-scoped fixture \nhave been discarded. As `test_whether_changes_to_stomach_contents_persist()`\nexists within a new class called `TestSomeOtherTestClass`, the code for\n`_mm_class_scoped` has been executed again, providing the original DataFrame\nvalues.\n\n##### **Balancing Isolation & Persistence**\n***\n\nWhile the persistence of fixtures may be useful for end to end tests, this\napproach reduces isolation in the test suite. Be aware that this may introduce\na bit of friction to your `pytest` development process. For example, it can be\ncommonplace to develop a new test and to check that it passes by invoking\n`pytest` with the keyword `-k` flag to run that single test\n(or subset of tests) only. This approach is useful if you have a costly test\nsuite and you just want to examine changes in a single unit.\n\nAt the current state of the test module, executing the entire test module by\nrunning `pytest ./tests/test_feed_characters.py` will pass. However, running\n`pytest -k \"TestSomeOtherTestClass\"` will fail. This is because the assertions\nin `TestSomeOtherTestClass` rely on code being executed within the preceding\ntest class. Tests in `TestSomeOtherTestClass` rely on changes elsewhere in your\ntest suite and by definition are no longer unit tests. For those developers who\nwork with [pytest-randomly](https://pypi.org/project/pytest-randomly/) to help\nsniff out poorly-isolated tests, this approach could cause a bit of a headache.\n\nA good compromise would be to ensure that the use of fixture scopes other than\n`function` are isolated and clearly documented within a test suite. Thoughtful\ngrouping of integration tests within test modules or classes can limit grief \nfor collaborating developers. Even better would be to\n[mark tests](https://docs.pytest.org/en/latest/how-to/mark.html) according to\ntheir scoped dependencies. This approach allows tests to be grouped and\nexecuted separately, though the implementation of this is beyond the scope of\nthis article. \n\n#### Fixtures Across Multiple Test Modules\n\nFinally in this section, we will explore fixture behaviour across more than one\ntest module. Below I define a new source module with a function used to update\nthe `mystery_machine` DataFrame. This function will update the `fave_food`\ncolumn for a character if it has already eaten. This is meant to represent a\ncharacter's preference for a dessert following a main course. Once more, this\nfunction will not deep copy the input DataFrame but will allow inplace\nadjustment.\n\n\"Delicious\n\n```{.python filename=\"update_food.py\"}\n\"\"\"Helping learners understand how to work with pytest fixtures.\"\"\"\nimport pandas as pd\n\n\ndef fancy_dessert(\n df: pd.DataFrame,\n fave_desserts: dict = {\n \"Daphne\": \"brownie\",\n \"Fred\": \"ice cream\",\n \"Scooby Doo\": \"apple crumble\",\n \"Shaggy\": \"pudding\",\n \"Velma\": \"banana bread\",\n },\n) -> pd.DataFrame:\n \"\"\"Update a characters favourite_food to a dessert if they have eaten.\n\n Iterates over a df, updating the fave_food value for a character if the\n stomach_contents are not 'empty'.\n\n Parameters\n ----------\n df : pd.DataFrame\n A dataframe with the following columns: \"name\": str, \"fave_food\": str,\n \"has_munchies\": bool, \"stomach_contents\": str.\n fave_desserts : dict, optional\n A mapping of \"name\" to a replacement favourite_food, by default\n { \"Daphne\": \"brownie\", \"Fred\": \"ice cream\",\n \"Scooby Doo\": \"apple crumble\", \"Shaggy\": \"pudding\",\n \"Velma\": \"banana bread\", }\n\n Returns\n -------\n pd.DataFrame\n Dataframe with updated fave_food values.\n\n \"\"\"\n for ind, row in df.iterrows():\n if row[\"stomach_contents\"] != \"empty\":\n # character has eaten, now they should prefer a dessert\n character = row[\"name\"]\n dessert = fave_desserts[character]\n print(f\"{character} now wants {dessert}.\")\n df.loc[ind, \"fave_food\"] = dessert\n else:\n # if not eaten, do not adjust\n pass\n return df\n\n```\nNote that the condition required for `fancy_dessert()` to take action is that\nthe contents of the character's `stomach_contents` should be not equal to\n\"empty\". Now to test this new src module, we create a new test module. We will\nrun assertions of the `fave_food` columns against the differently-scoped\nfixtures. \n\n```{.python filename=\"test_update_food.py\"}\n\"\"\"Testing pandas operations with test fixtures.\"\"\"\nfrom example_pkg.update_food import fancy_dessert\n\n\nclass TestFancyDessert:\n \"\"\"Tests for fancy_dessert().\"\"\"\n\n def test_fancy_dessert_updates_fixtures_as_expected(\n self,\n _mm_session_scoped,\n _mm_module_scoped,\n _mm_class_scoped,\n _mm_function_scoped,\n ):\n \"\"\"Test fancy_dessert() changes favourite_food values to dessert.\n\n These assertions depend on the current state of the scoped fixtures. If\n changes performed in\n test_feed_characters::TestServeFood::test_serve_food_updates_df()\n persist, then characters will not have empty stomach_contents,\n resulting in a switch of their favourite_food to dessert.\n \"\"\"\n # first, check update_food() with the session-scoped fixture.\n assert list(fancy_dessert(_mm_session_scoped)[\"fave_food\"].values) == [\n \"brownie\",\n \"ice cream\",\n \"apple crumble\",\n \"pudding\",\n \"banana bread\",\n ], (\n \"The changes to the session-scoped df 'stomach_contents' column\",\n \" have not persisted as expected.\",\n )\n # next, check update_food() with the module-scoped fixture.\n assert list(fancy_dessert(_mm_module_scoped)[\"fave_food\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], (\n \"The module-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n # now, check update_food() with the class-scoped fixture. Note that we\n # are now making assertions about changes from a different class.\n assert list(fancy_dessert(_mm_class_scoped)[\"fave_food\"].values) == [\n \"carrots\",\n \"beans\",\n \"scooby snacks\",\n \"burgers\",\n \"hot dogs\",\n ], (\n \"The class-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n # Finally, check update_food() with the function-scoped fixture. As\n # in TestServeFood::test_expected_states_within_same_class(), the\n # function-scoped fixture starts from scratch.\n assert list(\n fancy_dessert(_mm_function_scoped)[\"fave_food\"].values\n ) == [\"carrots\", \"beans\", \"scooby snacks\", \"burgers\", \"hot dogs\"], (\n \"The function-scoped df 'stomach_contents' column was not as\",\n \" expected\",\n )\n\n```\nNote that the only fixture expected to have been adjusted by `update_food()` is\n`_mm_session_scoped`. When running the `pytest` command, changes from executing\nthe first test module `test_feed_characters.py` propagate for this fixture\nonly. All other fixture scopes used will go through teardown and then setup\nonce more on execution of the second test module.\n\nThis arrangement is highly dependent on the order of which the test modules are\ncollected. `pytest` collects tests in alphabetical ordering by default, and as\nsuch `test_update_food.py` can be expected to be executed after\n`test_feed_characters.py`. This test module is highly dependent upon the order\nof the `pytest` execution. This makes the tests less portable and means that\nrunning the test module with `pytest tests/test_update_food.py` in isolation\nwould fail. I would once more suggest using\n[`pytest` marks](https://docs.pytest.org/en/latest/how-to/mark.html) to group\nthese types of tests and execute them separately to the rest of the test suite.\n\n## `ScopeMismatch` Error\n\nWhen working with `pytest` fixtures, occasionally you will encounter a\n`ScopeMismatch` exception. This may happen when attempting to use certain\n`pytest` plug-ins or perhaps if trying to use temporary directory fixtures like\n`tmp_path` with fixtures that are scoped differently to function-scope.\nOccasionally, you may encounter this exception when attempting to reference\nyour own fixture in other fixtures, as was done with the\n[`mystery_machine` fixture above](#define-fixtures). \n\nThe reason for `ScopeMismatch` is straightforward. Fixture scopes have a\nhierarchy, based on their persistence:\n\n> function < class < module < package < session\n\nFixtures with a greater scope in the hierarchy are not permitted to reference\nthose lower in the hierarchy. The way I remember this rule is that:\n\n> Fixtures must only reference equal or greater scopes.\n\nIt is unclear why this rule has been implemented other than to reduce\ncomplexity (which is reason enough in my book). There was talk about\nimplementing `scope=\"any\"` some time ago, but it looks like this idea was\nabandoned. To reproduce the error:\n\n```{.python filename=\"test_bad_scoping.py\"}\n\"\"\"Demomstrate ScopeMismatch error.\"\"\"\n\nimport pytest\n\n@pytest.fixture(scope=\"function\")\ndef _fix_a():\n return 1\n\n@pytest.fixture(scope=\"class\")\ndef _fix_b(_fix_a):\n return _fix_a + _fix_a\n\n\ndef test__fix_b_return_val(_fix_b):\n assert _fix_b == 2\n\n```\nExecuting this test module results in:\n```\n================================= ERRORS ======================================\n________________ ERROR at setup of test__fix_b_return_val _____________________\nScopeMismatch: You tried to access the function scoped fixture _fix_a with a\nclass scoped request object, involved factories:\ntests/test_bad_scoping.py:9: def _fix_b(_fix_a)\ntests/test_bad_scoping.py:5: def _fix_a()\n========================== short test summary info ============================\nERROR tests/test_bad_scoping.py::test__fix_b_return_val - Failed:\nScopeMismatch: You tried to access the function scoped fixture _fix_a with a\nclass scoped request object, involved factories:\n=========================== 1 error in 0.01s ==================================\n```\n\nThis error can be avoided by adjusting the fixture scopes to adhere to the\nhierarchy rule, so updating `_fix_a` to use a class scope or greater would\nresult in a passing test.\n\n## Summary\n\nHopefully by now you feel comfortable in when and how to use fixtures for\n`pytest`. We've covered quite a bit, including:\n\n* What fixtures are\n* Use-cases\n* Where to store them\n* How to reference them\n* How to scope them\n* How changes to fixtures persist or not\n* Handling scope errors \n\nIf you spot an error with this article, or have suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Clara\n* Dan C\n* Dan S\n* Edward\n* Ethan\n* Henry\n* Ian\n* Iva\n* Jay\n* Mark\n* Martin R\n* Martin W\n* Mat\n* Sergio\n\n

fin!

\n\n", "supporting": [ - "11-fiddly-bits-of-pytest_files" + "11-fiddly-bits-of-pytest_files/figure-html" ], "filters": [], "includes": { diff --git a/_freeze/blogs/12-pytest-tmp-path/execute-results/html.json b/_freeze/blogs/12-pytest-tmp-path/execute-results/html.json index 149015a..ab3fa6c 100644 --- a/_freeze/blogs/12-pytest-tmp-path/execute-results/html.json +++ b/_freeze/blogs/12-pytest-tmp-path/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "677667be1624d3fd80e210380cb20a06", + "hash": "22f94a3f5c68a80fd36bbaea5f60fcd2", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Pytest With `tmp_path` in Plain English\nauthor: Rich Leyshon\ndate: April 25 2024\ndescription: Plain English Discussion of Pytest Temporary Fixtures.\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - tmp_path\n - tmp_path_factory\n - fixtures\n - pytest-in-plain-english\nimage: 'https://images.pixexid.com/a-clock-with-gears-made-of-layered-textured-paper-and-a-glossy-metallic-face-s-0yp5gyd5.jpeg?h=699&q=70'\nimage-alt: 'A clock with gears made of layered, textured paper and a glossy metallic face, set against a backdrop of passing time by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)'\ntoc: true\n---\n\n
\n \n
Creative commons license by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses why and how we use pytest's\ntemporary fixtures `tmp_path` and `tmp_path_factory`. This blog is the second\nin a series of blogs called\n[pytest in plain English](/../index.html#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\nfollow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[temp-fixtures branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/temp-fixtures)\nof the repo.\n\n## What Are Temporary Fixtures?\n\nIn the previous [`pytest` in plain English](/blogs/11-fiddly-bits-of-pytest.qmd)\narticle, we discussed how to write our own fixtures to serve data to our tests.\nBut `pytest` comes with its own set of fixtures that are really useful in\ncertain situations. In this article, we will consider those fixtures that are\nused to create temporary directories and files.\n\n### Why Do We Need Temporary Fixtures?\n\nIf the code you need to test carries out file operations, then there are a few\nconsiderations needed when writing our tests. It is best practice in testing to\nensure the system state is unaffected by running the test suite. In the very\nworst cases I have encountered, running the test suite has resulted in\ntimestamped csvs being written to disk every time `pytest` was run. As\ndevelopers potentially run these tests hundreds of times while working on a\ncode base, this thoughtless little side-effect quickly results in a messy file\nsystem. \n\nJust to clarify - I'm not saying it's a bad idea to use timestamped file names.\nOr to have functions with these kinds of side effects - these features can be\nreally useful. The problem is when the test suite creates junk on your disk\nthat you weren't aware of...\n\nBy using temporary fixtures, we are ensuring the tests are isolated from each\nother and behave in dependable ways. If you ever encounter a test suite that\nbehaves differently on subsequent runs, then be suspicious of a messy test\nsuite with file operations that have changed the state of the system. In order\nfor us to reason about the state of the code, we need to be able to rely on the\nanswers we get from the tests, known in test engineering speak as\n**determinism**.\n\n### Let's Compare the Available Temporary Fixtures\n\nThe 2 fixtures that we should be working with as of 2024 are `tmp_path` and\n`tmp_path_factory`. Both of these newer temporary fixtures return\n`pathlib.Path` objects and are included with the `pytest` package in order to\nencourage developers to use them. No need to import `tempfile` or any other\ndependency to get what you need, it's all bundled up with your `pytest`\ninstallation.\n\n`tmp_path` is a function-scoped fixture. Meaning that if we use `tmp_path` in\n2 unit tests, then we will be served with 2 separate temporary directories to\nwork with. This should meet most developers' needs. But if you're doing\nsomething more complex with files, there are occasions where you may need a\nmore persistent temporary directory. Perhaps a bunch of your functions need to\nwork sequentially using files on disk and you need to test how all these units\nwork together. This kind of scenario can arise if you are working on really\nlarge files where in-memory operations become too costly. This is where\n`tmp_path_factory` can be useful, as it is a session-scoped temporary\nstructure. A `tmp_path_factory` structure will be created at the start of a\ntest suite and will persist until teardown happens once the last test has been\nexecuted.\n\n| Fixture Name | Scope | Teardown after each |\n| ------------------ | ---------| ------------------- |\n| `tmp_path` | function | test function |\n| `tmp_path_factory` | session | `pytest` session |\n\n### What About `tmpdir`?\n\nAh, the eagle-eyed among you may have noticed that the `pytest` package\ncontains other fixtures that are relevant to temporary structures. Namely\n`tmpdir` and `tmpdir_factory`. These fixtures are older equivalents of the\nfixtures we discussed above. The main difference is that instead of returning\n`pathlib.Path` objects, they return `py.path.local` objects. These fixtures\nwere written before `pathlib` had been adopted as the\n[standardised approach](https://peps.python.org/pep-0519/#standard-library-changes)\nto handling paths across multiple operating systems. The future of `tmpdir` and\n`tmpdir_factory` have been discussed for deprecation. These fixtures are being\nsunsetted and it is advised to port old test suites over to the new `tmp_path`\nfixture instead. The `pytest` team has\n[provided a utility](https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html#the-tmpdir-and-tmpdir-factory-fixtures)\nto help developers identify these issues in their old test suites. \n\nIn summary, don't use `tmpdir` any more and consider converting old code if you\nused it in the past... \n\n## How to Use Temporary Fixtures\n\n### Writing Source Code\n\nAs a reminder, the code for this section is located here.\n\nIn this deliberately silly example, let's say we have a poem sitting on our\ndisk in a text file. Thanks to chatGPT for the poem and MSFT Bing Copilot for\nthe image, making this a trivial consideration. Or should I really thank the\nmillions of people who wrote the content that these services trained on?\n\nSaving the text file in the chunk below to the `./tests/data/` folder is where\nyou would typically save data for your tests.\n\n\"A\n\n```{.abc filename=\"tests/data/jack-jill-2024.txt\"}\nIn the realm of data, where Jack and Jill dwell,\nThey ventured forth, their tale to tell.\nBut amidst the bytes, a glitch they found,\nA challenge profound, in algorithms bound.\n\nTheir circuits whirred, their processors spun,\nAs they analyzed the glitch, one by one.\nYet despite their prowess, misfortune struck,\nA bug so elusive, like lightning struck.\n\nTheir systems faltered, errors abound,\nAs frustration grew with each rebound.\nBut Jack and Jill, with minds so keen,\nRefused to let the glitch remain unseen.\n\nWith perseverance strong and logic clear,\nThey traced the bug to its hidden sphere.\nAnd with precision fine and code refined,\nThey patched the glitch, their brilliance defined.\n\nIn the end, though misfortune came their way,\nJack and Jill triumphed, without delay.\nFor in the realm of AI, where challenges frown,\nTheir intellect prevailed, wearing victory's crown.\n\nSo let their tale inspire, in bytes and code,\nWhere challenges rise on the digital road.\nFor Jack and Jill, with their AI might,\nShowed that even in darkness, there's always light.\n\n```\n\nLet's imagine we need a program that can edit the text and write new versions\nof the poem to disk. Let's go ahead and create a function that will read the\npoem from disk and replace any word that you'd like to change.\n\n::: {#a5749c24 .cell execution_count=1}\n``` {.python .cell-code}\n\"\"\"Demonstrating tmp_path & tmp_path_factory with a simple txt file.\"\"\"\nfrom pathlib import Path\nfrom typing import Union\n\ndef _update_a_term(\n txt_pth: Union[Path, str], target_pattern:str, replacement:str) -> str:\n \"\"\"Replace the target pattern in a body of text.\n\n Parameters\n ----------\n txt_pth : Union[Path, str]\n Path to a txt file.\n target_pattern : str\n The pattern to replace.\n replacement : str\n The replacement value.\n\n Returns\n -------\n str\n String with any occurrences of target_pattern replaced with specified\n replacement value.\n\n \"\"\"\n with open(txt_pth, \"r\") as f:\n txt = f.read()\n f.close()\n return txt.replace(target_pattern, replacement)\n```\n:::\n\n\nNow we can try using the function to rename a character in the rhyme, by\nrunning the below code in a python shell.\n\n::: {#62428a74 .cell execution_count=2}\n``` {.python .cell-code}\nfrom pyprojroot import here\nrhyme = _update_a_term(\n txt_pth=here(\"data/blogs/jack-jill-2024.txt\"),\n target_pattern=\"Jill\",\n replacement=\"Jock\")\nprint(rhyme[0:175])\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nIn the realm of data, where Jack and Jock dwell,\nThey ventured forth, their tale to tell.\nBut amidst the bytes, a glitch they found,\nA challenge profound, in algorithms bound.\n```\n:::\n:::\n\n\n::: {.callout-note collapse=\"true\"}\n\n#### Why Use Underscores?\n\nYou may have noticed that the above function starts with an underscore. This\nconvention means the function is not intended for use by the user. These\ninternal functions would typically have less defensive checks than those you\nintend to expose to your users. It's not an enforced thing but is considered\ngood practice. It means \"use at your own risk\" as internals often have less\ndocumentation, may not be directly tested and could be less stable than\nfunctions in the api.\n\n:::\n\nGreat, next we need a little utility function that will take our text and write\nit to a file of our choosing.\n\n::: {#711d1228 .cell execution_count=3}\n``` {.python .cell-code}\ndef _write_string_to_txt(some_txt:str, out_pth:Union[Path, str]) -> None:\n \"\"\"Write some string to a text file.\n\n Parameters\n ----------\n some_txt : str\n The text to write to file.\n out_pth : Union[Path, str]\n The path to the file.\n \n Returns\n -------\n None\n\n \"\"\"\n with open(out_pth, \"w\") as f:\n f.writelines(some_txt)\n f.close() \n```\n:::\n\n\nFinally, we need a wrapper function that will use the above functions, allowing\nthe user to read in the text file, replace a pattern and then write the new\npoem to file.\n\n::: {#f749a2be .cell execution_count=4}\n``` {.python .cell-code}\ndef update_poem(\n poem_pth:Union[Path, str],\n target_pattern:str,\n replacement:str,\n out_file:Union[Path, str]) -> None:\n \"\"\"Takes a txt file, replaces a pattern and writes to a new file.\n\n Parameters\n ----------\n poem_pth : Union[Path, str]\n Path to a txt file.\n target_pattern : str\n A pattern to update.\n replacement : str\n The replacement value.\n out_file : Union[Path, str]\n A file path to write to.\n\n \"\"\"\n txt = _update_a_term(poem_pth, target_pattern, replacement)\n _write_string_to_txt(txt, out_file)\n```\n:::\n\n\nHow do we know it works? We can use it and observe the output, as I did with\n`_update_a_term()` earlier, but this article is about testing. So let's get to\nit.\n\n### Testing the Source Code\n\nWe need to test `update_poem()` but it writes files to disk. We don't want to\nlitter our (and our colleagues') disks with files every time `pytest` runs.\nTherefore we need to ensure the function's `out_file` parameter is pointing at\na temporary directory. In that way, we can rely on the temporary structure's\nbehaviour on teardown to remove these files when pytest finishes doing its\nbusiness.\n\n::: {#1dc4edf6 .cell execution_count=5}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\nimport os\n\nimport pytest\n\nfrom example_pkg import update_poetry\n\ndef test_update_poem_writes_new_pattern_to_file(tmp_path):\n \"\"\"Check that update_poem changes the poem pattern and writes to file.\"\"\"\n new_poem_path = os.path.join(tmp_path, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n```\n:::\n\n\nBefore I go ahead and add a bunch of assertions in, look at how easy it is to\nuse `tmp_path`, blink and you'll miss it. You simply reference it in the\nsignature of the test where you wish to use it and then you are able to work\nwith it like you would any other path object.\n\nSo far in this test function, I specified that I'd like to read the text from a\nfile called `jack-jill-2024.txt`, replace the word \"glitch\" with \"bug\" wherever\nit occurs and then write this text to a file called `new_poem.txt` in a\ntemporary directory. \n\nSome simple tests for this little function:\n\n* Does the file I asked for exist?\n* Are the contents of that file as I expect?\n\nLet's go ahead and add in those assertions.\n\n::: {#d75066c3 .cell execution_count=6}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n\nimport os\n\nimport pytest\n\nfrom example_pkg import update_poetry\n\ndef test_update_poem_writes_new_pattern_to_file(tmp_path):\n \"\"\"Check that update_poem changes the poem pattern and writes to file.\"\"\"\n new_poem_path = os.path.join(tmp_path, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n # Now for the assertions\n assert os.path.exists(new_poem_path)\n assert os.listdir(tmp_path) == [\"new_poem.txt\"]\n # let's check what pattern was written - now we need to read in the\n # contents of the new file.\n with open(new_poem_path, \"r\") as f:\n what_was_written = f.read()\n f.close()\n assert \"glitch\" not in what_was_written\n assert \"bug\" in what_was_written\n\n```\n:::\n\n\nRunning `pytest` results in the below output.\n\n```\ncollected 1 item\n\ntests/test_update_poetry.py . [100%]\n\n============================== 1 passed in 0.01s ==============================\n```\n\nSo we prove that the function works how we hoped it would. But what if I want\nto work with the `new_poem.txt` file again in another test function? Let's add\nanother test to `test_update_poetry.py` and see what we get when we try to use\n`tmp_path` once more.\n\n::: {#aef83874 .cell execution_count=7}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\ndef test_do_i_get_a_new_tmp_path(tmp_path):\n \"\"\"Remind ourselves that tmp_path is function-scoped.\"\"\"\n assert \"new_poem\" not in os.listdir(tmp_path)\n assert os.listdir(tmp_path) == []\n\n```\n:::\n\n\nAs is demonstrated when running `pytest` once more, `tmp_path` is\nfunction-scoped and we have now lost the new poem with the bugs instead of the\nglitches. Drat! What to do...\n\n```\ncollected 2 items\n\ntests/test_update_poetry.py .. [100%]\n\n============================== 2 passed in 0.01s ==============================\n\n```\n\nAs mentioned earlier, `pytest` provides another fixture with more\nflexibility, called `tmp_path_factory`. As this fixture is session-scoped, we\ncan have full control over this fixture's scoping. \n\n::: {.callout-tip collapse=\"true\"}\n\n#### Fixture Scopes\n\nFor a refresher on the rules of scope referencing, please see the blog [Pytest Fixtures in Plain English](/blogs/11-fiddly-bits-of-pytest.qmd#scopemismatch-error).\n\n:::\n\n::: {#488cc6b5 .cell execution_count=8}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\n# def test_do_i_get_a_new_tmp_path(tmp_path): ...\n\n@pytest.fixture(scope=\"module\")\ndef _module_scoped_tmp(tmp_path_factory):\n yield tmp_path_factory.mktemp(\"put_poetry_here\", numbered=False)\n```\n:::\n\n\nNote that as `tmp_path_factory` is session-scoped, I'm free to reference it in\nanother fixture with any scope. Here I define a module-scoped fixture, which\nmeans teardown of `_module_scoped_tmp` will occur once the final test in this\ntest module completes. Now repeating the logic executed with `tmp_path` above,\nbut this time with our new module-scoped temporary directory, we get a\ndifferent outcome.\n\n::: {#64bbdd5d .cell execution_count=9}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\n# def test_do_i_get_a_new_tmp_path(tmp_path): ...\n\n@pytest.fixture(scope=\"module\")\ndef _module_scoped_tmp(tmp_path_factory):\n yield tmp_path_factory.mktemp(\"put_poetry_here\", numbered=False)\n\n\ndef test_module_scoped_tmp_exists(_module_scoped_tmp):\n new_poem_path = os.path.join(_module_scoped_tmp, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n assert os.path.exists(new_poem_path)\n with open(new_poem_path, \"r\") as f:\n what_was_written = f.read()\n f.close()\n assert \"glitch\" not in what_was_written\n assert \"bug\" in what_was_written\n assert os.listdir(_module_scoped_tmp) == [\"new_poem.txt\"]\n\n\ndef test_do_i_get_a_new_tmp_path_factory(_module_scoped_tmp):\n assert not os.listdir(_module_scoped_tmp) == [] # not empty...\n assert os.listdir(_module_scoped_tmp) == [\"new_poem.txt\"]\n # module-scoped fixture still contains file made in previous test function\n with open(os.path.join(_module_scoped_tmp, \"new_poem.txt\")) as f:\n found_txt = f.read()\n f.close()\n assert \"glitch\" not in found_txt\n assert \"bug\" in found_txt\n```\n:::\n\n\nExecuting `pytest` one final time demonstrates that the same output file\nwritten to disk with `test_module_scoped_tmp_exists()` is subsequently\navailable for further testing in `test_do_i_get_a_new_tmp_path_factory()`.\n\n```\ncollected 4 items\n\ntests/test_update_poetry.py .... [100%]\n\n============================== 4 passed in 0.01s ==============================\n```\n\nNote that the order that these 2 tests run in is now important. These tests are\nno longer isolated and trying to run the second test on its own with\n`pytest -k \"test_do_i_get_a_new_tmp_path_factory\"` would result in a failure.\nFor this reason, it may be advisable to pop the test functions within a common\ntest class, or even use\npytest marks\nto mark them as integration tests (more on this in a future blog). \n\n\n## Summary\n\nThe reasons we use temporary fixtures and how to use them has been demonstrated\nwith another silly (but hopefully relatable) little example. I have not gone\ninto the wealth of methods available in these temporary fixtures, but they have\nmany useful utilities. Maybe you're working with a complex nested directory\nstructure for example, the `glob` method would surely help with that.\n\nBelow are the public methods and attributes of `tmp_path`:\n\n```\n['absolute', 'anchor', 'as_posix', 'as_uri', 'chmod', 'cwd', 'drive', 'exists',\n'expanduser', 'glob', 'group', 'hardlink_to', 'home', 'is_absolute',\n'is_block_device', 'is_char_device', 'is_dir', 'is_fifo', 'is_file',\n'is_junction', 'is_mount', 'is_relative_to', 'is_reserved', 'is_socket',\n'is_symlink', 'iterdir', 'joinpath', 'lchmod', 'lstat', 'match', 'mkdir',\n'name', 'open', 'owner', 'parent', 'parents', 'parts', 'read_bytes',\n'read_text', 'readlink', 'relative_to', 'rename', 'replace', 'resolve',\n'rglob', 'rmdir', 'root', 'samefile', 'stat', 'stem', 'suffix', 'suffixes',\n'symlink_to', 'touch', 'unlink', 'walk', 'with_name', 'with_segments',\n'with_stem', 'with_suffix', 'write_bytes', 'write_text'] \n```\n\nIt is useful to\n[read the `pathlib.Path` docs](https://docs.python.org/3/library/pathlib.html#pathlib.Path)\nas both fixtures return this type and many of the methods above are inherited\nfrom these types. To read the `tmp_path` and `tmp_path_factory` implementation,\nI recommend reading the\n[tmp docstrings](https://github.com/pytest-dev/pytest/blob/main/src/_pytest/tmpdir.py)\non GitHub.\n\nIf you spot an error with this article, or have suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Charlie\n* Dan\n* Edward\n* Ian\n* Mark\n\n

fin!

\n\n", + "markdown": "---\ntitle: Pytest With `tmp_path` in Plain English\nauthor: Rich Leyshon\ndate: April 25 2024\ndescription: Plain English Discussion of Pytest Temporary Fixtures.\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - tmp_path\n - tmp_path_factory\n - fixtures\n - pytest-in-plain-english\nimage: 'https://images.pixexid.com/a-clock-with-gears-made-of-layered-textured-paper-and-a-glossy-metallic-face-s-0yp5gyd5.jpeg?h=699&q=70'\nimage-alt: 'A clock with gears made of layered, textured paper and a glossy metallic face, set against a backdrop of passing time by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)'\ntoc: true\n---\n\n
\n \n
Creative commons license by [Ralph](https://pixexid.com/profile/cjxrsxsl7000008s6h21jecoe)
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses why and how we use pytest's\ntemporary fixtures `tmp_path` and `tmp_path_factory`. This blog is the second\nin a series of blogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\nfollow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[temp-fixtures branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/temp-fixtures)\nof the repo.\n\n## What Are Temporary Fixtures?\n\nIn the previous [`pytest` in plain English](/blogs/11-fiddly-bits-of-pytest.qmd)\narticle, we discussed how to write our own fixtures to serve data to our tests.\nBut `pytest` comes with its own set of fixtures that are really useful in\ncertain situations. In this article, we will consider those fixtures that are\nused to create temporary directories and files.\n\n### Why Do We Need Temporary Fixtures?\n\nIf the code you need to test carries out file operations, then there are a few\nconsiderations needed when writing our tests. It is best practice in testing to\nensure the system state is unaffected by running the test suite. In the very\nworst cases I have encountered, running the test suite has resulted in\ntimestamped csvs being written to disk every time `pytest` was run. As\ndevelopers potentially run these tests hundreds of times while working on a\ncode base, this thoughtless little side-effect quickly results in a messy file\nsystem. \n\nJust to clarify - I'm not saying it's a bad idea to use timestamped file names.\nOr to have functions with these kinds of side effects - these features can be\nreally useful. The problem is when the test suite creates junk on your disk\nthat you weren't aware of...\n\nBy using temporary fixtures, we are ensuring the tests are isolated from each\nother and behave in dependable ways. If you ever encounter a test suite that\nbehaves differently on subsequent runs, then be suspicious of a messy test\nsuite with file operations that have changed the state of the system. In order\nfor us to reason about the state of the code, we need to be able to rely on the\nanswers we get from the tests, known in test engineering speak as\n**determinism**.\n\n### Let's Compare the Available Temporary Fixtures\n\nThe 2 fixtures that we should be working with as of 2024 are `tmp_path` and\n`tmp_path_factory`. Both of these newer temporary fixtures return\n`pathlib.Path` objects and are included with the `pytest` package in order to\nencourage developers to use them. No need to import `tempfile` or any other\ndependency to get what you need, it's all bundled up with your `pytest`\ninstallation.\n\n`tmp_path` is a function-scoped fixture. Meaning that if we use `tmp_path` in\n2 unit tests, then we will be served with 2 separate temporary directories to\nwork with. This should meet most developers' needs. But if you're doing\nsomething more complex with files, there are occasions where you may need a\nmore persistent temporary directory. Perhaps a bunch of your functions need to\nwork sequentially using files on disk and you need to test how all these units\nwork together. This kind of scenario can arise if you are working on really\nlarge files where in-memory operations become too costly. This is where\n`tmp_path_factory` can be useful, as it is a session-scoped temporary\nstructure. A `tmp_path_factory` structure will be created at the start of a\ntest suite and will persist until teardown happens once the last test has been\nexecuted.\n\n| Fixture Name | Scope | Teardown after each |\n| ------------------ | ---------| ------------------- |\n| `tmp_path` | function | test function |\n| `tmp_path_factory` | session | `pytest` session |\n\n### What About `tmpdir`?\n\nAh, the eagle-eyed among you may have noticed that the `pytest` package\ncontains other fixtures that are relevant to temporary structures. Namely\n`tmpdir` and `tmpdir_factory`. These fixtures are older equivalents of the\nfixtures we discussed above. The main difference is that instead of returning\n`pathlib.Path` objects, they return `py.path.local` objects. These fixtures\nwere written before `pathlib` had been adopted as the\n[standardised approach](https://peps.python.org/pep-0519/#standard-library-changes)\nto handling paths across multiple operating systems. The future of `tmpdir` and\n`tmpdir_factory` have been discussed for deprecation. These fixtures are being\nsunsetted and it is advised to port old test suites over to the new `tmp_path`\nfixture instead. The `pytest` team has\n[provided a utility](https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html#the-tmpdir-and-tmpdir-factory-fixtures)\nto help developers identify these issues in their old test suites. \n\nIn summary, don't use `tmpdir` any more and consider converting old code if you\nused it in the past... \n\n## How to Use Temporary Fixtures\n\n### Writing Source Code\n\nAs a reminder, the code for this section is located here.\n\nIn this deliberately silly example, let's say we have a poem sitting on our\ndisk in a text file. Thanks to chatGPT for the poem and MSFT Bing Copilot for\nthe image, making this a trivial consideration. Or should I really thank the\nmillions of people who wrote the content that these services trained on?\n\nSaving the text file in the chunk below to the `./tests/data/` folder is where\nyou would typically save data for your tests.\n\n\"A\n\n```{.abc filename=\"tests/data/jack-jill-2024.txt\"}\nIn the realm of data, where Jack and Jill dwell,\nThey ventured forth, their tale to tell.\nBut amidst the bytes, a glitch they found,\nA challenge profound, in algorithms bound.\n\nTheir circuits whirred, their processors spun,\nAs they analyzed the glitch, one by one.\nYet despite their prowess, misfortune struck,\nA bug so elusive, like lightning struck.\n\nTheir systems faltered, errors abound,\nAs frustration grew with each rebound.\nBut Jack and Jill, with minds so keen,\nRefused to let the glitch remain unseen.\n\nWith perseverance strong and logic clear,\nThey traced the bug to its hidden sphere.\nAnd with precision fine and code refined,\nThey patched the glitch, their brilliance defined.\n\nIn the end, though misfortune came their way,\nJack and Jill triumphed, without delay.\nFor in the realm of AI, where challenges frown,\nTheir intellect prevailed, wearing victory's crown.\n\nSo let their tale inspire, in bytes and code,\nWhere challenges rise on the digital road.\nFor Jack and Jill, with their AI might,\nShowed that even in darkness, there's always light.\n\n```\n\nLet's imagine we need a program that can edit the text and write new versions\nof the poem to disk. Let's go ahead and create a function that will read the\npoem from disk and replace any word that you'd like to change.\n\n::: {#9b36e765 .cell execution_count=1}\n``` {.python .cell-code}\n\"\"\"Demonstrating tmp_path & tmp_path_factory with a simple txt file.\"\"\"\nfrom pathlib import Path\nfrom typing import Union\n\ndef _update_a_term(\n txt_pth: Union[Path, str], target_pattern:str, replacement:str) -> str:\n \"\"\"Replace the target pattern in a body of text.\n\n Parameters\n ----------\n txt_pth : Union[Path, str]\n Path to a txt file.\n target_pattern : str\n The pattern to replace.\n replacement : str\n The replacement value.\n\n Returns\n -------\n str\n String with any occurrences of target_pattern replaced with specified\n replacement value.\n\n \"\"\"\n with open(txt_pth, \"r\") as f:\n txt = f.read()\n f.close()\n return txt.replace(target_pattern, replacement)\n```\n:::\n\n\nNow we can try using the function to rename a character in the rhyme, by\nrunning the below code in a python shell.\n\n::: {#bbf18052 .cell execution_count=2}\n``` {.python .cell-code}\nfrom pyprojroot import here\nrhyme = _update_a_term(\n txt_pth=here(\"data/blogs/jack-jill-2024.txt\"),\n target_pattern=\"Jill\",\n replacement=\"Jock\")\nprint(rhyme[0:175])\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nIn the realm of data, where Jack and Jock dwell,\nThey ventured forth, their tale to tell.\nBut amidst the bytes, a glitch they found,\nA challenge profound, in algorithms bound.\n```\n:::\n:::\n\n\n::: {.callout-note collapse=\"true\"}\n\n#### Why Use Underscores?\n\nYou may have noticed that the above function starts with an underscore. This\nconvention means the function is not intended for use by the user. These\ninternal functions would typically have less defensive checks than those you\nintend to expose to your users. It's not an enforced thing but is considered\ngood practice. It means \"use at your own risk\" as internals often have less\ndocumentation, may not be directly tested and could be less stable than\nfunctions in the api.\n\n:::\n\nGreat, next we need a little utility function that will take our text and write\nit to a file of our choosing.\n\n::: {#d4c0a362 .cell execution_count=3}\n``` {.python .cell-code}\ndef _write_string_to_txt(some_txt:str, out_pth:Union[Path, str]) -> None:\n \"\"\"Write some string to a text file.\n\n Parameters\n ----------\n some_txt : str\n The text to write to file.\n out_pth : Union[Path, str]\n The path to the file.\n \n Returns\n -------\n None\n\n \"\"\"\n with open(out_pth, \"w\") as f:\n f.writelines(some_txt)\n f.close() \n```\n:::\n\n\nFinally, we need a wrapper function that will use the above functions, allowing\nthe user to read in the text file, replace a pattern and then write the new\npoem to file.\n\n::: {#4b0c178b .cell execution_count=4}\n``` {.python .cell-code}\ndef update_poem(\n poem_pth:Union[Path, str],\n target_pattern:str,\n replacement:str,\n out_file:Union[Path, str]) -> None:\n \"\"\"Takes a txt file, replaces a pattern and writes to a new file.\n\n Parameters\n ----------\n poem_pth : Union[Path, str]\n Path to a txt file.\n target_pattern : str\n A pattern to update.\n replacement : str\n The replacement value.\n out_file : Union[Path, str]\n A file path to write to.\n\n \"\"\"\n txt = _update_a_term(poem_pth, target_pattern, replacement)\n _write_string_to_txt(txt, out_file)\n```\n:::\n\n\nHow do we know it works? We can use it and observe the output, as I did with\n`_update_a_term()` earlier, but this article is about testing. So let's get to\nit.\n\n### Testing the Source Code\n\nWe need to test `update_poem()` but it writes files to disk. We don't want to\nlitter our (and our colleagues') disks with files every time `pytest` runs.\nTherefore we need to ensure the function's `out_file` parameter is pointing at\na temporary directory. In that way, we can rely on the temporary structure's\nbehaviour on teardown to remove these files when pytest finishes doing its\nbusiness.\n\n::: {#f274e9da .cell execution_count=5}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\nimport os\n\nimport pytest\n\nfrom example_pkg import update_poetry\n\ndef test_update_poem_writes_new_pattern_to_file(tmp_path):\n \"\"\"Check that update_poem changes the poem pattern and writes to file.\"\"\"\n new_poem_path = os.path.join(tmp_path, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n```\n:::\n\n\nBefore I go ahead and add a bunch of assertions in, look at how easy it is to\nuse `tmp_path`, blink and you'll miss it. You simply reference it in the\nsignature of the test where you wish to use it and then you are able to work\nwith it like you would any other path object.\n\nSo far in this test function, I specified that I'd like to read the text from a\nfile called `jack-jill-2024.txt`, replace the word \"glitch\" with \"bug\" wherever\nit occurs and then write this text to a file called `new_poem.txt` in a\ntemporary directory. \n\nSome simple tests for this little function:\n\n* Does the file I asked for exist?\n* Are the contents of that file as I expect?\n\nLet's go ahead and add in those assertions.\n\n::: {#a7c83031 .cell execution_count=6}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n\nimport os\n\nimport pytest\n\nfrom example_pkg import update_poetry\n\ndef test_update_poem_writes_new_pattern_to_file(tmp_path):\n \"\"\"Check that update_poem changes the poem pattern and writes to file.\"\"\"\n new_poem_path = os.path.join(tmp_path, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n # Now for the assertions\n assert os.path.exists(new_poem_path)\n assert os.listdir(tmp_path) == [\"new_poem.txt\"]\n # let's check what pattern was written - now we need to read in the\n # contents of the new file.\n with open(new_poem_path, \"r\") as f:\n what_was_written = f.read()\n f.close()\n assert \"glitch\" not in what_was_written\n assert \"bug\" in what_was_written\n\n```\n:::\n\n\nRunning `pytest` results in the below output.\n\n```\ncollected 1 item\n\ntests/test_update_poetry.py . [100%]\n\n============================== 1 passed in 0.01s ==============================\n```\n\nSo we prove that the function works how we hoped it would. But what if I want\nto work with the `new_poem.txt` file again in another test function? Let's add\nanother test to `test_update_poetry.py` and see what we get when we try to use\n`tmp_path` once more.\n\n::: {#69bcfe8a .cell execution_count=7}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\ndef test_do_i_get_a_new_tmp_path(tmp_path):\n \"\"\"Remind ourselves that tmp_path is function-scoped.\"\"\"\n assert \"new_poem\" not in os.listdir(tmp_path)\n assert os.listdir(tmp_path) == []\n\n```\n:::\n\n\nAs is demonstrated when running `pytest` once more, `tmp_path` is\nfunction-scoped and we have now lost the new poem with the bugs instead of the\nglitches. Drat! What to do...\n\n```\ncollected 2 items\n\ntests/test_update_poetry.py .. [100%]\n\n============================== 2 passed in 0.01s ==============================\n\n```\n\nAs mentioned earlier, `pytest` provides another fixture with more\nflexibility, called `tmp_path_factory`. As this fixture is session-scoped, we\ncan have full control over this fixture's scoping. \n\n::: {.callout-tip collapse=\"true\"}\n\n#### Fixture Scopes\n\nFor a refresher on the rules of scope referencing, please see the blog [Pytest Fixtures in Plain English](/blogs/11-fiddly-bits-of-pytest.qmd#scopemismatch-error).\n\n:::\n\n::: {#cca79ae0 .cell execution_count=8}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\n# def test_do_i_get_a_new_tmp_path(tmp_path): ...\n\n@pytest.fixture(scope=\"module\")\ndef _module_scoped_tmp(tmp_path_factory):\n yield tmp_path_factory.mktemp(\"put_poetry_here\", numbered=False)\n```\n:::\n\n\nNote that as `tmp_path_factory` is session-scoped, I'm free to reference it in\nanother fixture with any scope. Here I define a module-scoped fixture, which\nmeans teardown of `_module_scoped_tmp` will occur once the final test in this\ntest module completes. Now repeating the logic executed with `tmp_path` above,\nbut this time with our new module-scoped temporary directory, we get a\ndifferent outcome.\n\n::: {#18015746 .cell execution_count=9}\n``` {.python .cell-code}\n\"\"\"Tests for update_poetry module.\"\"\"\n# import statements ...\n\n# def test_update_poem_writes_new_pattern_to_file(tmp_path): ...\n\n# def test_do_i_get_a_new_tmp_path(tmp_path): ...\n\n@pytest.fixture(scope=\"module\")\ndef _module_scoped_tmp(tmp_path_factory):\n yield tmp_path_factory.mktemp(\"put_poetry_here\", numbered=False)\n\n\ndef test_module_scoped_tmp_exists(_module_scoped_tmp):\n new_poem_path = os.path.join(_module_scoped_tmp, \"new_poem.txt\")\n update_poetry.update_poem(\n poem_pth=\"tests/data/jack-jill-2024.txt\",\n target_pattern=\"glitch\",\n replacement=\"bug\",\n out_file=new_poem_path\n )\n assert os.path.exists(new_poem_path)\n with open(new_poem_path, \"r\") as f:\n what_was_written = f.read()\n f.close()\n assert \"glitch\" not in what_was_written\n assert \"bug\" in what_was_written\n assert os.listdir(_module_scoped_tmp) == [\"new_poem.txt\"]\n\n\ndef test_do_i_get_a_new_tmp_path_factory(_module_scoped_tmp):\n assert not os.listdir(_module_scoped_tmp) == [] # not empty...\n assert os.listdir(_module_scoped_tmp) == [\"new_poem.txt\"]\n # module-scoped fixture still contains file made in previous test function\n with open(os.path.join(_module_scoped_tmp, \"new_poem.txt\")) as f:\n found_txt = f.read()\n f.close()\n assert \"glitch\" not in found_txt\n assert \"bug\" in found_txt\n```\n:::\n\n\nExecuting `pytest` one final time demonstrates that the same output file\nwritten to disk with `test_module_scoped_tmp_exists()` is subsequently\navailable for further testing in `test_do_i_get_a_new_tmp_path_factory()`.\n\n```\ncollected 4 items\n\ntests/test_update_poetry.py .... [100%]\n\n============================== 4 passed in 0.01s ==============================\n```\n\nNote that the order that these 2 tests run in is now important. These tests are\nno longer isolated and trying to run the second test on its own with\n`pytest -k \"test_do_i_get_a_new_tmp_path_factory\"` would result in a failure.\nFor this reason, it may be advisable to pop the test functions within a common\ntest class, or even use\npytest marks\nto mark them as integration tests (more on this in a future blog). \n\n\n## Summary\n\nThe reasons we use temporary fixtures and how to use them has been demonstrated\nwith another silly (but hopefully relatable) little example. I have not gone\ninto the wealth of methods available in these temporary fixtures, but they have\nmany useful utilities. Maybe you're working with a complex nested directory\nstructure for example, the `glob` method would surely help with that.\n\nBelow are the public methods and attributes of `tmp_path`:\n\n```\n['absolute', 'anchor', 'as_posix', 'as_uri', 'chmod', 'cwd', 'drive', 'exists',\n'expanduser', 'glob', 'group', 'hardlink_to', 'home', 'is_absolute',\n'is_block_device', 'is_char_device', 'is_dir', 'is_fifo', 'is_file',\n'is_junction', 'is_mount', 'is_relative_to', 'is_reserved', 'is_socket',\n'is_symlink', 'iterdir', 'joinpath', 'lchmod', 'lstat', 'match', 'mkdir',\n'name', 'open', 'owner', 'parent', 'parents', 'parts', 'read_bytes',\n'read_text', 'readlink', 'relative_to', 'rename', 'replace', 'resolve',\n'rglob', 'rmdir', 'root', 'samefile', 'stat', 'stem', 'suffix', 'suffixes',\n'symlink_to', 'touch', 'unlink', 'walk', 'with_name', 'with_segments',\n'with_stem', 'with_suffix', 'write_bytes', 'write_text'] \n```\n\nIt is useful to\n[read the `pathlib.Path` docs](https://docs.python.org/3/library/pathlib.html#pathlib.Path)\nas both fixtures return this type and many of the methods above are inherited\nfrom these types. To read the `tmp_path` and `tmp_path_factory` implementation,\nI recommend reading the\n[tmp docstrings](https://github.com/pytest-dev/pytest/blob/main/src/_pytest/tmpdir.py)\non GitHub.\n\nIf you spot an error with this article, or have suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Charlie\n* Dan\n* Edward\n* Ian\n* Mark\n\n

fin!

\n\n", "supporting": [ "12-pytest-tmp-path_files/figure-html" ], diff --git a/_freeze/blogs/13-pytest-parametrize/execute-results/html.json b/_freeze/blogs/13-pytest-parametrize/execute-results/html.json index 8b6225f..a8525e6 100644 --- a/_freeze/blogs/13-pytest-parametrize/execute-results/html.json +++ b/_freeze/blogs/13-pytest-parametrize/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "ec27f13de99b4fbdab6225fa3bd7139d", + "hash": "503602e0a0415e0829e3b540ce8b6ad2", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Parametrized Tests With Pytest in Plain English\nauthor: Rich Leyshon\ndate: June 07 2024\ndescription: Plain English Discussion of Pytest Parametrize\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - parametrize\n - pytest-in-plain-english\nimage: 'https://i.imgur.com/n1flYqU.jpeg'\nimage-alt: Complex sushi conveyor belt with a futuristic theme in a pastel palette.\ntoc: true\ncss: /www/13-pytest-parametrize/styles.css\n---\n\n
\n \n
Complex sushi conveyor belt with a futuristic theme in a pastel palette.
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses what parametrized tests mean\nand how to implement them with `pytest`. This blog is the third in a series of\nblogs called\n[pytest in plain English](/../index.html#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\nfollow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[parametrize branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/parametrize)\nof the repo.\n\n## Overview\n\n### What Are Parametrized Tests?\n\nParametrized tests are simply tests that are applied recursively to multiple\ninput values. For example, rather than testing a function on one input value,\na list of different values could be passed as a parametrized fixture.\n\nA standard approach to testing could look like Figure 1 below, where separate\ntests are defined for the different values we need to check. This would likely\nresult in a fair amount of repeated boilerplate code.\n\n![Figure 1: Testing multiple values without parametrization](https://i.imgur.com/obEM4Oo.png)\n\nInstead, we can reduce the number of tests down to 1 and pass a list of tuples\nto the test instead. Each tuple should contain a parameter value and the\nexpected result, as illustrated in Figure 2.\n\n![Figure 2: Parametrized testing of multiple values](https://i.imgur.com/12QNQxt.png)\n\nSo let's imagine we have a simple function called `double()`, the setup for the\nparametrized list is illustrated in Figure 3.\n\n![Figure 3: Exemplified paramatrization for `test_double()`](https://i.imgur.com/9jqdR9O.png)\n\n### Why use Parametrization?\n\nThis approach allows us to thoroughly check the behaviour of our functions\nagainst multiple values, ensuring that edge-cases are safely treated or\nexceptions are raised as expected. \n\nIn this way, we serve multiple parameters and expected outcomes to a single\ntest, reducing boilerplate code. Parametrization is not a silver bullet, and we\nstill need to define all of our parameters and results in a parametrized\nfixture. This approach is not quite as flexible as the property-based testing\nachievable with a package such as\n[`hypothesis`](https://hypothesis.readthedocs.io/en/latest/). However, the\nlearning curve for `hypothesis` is a bit greater and may be disproportionate to\nthe job at hand.\n\nFor the reasons outlined above, there are likely many competent python\ndevelopers that never use parametrized fixtures. But parametrization does allow\nus to avoid implementing tests with a `for` loop or vectorized approaches to\nthe same outcomes. When coupled with programmatic approaches to generating our\ninput parameters, many lines of code can be saved. And things get even more\ninteresting when we pass multiple parametrized fixtures to our tests, which\nI'll come to in a bit. For these reasons, I believe that awareness of\nparametrization should be promoted among python developers as a useful solution\nin the software development toolkit.\n\n## Implementing Parametrization\n\nIn this section, we will compare some very simple examples of tests with and\nwithout parametrization. Feel free to clone the repository and check out to the\n[example code](https://github.com/r-leyshon/pytest-fiddly-examples/tree/parametrize)\nbranch to run the examples.\n\n### Define the Source Code\n\nHere we define a very basic function that checks whether an integer is prime.\nIf a prime is encountered, then True is returned. If not, then False. The value\n1 gets its own treatment (return `False`). Lastly, we include some basic\ndefensive checks, we return a `TypeError` if anything other than integer is\npassed to the function and a `ValueError` if the integer is less than or equal\nto 0.\n\n::: {#c063e63a .cell execution_count=1}\n``` {.python .cell-code}\ndef is_num_prime(pos_int: int) -> bool:\n \"\"\"Check if a positive integer is a prime number.\n\n Parameters\n ----------\n pos_int : int\n A positive integer.\n\n Returns\n -------\n bool\n True if the number is a prime number.\n\n Raises\n ------\n TypeError\n Value passed to `pos_int` is not an integer.\n ValueError\n Value passed to `pos_int` is less than or equal to 0.\n \"\"\"\n if not isinstance(pos_int, int):\n raise TypeError(\"`pos_int` must be a positive integer.\")\n if pos_int <= 0:\n raise ValueError(\"`pos_int` must be a positive integer.\")\n elif pos_int == 1:\n return False\n else:\n for i in range(2, (pos_int // 2) + 1):\n # If divisible by any number 2<>(n/2)+1, it is not prime\n if (pos_int % i) == 0:\n return False\n else:\n return True\n```\n:::\n\n\nRunning this function with a range of values demonstrates its behaviour.\n\n::: {#f516e3c8 .cell execution_count=2}\n``` {.python .cell-code}\nfor i in range(1, 11):\n print(f\"{i}: {is_num_prime(i)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n1: False\n2: True\n3: True\n4: False\n5: True\n6: False\n7: True\n8: False\n9: False\n10: False\n```\n:::\n:::\n\n\n### Let's Get Testing\n\nLet's begin with the defensive tests. Let's say I need to check that the\nfunction can be relied upon to raise on a number of conditions. The typical\napproach may be to test the raise conditions within a dedicated test function.\n\n::: {#1e22ae24 .cell execution_count=3}\n``` {.python .cell-code}\n\"\"\"Tests for primes module.\"\"\"\nimport pytest\n\nfrom example_pkg.primes import is_num_prime\n\n\ndef test_is_num_primes_exceptions_manually():\n \"\"\"Testing the function's defensive checks.\n\n Here we have to repeat a fair bit of pytest boilerplate.\n \"\"\"\n with pytest.raises(TypeError, match=\"must be a positive integer.\"):\n is_num_prime(1.0)\n with pytest.raises(ValueError, match=\"must be a positive integer.\"):\n is_num_prime(-1)\n```\n:::\n\n\nWithin this function, I can run multiple assertions against several hard-coded\ninputs. I'm only checking against a couple of values here but production-ready\ncode may test against many more cases. To do that, I'd need to have a lot of\nrepeated `pytest.raises` statements. Perhaps more importantly, watch what\nhappens when I run the test.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_manually\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected \n\ntests/test_primes.py . [100%]\n\n======================= 1 passed, 55 deselected in 0.01s ======================\n\n```\n\nNotice that both assertions will either pass or fail together as one test. This\ncould potentially make it more challenging to troubleshoot a failing pipeline.\nIt could be better to have separate test functions for each value, but that\nseems like an awful lot of work...\n\n### ...Enter Parametrize\n\nNow to start using parametrize, we need to use the `@pytest.mark.parametrize`\ndecorator, which takes 2 arguments, a string and an iterable.\n\n::: {#f1b06324 .cell execution_count=4}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_values, exception_types\", [(1.0, TypeError), (-1, ValueError)]\n )\n```\n:::\n\n\nThe string should contain comma separated values for the names that you would\nlike to refer to when iterating through the iterable. They can be any\nplaceholder you would wish to use in your test. These names will map to the\nindex of elements in the iterable.\n\nSo when I use the fixture with a test, I will expect to inject the following\nvalues:\n\niteration 1... \"some_values\" = 1.0, \"exception_types\" = TypeError \niteration 2... \"some_values\" = -1, \"exception_types\" = ValueError\n\nLet's go ahead and use this parametrized fixture with a test.\n\n::: {#0249cde1 .cell execution_count=5}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_values, exception_types\", [(1.0, TypeError), (-1, ValueError)]\n )\ndef test_is_num_primes_exceptions_parametrized(some_values, exception_types):\n \"\"\"The same defensive checks but this time with parametrized input.\n\n Less lines in the test but if we increase the number of cases, we need to\n add more lines to the parametrized fixture instead.\n \"\"\"\n with pytest.raises(exception_types, match=\"must be a positive integer.\"):\n is_num_prime(some_values)\n```\n:::\n\n\nThe outcome for running this test is shown below.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_parametrized\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 54 deselected / 2 selected \n\ntests/test_primes.py .. [100%]\n\n======================= 2 passed, 54 deselected in 0.01s ======================\n\n```\n\nIt's a subtle difference, but notice that we now get 2 passing tests rather\nthan 1? We can make this more explicit by passing the `-v` flag (for verbose)\nwhen we invoke `pytest`.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_parametrized\" -v \n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 54 deselected / 2 selected \n\ntest_is_num_primes_exceptions_parametrized[1.0-TypeError] PASSED [ 50%]\ntest_is_num_primes_exceptions_parametrized[-1-ValueError] PASSED [100%]\n\n======================= 2 passed, 54 deselected in 0.01s ======================\n\n```\n\nIn this way, we get a helpful printout of the test and parameter combination\nbeing executed. This can be very helpful in identifying problem cases.\n\n### Yet More Cases\n\nNext up, we may wish to check return values for our function with several\nmore cases. To keep things simple, let's write a test that checks the return\nvalues for a range of numbers between 1 and 5.\n\n::: {#d38839d9 .cell execution_count=6}\n``` {.python .cell-code}\ndef test_is_num_primes_manually():\n \"\"\"Test several positive integers return expected boolean.\n\n This is quite a few lines of code. Note that this runs as a single test.\n \"\"\"\n assert is_num_prime(1) == False\n assert is_num_prime(2) == True\n assert is_num_prime(3) == True\n assert is_num_prime(4) == False\n assert is_num_prime(5) == True\n```\n:::\n\n\nOne way that this can be serialised is by using a list of parameters and\nexpected results.\n\n::: {#7883fa6b .cell execution_count=7}\n``` {.python .cell-code}\ndef test_is_num_primes_with_list():\n \"\"\"Test the same values using lists.\n\n Less lines but is run as a single test.\n \"\"\"\n answers = [is_num_prime(i) for i in range(1, 6)]\n assert answers == [False, True, True, False, True]\n```\n:::\n\n\nThis is certainly neater than the previous example. Although both\nimplementations will evaluate as a single test, so a failing instance will not\nbe explicitly indicated in the `pytest` report.\n\n```\n% pytest -k \"test_is_num_primes_with_list\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected \n\ntests/test_primes.py . [100%]\n\n======================= 1 passed, 55 deselected in 0.01s ======================\n```\n\nTo parametrize the equivalent test, we can take the below approach.\n\n::: {#6e5548b6 .cell execution_count=8}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_integers, answers\",\n [(1, False), (2, True), (3, True), (4, False), (5, True)]\n )\ndef test_is_num_primes_parametrized(some_integers, answers):\n \"\"\"The same tests but this time with parametrized input.\n\n Fewer lines and 5 separate tests are run by pytest.\n \"\"\"\n assert is_num_prime(some_integers) == answers\n```\n:::\n\n\nThis is slightly more lines than `test_is_num_primes_with_list` but has the\nadvantage of being run as separate tests:\n\n```\n% pytest -k \"test_is_num_primes_parametrized\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 51 deselected / 5 selected \n\ntests/test_primes.py::test_is_num_primes_parametrized[1-False] PASSED [ 20%]\ntests/test_primes.py::test_is_num_primes_parametrized[2-True] PASSED [ 40%]\ntests/test_primes.py::test_is_num_primes_parametrized[3-True] PASSED [ 60%]\ntests/test_primes.py::test_is_num_primes_parametrized[4-False] PASSED [ 80%]\ntests/test_primes.py::test_is_num_primes_parametrized[5-True] PASSED [100%]\n\n======================= 5 passed, 51 deselected in 0.01s ======================\n\n```\n\nWhere this approach really comes into its own is when the number of cases you\nneed to test increases, you can explore ways of generating cases rather than\nhard-coding the values, as in the previous examples.\n\nIn the example below, we can use the `range()` function to generate the \nintegers we need to test, and then zipping these cases to their expected return\nvalues.\n\n::: {#693ee676 .cell execution_count=9}\n``` {.python .cell-code}\n# if my list of cases is growing, I can employ other tactics...\nin_ = range(1, 21)\nout = [\n False, True, True, False, True, False, True, False, False, False,\n True, False, True, False, False, False, True, False, True, False,\n ]\n\n\n@pytest.mark.parametrize(\"some_integers, some_answers\", zip(in_, out))\ndef test_is_num_primes_with_zipped_lists(some_integers, some_answers):\n \"\"\"The same tests but this time with zipped inputs.\"\"\"\n assert is_num_prime(some_integers) == some_answers\n```\n:::\n\n\nRunning this test yields the following result:\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_is_num_primes_with_zipped_lists\" -v \n============================= test session starts =============================\nplatform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0\ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\nplugins: anyio-4.0.0\ncollected 56 items / 36 deselected / 20 selected\n\n/test_primes.py::test_is_num_primes_with_zipped_lists[1-False] PASSED [ 5%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[2-True] PASSED [ 10%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[3-True] PASSED [ 15%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[4-False] PASSED [ 20%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[5-True] PASSED [ 25%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[6-False] PASSED [ 30%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[7-True] PASSED [ 35%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[8-False] PASSED [ 40%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[9-False] PASSED [ 45%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[10-False] PASSED [ 50%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[11-True] PASSED [ 55%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[12-False] PASSED [ 60%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[13-True] PASSED [ 65%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[14-False] PASSED [ 70%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[15-False] PASSED [ 75%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[16-False] PASSED [ 80%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[17-True] PASSED [ 85%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[18-False] PASSED [ 90%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[19-True] PASSED [ 95%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[20-False] PASSED [100%]\n\n====================== 20 passed, 36 deselected in 0.02s ======================\n```\n\n:::\n\n## Stacked Parametrization\n\n\"People\n\nParametrize gets really interesting when you have a situation where you need to\ntest **combinations of input parameters** against expected outputs. In this\nscenario, stacked parametrization allows you to set up all combinations with\nvery little fuss. \n\nFor this section, I will define a new function built on top of our\n`is_num_prime()` function. This function will take 2 positive integers and add\nthem together, but only if both of the input integers are prime. Otherwise,\nwe'll simply return the input numbers. To keep things simple, we'll always\nreturn a tuple in all cases.\n\n::: {#53dc833c .cell execution_count=10}\n``` {.python .cell-code}\ndef sum_if_prime(pos_int1: int, pos_int2: int) -> tuple:\n \"\"\"Sum 2 integers only if they are prime numbers.\n\n Parameters\n ----------\n pos_int1 : int\n A positive integer.\n pos_int2 : int\n A positive integer.\n\n Returns\n -------\n tuple\n Tuple of one integer if both inputs are prime numbers, else returns a\n tuple of the inputs.\n \"\"\"\n if is_num_prime(pos_int1) and is_num_prime(pos_int2):\n return (pos_int1 + pos_int2,)\n else:\n return (pos_int1, pos_int2)\n```\n:::\n\n\nThen using this function with a range of numbers:\n\n::: {#67be63d5 .cell execution_count=11}\n``` {.python .cell-code}\nfor i in range(1, 6):\n print(f\"{i} and {i} result: {sum_if_prime(i, i)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n1 and 1 result: (1, 1)\n2 and 2 result: (4,)\n3 and 3 result: (6,)\n4 and 4 result: (4, 4)\n5 and 5 result: (10,)\n```\n:::\n:::\n\n\nTesting combinations of input parameters for this function will quickly become\nburdensome:\n\n::: {#efab759e .cell execution_count=12}\n``` {.python .cell-code}\nfrom example_pkg.primes import sum_if_prime\n\n\ndef test_sum_if_prime_with_manual_combinations():\n \"\"\"Manually check several cases.\"\"\"\n assert sum_if_prime(1, 1) == (1, 1)\n assert sum_if_prime(1, 2) == (1, 2)\n assert sum_if_prime(1, 3) == (1, 3)\n assert sum_if_prime(1, 4) == (1, 4)\n assert sum_if_prime(1, 5) == (1, 5)\n assert sum_if_prime(2, 1) == (2, 1)\n assert sum_if_prime(2, 2) == (4,) # the first case where both are primes\n assert sum_if_prime(2, 3) == (5,) \n assert sum_if_prime(2, 4) == (2, 4)\n assert sum_if_prime(2, 5) == (7,)\n # ...\n```\n:::\n\n\n```\n% pytest -k \"test_sum_if_prime_with_manual_combinations\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected\n\ntests/test_primes.py . [100%]\n\n====================== 1 passed, 55 deselected in 0.01s =======================\n\n```\n\n### Single Assertions\n\nBecause we take more than one input parameter, we can use stacked\nparametrization to easily inject all combinations of parameters to a test.\nSimply put, this means that we pass more than one parametrized fixture to the\nsame test. Behind the scenes, `pytest` prepares all parameter combinations to\ninject into our test. \n\nThis allows us to very easily pass all parameter combinations to a\n**single assertion statement**, as in the diagram below.\n\n![Stacked parametrization against a single assertion](https://i.imgur.com/gyx5Jy4.png)\n\nTo use stacked parametrization against our `sum_if_prime()` function, we can\nuse 2 separate iterables:\n\n::: {#adcc1f73 .cell execution_count=13}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\"first_ints\", range(1,6))\n@pytest.mark.parametrize(\"second_ints\", range(1,6))\ndef test_sum_if_prime_stacked_parametrized_inputs(\n first_ints, second_ints, expected_answers):\n \"\"\"Using stacked parameters to set up combinations of all cases.\"\"\"\n assert isinstance(sum_if_prime(first_ints, second_ints), tuple)\n```\n:::\n\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_sum_if_prime_stacked_parametrized_inputs\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\nplugins: anyio-4.0.0\ncollected 56 items / 31 deselected / 25 selected\n\ntest_sum_if_prime_stacked_parametrized_inputs[1-1] PASSED [ 4%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-2] PASSED [ 8%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-3] PASSED [ 12%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-4] PASSED [ 16%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-5] PASSED [ 20%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-1] PASSED [ 24%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-2] PASSED [ 28%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-3] PASSED [ 32%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-4] PASSED [ 36%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-5] PASSED [ 40%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-1] PASSED [ 44%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-2] PASSED [ 48%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-3] PASSED [ 52%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-4] PASSED [ 56%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-5] PASSED [ 60%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-1] PASSED [ 64%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-2] PASSED [ 68%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-3] PASSED [ 72%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-4] PASSED [ 76%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-5] PASSED [ 80%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-1] PASSED [ 84%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-2] PASSED [ 88%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-3] PASSED [ 92%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-4] PASSED [ 96%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-5] PASSED [100%]\n\n====================== 25 passed, 31 deselected in 0.01s ======================\n\n```\n:::\n
\nThe above test; which is 6 lines long; executed 25 tests. This is clearly a very\nbeneficial feature of `pytest`. However, the eagle-eyed among you may have\nspotted a problem - this is only going to work if the expected answer is always\nthe same. The test we defined is only checking that a `tuple` is returned in\nall cases. How can we ensure that we serve the expected answers to the test\ntoo? This is where things get a little fiddly.\n\n### Multiple Assertions\n\nTo test our function against combinations of parameters with\n**different expected answers**, we must employ a dictionary mapping of the\nparameter combinations as keys and the expected assertions as values.\n\n![Using a dictionary to map multiple assertions against stacked parametrized fixtures](https://i.imgur.com/DMkpVG7.png)\n\nTo do this, we need to define a new fixture, which will return the required\ndictionary mapping of parameters to expected values.\n\n::: {#62db2ca7 .cell execution_count=14}\n``` {.python .cell-code}\n# Using stacked parametrization, we can avoid manually typing the cases out,\n# though we do still need to define a dictionary of the expected answers...\n@pytest.fixture\ndef expected_answers() -> dict:\n \"\"\"A dictionary of expected answers for all combinations of 1 through 5.\n\n First key corresponds to `pos_int1` and second key is `pos_int2`.\n\n Returns\n -------\n dict\n Dictionary of cases and their expected tuples.\n \"\"\"\n expected= {\n 1: {1: (1,1), 2: (1,2), 3: (1,3), 4: (1,4), 5: (1,5),},\n 2: {1: (2,1), 2: (4,), 3: (5,), 4: (2,4), 5: (7,),},\n 3: {1: (3,1), 2: (5,), 3: (6,), 4: (3,4), 5: (8,),},\n 4: {1: (4,1), 2: (4,2), 3: (4,3), 4: (4,4), 5: (4,5),},\n 5: {1: (5,1), 2: (7,), 3: (8,), 4: (5,4), 5: (10,),},\n }\n return expected\n```\n:::\n\n\nPassing our `expected_answers` fixture to our test will allow us to match all\nparameter combinations to their expected answer. Let's update\n`test_sum_if_prime_stacked_parametrized_inputs` to use the parameter values to\naccess the expected assertion value from the dictionary.\n\n::: {#2c387a76 .cell execution_count=15}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\"first_ints\", range(1,6))\n@pytest.mark.parametrize(\"second_ints\", range(1,6))\ndef test_sum_if_prime_stacked_parametrized_inputs(\n first_ints, second_ints, expected_answers):\n \"\"\"Using stacked parameters to set up combinations of all cases.\"\"\"\n assert isinstance(sum_if_prime(first_ints, second_ints), tuple)\n answer = sum_if_prime(first_ints, second_ints)\n # using the parametrized values, pull out their keys from the\n # expected_answers dictionary\n assert answer == expected_answers[first_ints][second_ints]\n```\n:::\n\n\nFinally, running this test produces the below `pytest` report.\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_sum_if_prime_stacked_parametrized_inputs\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 31 deselected / 25 selected\n\ntest_sum_if_prime_stacked_parametrized_inputs[1-1] PASSED [ 4%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-2] PASSED [ 8%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-3] PASSED [ 12%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-4] PASSED [ 16%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-5] PASSED [ 20%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-1] PASSED [ 24%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-2] PASSED [ 28%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-3] PASSED [ 32%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-4] PASSED [ 36%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-5] PASSED [ 40%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-1] PASSED [ 44%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-2] PASSED [ 48%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-3] PASSED [ 52%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-4] PASSED [ 56%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-5] PASSED [ 60%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-1] PASSED [ 64%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-2] PASSED [ 68%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-3] PASSED [ 72%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-4] PASSED [ 76%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-5] PASSED [ 80%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-1] PASSED [ 84%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-2] PASSED [ 88%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-3] PASSED [ 92%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-4] PASSED [ 96%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-5] PASSED [100%]\n\n====================== 25 passed, 31 deselected in 0.01s ======================\n```\n\n:::\n\n## Summary\n\nThere you have it - how to use basic and stacked parametrization in your tests.\nWe have:\n\n* used parametrize to inject multiple parameter values to a single test.\n* used stacked parametrize to test combinations of parameters against a single\nassertion.\n* used a nested dictionary fixture to map stacked parametrize input\ncombinations to different expected assertion values.\n\nIf you spot an error with this article, or have a suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Charlie\n* Ethan\n* Henry\n* Sergio\n\nThe diagrams used in this article were produced with the excellent\n[Excalidraw](https://excalidraw.com/), with thanks to Mat for the\nrecommendation.\n\n

fin!

\n\n", + "markdown": "---\ntitle: Parametrized Tests With Pytest in Plain English\nauthor: Rich Leyshon\ndate: June 07 2024\ndescription: Plain English Discussion of Pytest Parametrize\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - parametrize\n - pytest-in-plain-english\nimage: 'https://i.imgur.com/n1flYqU.jpeg'\nimage-alt: Complex sushi conveyor belt with a futuristic theme in a pastel palette.\ntoc: true\ncss: /www/13-pytest-parametrize/styles.css\n---\n\n
\n \n
Complex sushi conveyor belt with a futuristic theme in a pastel palette.
\n
\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses what parametrized tests mean\nand how to implement them with `pytest`. This blog is the third in a series of\nblogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\nfollow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[parametrize branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/parametrize)\nof the repo.\n\n## Overview\n\n### What Are Parametrized Tests?\n\nParametrized tests are simply tests that are applied recursively to multiple\ninput values. For example, rather than testing a function on one input value,\na list of different values could be passed as a parametrized fixture.\n\nA standard approach to testing could look like Figure 1 below, where separate\ntests are defined for the different values we need to check. This would likely\nresult in a fair amount of repeated boilerplate code.\n\n![Figure 1: Testing multiple values without parametrization](https://i.imgur.com/obEM4Oo.png)\n\nInstead, we can reduce the number of tests down to 1 and pass a list of tuples\nto the test instead. Each tuple should contain a parameter value and the\nexpected result, as illustrated in Figure 2.\n\n![Figure 2: Parametrized testing of multiple values](https://i.imgur.com/12QNQxt.png)\n\nSo let's imagine we have a simple function called `double()`, the setup for the\nparametrized list is illustrated in Figure 3.\n\n![Figure 3: Exemplified paramatrization for `test_double()`](https://i.imgur.com/9jqdR9O.png)\n\n### Why use Parametrization?\n\nThis approach allows us to thoroughly check the behaviour of our functions\nagainst multiple values, ensuring that edge-cases are safely treated or\nexceptions are raised as expected. \n\nIn this way, we serve multiple parameters and expected outcomes to a single\ntest, reducing boilerplate code. Parametrization is not a silver bullet, and we\nstill need to define all of our parameters and results in a parametrized\nfixture. This approach is not quite as flexible as the property-based testing\nachievable with a package such as\n[`hypothesis`](https://hypothesis.readthedocs.io/en/latest/). However, the\nlearning curve for `hypothesis` is a bit greater and may be disproportionate to\nthe job at hand.\n\nFor the reasons outlined above, there are likely many competent python\ndevelopers that never use parametrized fixtures. But parametrization does allow\nus to avoid implementing tests with a `for` loop or vectorized approaches to\nthe same outcomes. When coupled with programmatic approaches to generating our\ninput parameters, many lines of code can be saved. And things get even more\ninteresting when we pass multiple parametrized fixtures to our tests, which\nI'll come to in a bit. For these reasons, I believe that awareness of\nparametrization should be promoted among python developers as a useful solution\nin the software development toolkit.\n\n## Implementing Parametrization\n\nIn this section, we will compare some very simple examples of tests with and\nwithout parametrization. Feel free to clone the repository and check out to the\n[example code](https://github.com/r-leyshon/pytest-fiddly-examples/tree/parametrize)\nbranch to run the examples.\n\n### Define the Source Code\n\nHere we define a very basic function that checks whether an integer is prime.\nIf a prime is encountered, then True is returned. If not, then False. The value\n1 gets its own treatment (return `False`). Lastly, we include some basic\ndefensive checks, we return a `TypeError` if anything other than integer is\npassed to the function and a `ValueError` if the integer is less than or equal\nto 0.\n\n::: {#fd9bba60 .cell execution_count=1}\n``` {.python .cell-code}\ndef is_num_prime(pos_int: int) -> bool:\n \"\"\"Check if a positive integer is a prime number.\n\n Parameters\n ----------\n pos_int : int\n A positive integer.\n\n Returns\n -------\n bool\n True if the number is a prime number.\n\n Raises\n ------\n TypeError\n Value passed to `pos_int` is not an integer.\n ValueError\n Value passed to `pos_int` is less than or equal to 0.\n \"\"\"\n if not isinstance(pos_int, int):\n raise TypeError(\"`pos_int` must be a positive integer.\")\n if pos_int <= 0:\n raise ValueError(\"`pos_int` must be a positive integer.\")\n elif pos_int == 1:\n return False\n else:\n for i in range(2, (pos_int // 2) + 1):\n # If divisible by any number 2<>(n/2)+1, it is not prime\n if (pos_int % i) == 0:\n return False\n else:\n return True\n```\n:::\n\n\nRunning this function with a range of values demonstrates its behaviour.\n\n::: {#5a491729 .cell execution_count=2}\n``` {.python .cell-code}\nfor i in range(1, 11):\n print(f\"{i}: {is_num_prime(i)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n1: False\n2: True\n3: True\n4: False\n5: True\n6: False\n7: True\n8: False\n9: False\n10: False\n```\n:::\n:::\n\n\n### Let's Get Testing\n\nLet's begin with the defensive tests. Let's say I need to check that the\nfunction can be relied upon to raise on a number of conditions. The typical\napproach may be to test the raise conditions within a dedicated test function.\n\n::: {#aaf71d0e .cell execution_count=3}\n``` {.python .cell-code}\n\"\"\"Tests for primes module.\"\"\"\nimport pytest\n\nfrom example_pkg.primes import is_num_prime\n\n\ndef test_is_num_primes_exceptions_manually():\n \"\"\"Testing the function's defensive checks.\n\n Here we have to repeat a fair bit of pytest boilerplate.\n \"\"\"\n with pytest.raises(TypeError, match=\"must be a positive integer.\"):\n is_num_prime(1.0)\n with pytest.raises(ValueError, match=\"must be a positive integer.\"):\n is_num_prime(-1)\n```\n:::\n\n\nWithin this function, I can run multiple assertions against several hard-coded\ninputs. I'm only checking against a couple of values here but production-ready\ncode may test against many more cases. To do that, I'd need to have a lot of\nrepeated `pytest.raises` statements. Perhaps more importantly, watch what\nhappens when I run the test.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_manually\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected \n\ntests/test_primes.py . [100%]\n\n======================= 1 passed, 55 deselected in 0.01s ======================\n\n```\n\nNotice that both assertions will either pass or fail together as one test. This\ncould potentially make it more challenging to troubleshoot a failing pipeline.\nIt could be better to have separate test functions for each value, but that\nseems like an awful lot of work...\n\n### ...Enter Parametrize\n\nNow to start using parametrize, we need to use the `@pytest.mark.parametrize`\ndecorator, which takes 2 arguments, a string and an iterable.\n\n::: {#d785773b .cell execution_count=4}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_values, exception_types\", [(1.0, TypeError), (-1, ValueError)]\n )\n```\n:::\n\n\nThe string should contain comma separated values for the names that you would\nlike to refer to when iterating through the iterable. They can be any\nplaceholder you would wish to use in your test. These names will map to the\nindex of elements in the iterable.\n\nSo when I use the fixture with a test, I will expect to inject the following\nvalues:\n\niteration 1... \"some_values\" = 1.0, \"exception_types\" = TypeError \niteration 2... \"some_values\" = -1, \"exception_types\" = ValueError\n\nLet's go ahead and use this parametrized fixture with a test.\n\n::: {#7776a4c1 .cell execution_count=5}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_values, exception_types\", [(1.0, TypeError), (-1, ValueError)]\n )\ndef test_is_num_primes_exceptions_parametrized(some_values, exception_types):\n \"\"\"The same defensive checks but this time with parametrized input.\n\n Less lines in the test but if we increase the number of cases, we need to\n add more lines to the parametrized fixture instead.\n \"\"\"\n with pytest.raises(exception_types, match=\"must be a positive integer.\"):\n is_num_prime(some_values)\n```\n:::\n\n\nThe outcome for running this test is shown below.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_parametrized\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 54 deselected / 2 selected \n\ntests/test_primes.py .. [100%]\n\n======================= 2 passed, 54 deselected in 0.01s ======================\n\n```\n\nIt's a subtle difference, but notice that we now get 2 passing tests rather\nthan 1? We can make this more explicit by passing the `-v` flag (for verbose)\nwhen we invoke `pytest`.\n\n```\n% pytest -k \"test_is_num_primes_exceptions_parametrized\" -v \n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 54 deselected / 2 selected \n\ntest_is_num_primes_exceptions_parametrized[1.0-TypeError] PASSED [ 50%]\ntest_is_num_primes_exceptions_parametrized[-1-ValueError] PASSED [100%]\n\n======================= 2 passed, 54 deselected in 0.01s ======================\n\n```\n\nIn this way, we get a helpful printout of the test and parameter combination\nbeing executed. This can be very helpful in identifying problem cases.\n\n### Yet More Cases\n\nNext up, we may wish to check return values for our function with several\nmore cases. To keep things simple, let's write a test that checks the return\nvalues for a range of numbers between 1 and 5.\n\n::: {#c27359d9 .cell execution_count=6}\n``` {.python .cell-code}\ndef test_is_num_primes_manually():\n \"\"\"Test several positive integers return expected boolean.\n\n This is quite a few lines of code. Note that this runs as a single test.\n \"\"\"\n assert is_num_prime(1) == False\n assert is_num_prime(2) == True\n assert is_num_prime(3) == True\n assert is_num_prime(4) == False\n assert is_num_prime(5) == True\n```\n:::\n\n\nOne way that this can be serialised is by using a list of parameters and\nexpected results.\n\n::: {#d092fcd4 .cell execution_count=7}\n``` {.python .cell-code}\ndef test_is_num_primes_with_list():\n \"\"\"Test the same values using lists.\n\n Less lines but is run as a single test.\n \"\"\"\n answers = [is_num_prime(i) for i in range(1, 6)]\n assert answers == [False, True, True, False, True]\n```\n:::\n\n\nThis is certainly neater than the previous example. Although both\nimplementations will evaluate as a single test, so a failing instance will not\nbe explicitly indicated in the `pytest` report.\n\n```\n% pytest -k \"test_is_num_primes_with_list\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected \n\ntests/test_primes.py . [100%]\n\n======================= 1 passed, 55 deselected in 0.01s ======================\n```\n\nTo parametrize the equivalent test, we can take the below approach.\n\n::: {#69967ecb .cell execution_count=8}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\n \"some_integers, answers\",\n [(1, False), (2, True), (3, True), (4, False), (5, True)]\n )\ndef test_is_num_primes_parametrized(some_integers, answers):\n \"\"\"The same tests but this time with parametrized input.\n\n Fewer lines and 5 separate tests are run by pytest.\n \"\"\"\n assert is_num_prime(some_integers) == answers\n```\n:::\n\n\nThis is slightly more lines than `test_is_num_primes_with_list` but has the\nadvantage of being run as separate tests:\n\n```\n% pytest -k \"test_is_num_primes_parametrized\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 51 deselected / 5 selected \n\ntests/test_primes.py::test_is_num_primes_parametrized[1-False] PASSED [ 20%]\ntests/test_primes.py::test_is_num_primes_parametrized[2-True] PASSED [ 40%]\ntests/test_primes.py::test_is_num_primes_parametrized[3-True] PASSED [ 60%]\ntests/test_primes.py::test_is_num_primes_parametrized[4-False] PASSED [ 80%]\ntests/test_primes.py::test_is_num_primes_parametrized[5-True] PASSED [100%]\n\n======================= 5 passed, 51 deselected in 0.01s ======================\n\n```\n\nWhere this approach really comes into its own is when the number of cases you\nneed to test increases, you can explore ways of generating cases rather than\nhard-coding the values, as in the previous examples.\n\nIn the example below, we can use the `range()` function to generate the \nintegers we need to test, and then zipping these cases to their expected return\nvalues.\n\n::: {#566ff690 .cell execution_count=9}\n``` {.python .cell-code}\n# if my list of cases is growing, I can employ other tactics...\nin_ = range(1, 21)\nout = [\n False, True, True, False, True, False, True, False, False, False,\n True, False, True, False, False, False, True, False, True, False,\n ]\n\n\n@pytest.mark.parametrize(\"some_integers, some_answers\", zip(in_, out))\ndef test_is_num_primes_with_zipped_lists(some_integers, some_answers):\n \"\"\"The same tests but this time with zipped inputs.\"\"\"\n assert is_num_prime(some_integers) == some_answers\n```\n:::\n\n\nRunning this test yields the following result:\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_is_num_primes_with_zipped_lists\" -v \n============================= test session starts =============================\nplatform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0\ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\nplugins: anyio-4.0.0\ncollected 56 items / 36 deselected / 20 selected\n\n/test_primes.py::test_is_num_primes_with_zipped_lists[1-False] PASSED [ 5%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[2-True] PASSED [ 10%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[3-True] PASSED [ 15%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[4-False] PASSED [ 20%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[5-True] PASSED [ 25%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[6-False] PASSED [ 30%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[7-True] PASSED [ 35%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[8-False] PASSED [ 40%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[9-False] PASSED [ 45%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[10-False] PASSED [ 50%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[11-True] PASSED [ 55%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[12-False] PASSED [ 60%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[13-True] PASSED [ 65%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[14-False] PASSED [ 70%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[15-False] PASSED [ 75%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[16-False] PASSED [ 80%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[17-True] PASSED [ 85%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[18-False] PASSED [ 90%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[19-True] PASSED [ 95%]\n/test_primes.py::test_is_num_primes_with_zipped_lists[20-False] PASSED [100%]\n\n====================== 20 passed, 36 deselected in 0.02s ======================\n```\n\n:::\n\n## Stacked Parametrization\n\n\"People\n\nParametrize gets really interesting when you have a situation where you need to\ntest **combinations of input parameters** against expected outputs. In this\nscenario, stacked parametrization allows you to set up all combinations with\nvery little fuss. \n\nFor this section, I will define a new function built on top of our\n`is_num_prime()` function. This function will take 2 positive integers and add\nthem together, but only if both of the input integers are prime. Otherwise,\nwe'll simply return the input numbers. To keep things simple, we'll always\nreturn a tuple in all cases.\n\n::: {#0fa6ed32 .cell execution_count=10}\n``` {.python .cell-code}\ndef sum_if_prime(pos_int1: int, pos_int2: int) -> tuple:\n \"\"\"Sum 2 integers only if they are prime numbers.\n\n Parameters\n ----------\n pos_int1 : int\n A positive integer.\n pos_int2 : int\n A positive integer.\n\n Returns\n -------\n tuple\n Tuple of one integer if both inputs are prime numbers, else returns a\n tuple of the inputs.\n \"\"\"\n if is_num_prime(pos_int1) and is_num_prime(pos_int2):\n return (pos_int1 + pos_int2,)\n else:\n return (pos_int1, pos_int2)\n```\n:::\n\n\nThen using this function with a range of numbers:\n\n::: {#fec7d59c .cell execution_count=11}\n``` {.python .cell-code}\nfor i in range(1, 6):\n print(f\"{i} and {i} result: {sum_if_prime(i, i)}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n1 and 1 result: (1, 1)\n2 and 2 result: (4,)\n3 and 3 result: (6,)\n4 and 4 result: (4, 4)\n5 and 5 result: (10,)\n```\n:::\n:::\n\n\nTesting combinations of input parameters for this function will quickly become\nburdensome:\n\n::: {#c665c1f1 .cell execution_count=12}\n``` {.python .cell-code}\nfrom example_pkg.primes import sum_if_prime\n\n\ndef test_sum_if_prime_with_manual_combinations():\n \"\"\"Manually check several cases.\"\"\"\n assert sum_if_prime(1, 1) == (1, 1)\n assert sum_if_prime(1, 2) == (1, 2)\n assert sum_if_prime(1, 3) == (1, 3)\n assert sum_if_prime(1, 4) == (1, 4)\n assert sum_if_prime(1, 5) == (1, 5)\n assert sum_if_prime(2, 1) == (2, 1)\n assert sum_if_prime(2, 2) == (4,) # the first case where both are primes\n assert sum_if_prime(2, 3) == (5,) \n assert sum_if_prime(2, 4) == (2, 4)\n assert sum_if_prime(2, 5) == (7,)\n # ...\n```\n:::\n\n\n```\n% pytest -k \"test_sum_if_prime_with_manual_combinations\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 55 deselected / 1 selected\n\ntests/test_primes.py . [100%]\n\n====================== 1 passed, 55 deselected in 0.01s =======================\n\n```\n\n### Single Assertions\n\nBecause we take more than one input parameter, we can use stacked\nparametrization to easily inject all combinations of parameters to a test.\nSimply put, this means that we pass more than one parametrized fixture to the\nsame test. Behind the scenes, `pytest` prepares all parameter combinations to\ninject into our test. \n\nThis allows us to very easily pass all parameter combinations to a\n**single assertion statement**, as in the diagram below.\n\n![Stacked parametrization against a single assertion](https://i.imgur.com/gyx5Jy4.png)\n\nTo use stacked parametrization against our `sum_if_prime()` function, we can\nuse 2 separate iterables:\n\n::: {#7d32bf19 .cell execution_count=13}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\"first_ints\", range(1,6))\n@pytest.mark.parametrize(\"second_ints\", range(1,6))\ndef test_sum_if_prime_stacked_parametrized_inputs(\n first_ints, second_ints, expected_answers):\n \"\"\"Using stacked parameters to set up combinations of all cases.\"\"\"\n assert isinstance(sum_if_prime(first_ints, second_ints), tuple)\n```\n:::\n\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_sum_if_prime_stacked_parametrized_inputs\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\nplugins: anyio-4.0.0\ncollected 56 items / 31 deselected / 25 selected\n\ntest_sum_if_prime_stacked_parametrized_inputs[1-1] PASSED [ 4%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-2] PASSED [ 8%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-3] PASSED [ 12%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-4] PASSED [ 16%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-5] PASSED [ 20%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-1] PASSED [ 24%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-2] PASSED [ 28%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-3] PASSED [ 32%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-4] PASSED [ 36%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-5] PASSED [ 40%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-1] PASSED [ 44%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-2] PASSED [ 48%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-3] PASSED [ 52%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-4] PASSED [ 56%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-5] PASSED [ 60%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-1] PASSED [ 64%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-2] PASSED [ 68%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-3] PASSED [ 72%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-4] PASSED [ 76%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-5] PASSED [ 80%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-1] PASSED [ 84%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-2] PASSED [ 88%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-3] PASSED [ 92%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-4] PASSED [ 96%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-5] PASSED [100%]\n\n====================== 25 passed, 31 deselected in 0.01s ======================\n\n```\n:::\n
\nThe above test; which is 6 lines long; executed 25 tests. This is clearly a very\nbeneficial feature of `pytest`. However, the eagle-eyed among you may have\nspotted a problem - this is only going to work if the expected answer is always\nthe same. The test we defined is only checking that a `tuple` is returned in\nall cases. How can we ensure that we serve the expected answers to the test\ntoo? This is where things get a little fiddly.\n\n### Multiple Assertions\n\nTo test our function against combinations of parameters with\n**different expected answers**, we must employ a dictionary mapping of the\nparameter combinations as keys and the expected assertions as values.\n\n![Using a dictionary to map multiple assertions against stacked parametrized fixtures](https://i.imgur.com/DMkpVG7.png)\n\nTo do this, we need to define a new fixture, which will return the required\ndictionary mapping of parameters to expected values.\n\n::: {#ff6598d9 .cell execution_count=14}\n``` {.python .cell-code}\n# Using stacked parametrization, we can avoid manually typing the cases out,\n# though we do still need to define a dictionary of the expected answers...\n@pytest.fixture\ndef expected_answers() -> dict:\n \"\"\"A dictionary of expected answers for all combinations of 1 through 5.\n\n First key corresponds to `pos_int1` and second key is `pos_int2`.\n\n Returns\n -------\n dict\n Dictionary of cases and their expected tuples.\n \"\"\"\n expected= {\n 1: {1: (1,1), 2: (1,2), 3: (1,3), 4: (1,4), 5: (1,5),},\n 2: {1: (2,1), 2: (4,), 3: (5,), 4: (2,4), 5: (7,),},\n 3: {1: (3,1), 2: (5,), 3: (6,), 4: (3,4), 5: (8,),},\n 4: {1: (4,1), 2: (4,2), 3: (4,3), 4: (4,4), 5: (4,5),},\n 5: {1: (5,1), 2: (7,), 3: (8,), 4: (5,4), 5: (10,),},\n }\n return expected\n```\n:::\n\n\nPassing our `expected_answers` fixture to our test will allow us to match all\nparameter combinations to their expected answer. Let's update\n`test_sum_if_prime_stacked_parametrized_inputs` to use the parameter values to\naccess the expected assertion value from the dictionary.\n\n::: {#7e072d89 .cell execution_count=15}\n``` {.python .cell-code}\n@pytest.mark.parametrize(\"first_ints\", range(1,6))\n@pytest.mark.parametrize(\"second_ints\", range(1,6))\ndef test_sum_if_prime_stacked_parametrized_inputs(\n first_ints, second_ints, expected_answers):\n \"\"\"Using stacked parameters to set up combinations of all cases.\"\"\"\n assert isinstance(sum_if_prime(first_ints, second_ints), tuple)\n answer = sum_if_prime(first_ints, second_ints)\n # using the parametrized values, pull out their keys from the\n # expected_answers dictionary\n assert answer == expected_answers[first_ints][second_ints]\n```\n:::\n\n\nFinally, running this test produces the below `pytest` report.\n\n:::{.scrolling}\n\n```\n% pytest -k \"test_sum_if_prime_stacked_parametrized_inputs\" -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 \ncachedir: .pytest_cache\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 56 items / 31 deselected / 25 selected\n\ntest_sum_if_prime_stacked_parametrized_inputs[1-1] PASSED [ 4%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-2] PASSED [ 8%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-3] PASSED [ 12%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-4] PASSED [ 16%]\ntest_sum_if_prime_stacked_parametrized_inputs[1-5] PASSED [ 20%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-1] PASSED [ 24%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-2] PASSED [ 28%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-3] PASSED [ 32%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-4] PASSED [ 36%]\ntest_sum_if_prime_stacked_parametrized_inputs[2-5] PASSED [ 40%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-1] PASSED [ 44%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-2] PASSED [ 48%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-3] PASSED [ 52%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-4] PASSED [ 56%]\ntest_sum_if_prime_stacked_parametrized_inputs[3-5] PASSED [ 60%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-1] PASSED [ 64%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-2] PASSED [ 68%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-3] PASSED [ 72%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-4] PASSED [ 76%]\ntest_sum_if_prime_stacked_parametrized_inputs[4-5] PASSED [ 80%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-1] PASSED [ 84%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-2] PASSED [ 88%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-3] PASSED [ 92%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-4] PASSED [ 96%]\ntest_sum_if_prime_stacked_parametrized_inputs[5-5] PASSED [100%]\n\n====================== 25 passed, 31 deselected in 0.01s ======================\n```\n\n:::\n\n## Summary\n\nThere you have it - how to use basic and stacked parametrization in your tests.\nWe have:\n\n* used parametrize to inject multiple parameter values to a single test.\n* used stacked parametrize to test combinations of parameters against a single\nassertion.\n* used a nested dictionary fixture to map stacked parametrize input\ncombinations to different expected assertion values.\n\nIf you spot an error with this article, or have a suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Charlie\n* Ethan\n* Henry\n* Sergio\n\nThe diagrams used in this article were produced with the excellent\n[Excalidraw](https://excalidraw.com/), with thanks to Mat for the\nrecommendation.\n\n

fin!

\n\n", "supporting": [ - "13-pytest-parametrize_files" + "13-pytest-parametrize_files/figure-html" ], "filters": [], "includes": {} diff --git a/_freeze/blogs/15-pytest-mocking/execute-results/html.json b/_freeze/blogs/15-pytest-mocking/execute-results/html.json index e3d9d5e..80e2be0 100644 --- a/_freeze/blogs/15-pytest-mocking/execute-results/html.json +++ b/_freeze/blogs/15-pytest-mocking/execute-results/html.json @@ -1,10 +1,10 @@ { - "hash": "6a89955d3f602f67b11d85df5ce43957", + "hash": "53fc3c2ddf96776b76f789aa8c83de24", "result": { "engine": "jupyter", - "markdown": "---\ntitle: Mocking With Pytest in Plain English\nauthor: Rich Leyshon\ndate: July 14 2024\ndescription: Plain English Comparison of Mocking Approaches in Python\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - mocking\n - pytest-in-plain-english\n - mockito\n - MagicMock\n - monkeypatch\nimage: 'https://i.imgur.com/K0mxjuF.jpeg'\nimage-alt: Soul singer with Joker makeup.\ntoc: true\ncss: /www/13-pytest-parametrize/styles.css\n---\n\n
\n \n
The Joker sings the Green Green Grass of Home.
\n
\n\n> \"A day without laughter is a day wasted.\" Charlie Chaplin\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses the dark art of mocking, why\nyou should do it and the nuts and bolts of implementing mocked tests. This blog\nis the fourth in a series of blogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n### What does Mocking Mean?\n\nCode often has external dependencies:\n\n* Web APIs (as in this article)\n* Websites (if scraping / crawling)\n* External code (importing packages)\n* Data feeds and databases\n* Environment variables\n\nAs developers cannot control the behaviour of those dependencies, they would\nnot write tests dependent upon them. In order to test their source code that\ndepends on these services, developers need to replace the properties of these\nservices when the test suite runs. Injecting replacement values into the code\nat runtime is generally referred to as mocking. Mocking these values means that\ndevelopers can feed dependable results to their code and make reliable\nassertions about the code's behaviour, without changes in the 'outside world'\naffecting outcomes in the system under test.\n\nDevelopers who write unit tests may also mock their own code. The \"unit\" in the\nterm \"unit test\" implies complete isolation from external dependencies. Mocking\nis an indispensible tool in achieving that isolation within a test suite. It\nensures that code can be efficiently verified in any order, without\ndependencies on other elements in your codebase. However, mocking also adds to\ncode complexity, increasing cognitive load and generally making things harder\nto debug.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\nThe code may not always be optimal, favouring a simplistic approach wherever\npossible.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python, HTTP requests and some\nfamiliarity with `pytest` and packaging. The type of programmer who has\nwondered about how to follow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1 requests mockito`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[mocking branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking)\nof the repo.\n\n## Overview\n\nMocking is one of the trickier elements of testing. It's a bit niche and is\noften perceived to be too hacky to be worth the effort. The options for mocking\nin python are numerous and this adds to the complexity of many example\nimplementations you will find online. \n\nThere is also a compromise in simplicity versus flexibility. Some of the\noptions available are quite involved and can be adapted to the nichest of\ncases, but may not be the best option for those new to mocking. With this in\nmind, I present 3 alternative methods for mocking python source code. So if\nyou'll forgive me, this is the first of the\n[`pytest` in plain English](/blogs/index.qmd#category=pytest-in-plain-english)\nseries where I introduce alternative testing practices from beyond the `pytest`\npackage.\n\n1. [**monkeypatch**][monkeypatch]: The `pytest` fixture designed for mocking. The\norigin of the fixture's name is debated but potentially arose from the term\n'guerrilla patch' which may have been misinterpreted as 'gorilla patch'. This\nis the concept of modifying source code at runtime, which probably sounds a bit\nlike 'monkeying with the code'.\n2. [**MagicMock**][magicmock]: This is the mocking object provided by python3's\nbuiltin `unittest` package.\n3. [**mockito**][mockito]: This package is based upon the popular Java framework\nof the same name. Despite having a user-friendly syntax, `mockito` is robust\nand secure.\n\n:::{.callout-note collapse=\"true\"}\n\n[monkeypatch]: https://docs.pytest.org/en/stable/how-to/monkeypatch.html\n[magicmock]: https://docs.python.org/3/library/unittest.mock.html\n[mockito]: https://mockito-python.readthedocs.io/en/latest/walk-through.html\n\n\n### A note on the language\n\nMocking has a bunch of synonyms & related language which can be a bit\noff-putting. All of the below terms are associated with mocking. Some may be\npreferred to the communities of specific programming frameworks over others.\n\n| Term | Brief Meaning | Frameworks/Libraries |\n|---------------|---------------|----------------------|\n| Mocking | Creating objects that simulate the behaviour of real objects for testing | Mockito (Java), unittest.mock (Python), Jest (JavaScript), Moq (.NET) |\n| Spying | Observing and recording method calls on real objects | Mockito (Java), Sinon (JavaScript), unittest.mock (Python), RSpec (Ruby) |\n| Stubbing | Replacing methods with predefined behaviours or return values | Sinon (JavaScript), RSpec (Ruby), PHPUnit (PHP), unittest.mock (Python) |\n| Patching | Temporarily modifying or replacing parts of code for testing | unittest.mock (Python), pytest-mock (Python), PowerMock (Java) |\n| Faking | Creating simplified implementations of complex dependencies | Faker (multiple languages), Factory Boy (Python), FactoryGirl (Ruby) |\n| Dummy Objects | Placeholder objects passed around but never actually used | Can be created in any testing framework |\n\n:::\n\n## Mocking in Python\n\nThis section will walk through some code that uses HTTP requests to an external\nservice and how we can go about testing the code's behaviour without relying on\nthat service being available. Feel free to clone the repository and check out\nto the\n[example code](https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking)\nbranch to run the examples.\n\nThe purpose of the code is to retrieve jokes from \nlike so:\n\n\n\n::: {#ddd5bf31 .cell execution_count=2}\n``` {.python .cell-code}\nfor _ in range(3):\n print(get_joke(f=\"application/json\"))\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nHow do robots eat guacamole? With computer chips.\nI was thinking about moving to Moscow but there is no point Russian into things.\nWhy do fish live in salt water? Because pepper makes them sneeze!\n```\n:::\n:::\n\n\n::: {.callout-caution}\n\nThe jokes are provided by and are not curated by\nme. In my testing of the service I have found the jokes to be harmless fun, but\nI cannot guarantee that. If an offensive joke is returned, this is\nunintentional but\n[let me know about it](https://github.com/r-leyshon/blogging/issues) and I will\ngenerate new jokes.\n\n:::\n\n### Define the Source Code\n\nThe function `get_joke()` uses 2 internals:\n\n1. `_query_endpoint()` Used to construct the HTTP request with required headers\nand user agent.\n2. `_handle_response()` Used to catch HTTP errors, or to pull the text out of\nthe various response formats.\n\n::: {#16733e20 .cell execution_count=3}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n \"\"\"Utility for formatting query string & requesting endpoint.\"\"\"\n HEADERS = {\n \"User-Agent\": usr_agent,\n \"Accept\": f,\n }\n resp = requests.get(endp, headers=HEADERS)\n return resp\n```\n:::\n\n\nKeeping separate, the part of the codebase that you wish to target for mocking\nis often the simplest way to go about things. The target for our mocking will\nbe the command that integrates with the external service, so `requests.get()`\nhere. \n\nThe use of `requests.get()` in the code above depends on a few things:\n\n1. An endpoint string.\n2. A dictionary with string values for the keys \"User-Agent\" and \"Accept\".\n\nWe'll need to consider those dependencies when mocking. Once we return a\nresponse from the external service, we need a utility to handle the various\nstatuses of that response:\n\n::: {#49415e6c .cell execution_count=4}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n ...\n\n\ndef _handle_response(r: requests.models.Response) -> str:\n \"\"\"Utility for handling reponse object & returning text content.\n\n Parameters\n ----------\n r : requests.models.Response\n Response returned from webAPI endpoint.\n\n Raises\n ------\n NotImplementedError\n Requested format `f` was not either 'text/plain' or 'application/json'. \n requests.HTTPError\n HTTP error was encountered.\n \"\"\"\n if r.ok:\n c_type = r.headers[\"Content-Type\"]\n if c_type == \"application/json\":\n content = r.json()\n content = content[\"joke\"]\n elif c_type == \"text/plain\":\n content = r.text\n else:\n raise NotImplementedError(\n \"This client accepts 'application/json' or 'text/plain' format\"\n )\n else:\n raise requests.HTTPError(\n f\"{r.status_code}: {r.reason}\"\n )\n return content\n```\n:::\n\n\nOnce `_query_endpoint()` gets us a response, we can feed it into\n`_handle_response()`, where different logic is executed depending on the\nresponse's properties. Specifically, any response we want to mock would need\nthe following:\n\n1. headers, containing a dictionary eg: `{\"content_type\": \"plain/text\"}`\n2. A `json()` method.\n3. `text`, `status_code` and `reason` attributes.\n\nFinally, the above functions get wrapped in the `get_joke()` function below:\n\n::: {#459f0941 .cell execution_count=5}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n ...\n\n\ndef _handle_response(r: requests.models.Response) -> str:\n ...\n\n\ndef get_joke(\n endp:str = \"https://icanhazdadjoke.com/\",\n usr_agent:str = \"datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)\", \n f:str = \"text/plain\",\n) -> str:\n \"\"\"Request a joke from icanhazdadjoke.com.\n\n Ask for a joke in either plain text or JSON format. Return the joke text.\n\n Parameters\n ----------\n endp : str, optional\n Endpoint to query, by default \"https://icanhazdadjoke.com/\"\n usr_agent : str, optional\n User agent value, by default\n \"datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)\"\n f : str, optional\n Format to request eg \"application.json\", by default \"text/plain\"\n\n Returns\n -------\n str\n Joke text.\n \"\"\"\n r = _query_endpoint(endp=endp, usr_agent=usr_agent, f=f)\n return _handle_response(r)\n```\n:::\n\n\n### Let's Get Testing\n\nThe behaviour in `get_joke()` is summarised in the flowchart below:\n\n\"get_joke()\n\nThere are 4 outcomes to check, coloured red and green in the process chart\nabove.\n\n1. `get_joke()` successfully returns joke text when the user asked for json\nformat.\n2. `get_joke()` successfully returns joke text when the user asked for plain\ntext.\n3. `get_joke()` raises `NotImplementedError` if any other valid format is asked\nfor. Note that the API also accepts HTML and image formats, though parsing the\njoke text out of those is more involved and beyond the scope of this blog.\n4. `get_joke()` raises a `HTTPError` if the response from the API was not ok.\n\nNote that the event that we wish to target for mocking is highlighted in blue - \nwe don't want our tests to execute any real requests.\n\nThe strategy for testing this function without making requests to the web API\nis composed of 4 similar steps, regardless of the package used to implement the\nmocking. \n\n\"4\n\n1. **Mock:** Define the object or property that you wish to use as a\nreplacement. This could be a static value or something a bit more involved,\nlike a mock class that can return dynamic values depending upon the values it\nreceives. \n2. **Patch:** Replace part of the source code with our mock value. \n3. **Use:** Use the source code to return a value.\n4. **Assert:** Check the returned value is what you expect.\n\nIn the examples that follow, I will label the equivalent steps for the various\nmocking implementations.\n\n### The \"Ultimate Joke\"\n\nWhat hard-coded text shall I use for my expected joke? I'll\n[create a fixture](/blogs/11-fiddly-bits-of-pytest.qmd) that will serve\nup this joke text to all of the test modules used below. I'm only going to\ndefine it once and then refer to it throughout several examples below. So it\nneeds to be a pretty memorable, awesome joke.\n\n::: {#2d8a6491 .cell execution_count=6}\n``` {.python .cell-code}\nimport pytest\n\n\n@pytest.fixture(scope=\"session\")\ndef ULTI_JOKE():\n return (\"Doc, I can't stop singing 'The Green, Green Grass of Home.' That \"\n \"sounds like Tom Jones Syndrome. Is it common? Well, It's Not Unusual.\")\n```\n:::\n\n\nBeing a Welshman, I may be a bit biased. But that's a pretty memorable dad joke\nin my opinion. This joke will be available to every test within my test suite\nwhen I execute `pytest` from the command line. The assertions that we will\nuse when using `get_joke()` will expect this string to be returned. If some\nother joke is returned, then we have not mocked correctly and an HTTP request\nwas sent to the API.\n\n### Mocking Everything\n\nI'll start with an example of how to mock `get_joke()` completely. This is an\nintentionally bad idea. In doing this, the test won't actually be executing any\nof the code, just returning a hard-coded value for the joke text. All this does\nis prove that the mocking works as expected and has nothing to do with the\nlogic in our source code. \n\nSo why am I doing it? Hopefully I can illustrate the most basic implementation\nof mocking in this way. I'm not having to think about how I can mock a response\nobject with all the required properties. I just need to provide some hard coded\ntext. \n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#08ca2e13 .cell execution_count=7}\n``` {.python .cell-code}\nimport example_pkg.only_joking\n\n\ndef test_get_joke_monkeypatched_entirely(monkeypatch, ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1\n def _mock_joke():\n \"\"\"Return the joke text.\n\n monkeypatch.setattr expects the value argument to be callable. In plain\n English, a function or class.\"\"\"\n return ULTI_JOKE\n # step 2\n monkeypatch.setattr(\n target=example_pkg.only_joking,\n name=\"get_joke\",\n value=_mock_joke\n )\n # step 3 & 4\n # Use the module's namespace to correspond with the monkeypatch\n assert example_pkg.only_joking.get_joke() == ULTI_JOKE \n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * Step 2 requires the hard coded text to be returned from a callable, like\n a function or class. So we define `_mock_joke` to serve the text in the\n required format.\n* **Step 2**\n * `monkeypatch.setattr()` is able to take the module namespace that we\n imported as the target. This must be the namespace where the function\n (or variable etc) is defined.\n* **Step 3**\n * When invoking the function, be sure to reference the function in the same\n way as it was monkeypatched.\n * Aliases can also be used if preferable\n (eg `import example_pkg.only_joking as jk`). Be sure to update your\n reference to `get_joke()` in step 2 and 3 to match your import statement.\n\n:::\n\n#### MagicMock\n\n::: {#997002bb .cell execution_count=8}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_magicmocked_entirely(ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1\n _mock_joke = MagicMock(return_value=ULTI_JOKE)\n # step 2\n with patch(\"example_pkg.only_joking.get_joke\", _mock_joke):\n # step 3\n joke = example_pkg.only_joking.get_joke()\n # step 4\n assert joke == ULTI_JOKE\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * `MagicMock()` allows us to return static values as mock objects.\n* **Step 3**\n * When you use `get_joke()`, be sure to call reference the namespace in the\n same way as to your patch in step 2.\n\n:::\n\n#### mockito\n\n::: {#59a30ec3 .cell execution_count=9}\n``` {.python .cell-code}\nfrom mockito import when, unstub\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_mockitoed_entirely(ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1 & 2\n when(example_pkg.only_joking).get_joke().thenReturn(ULTI_JOKE)\n # step 3\n joke = example_pkg.only_joking.get_joke()\n # step 4\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1 & 2**\n * `mockito`'s intuitive `when(...).thenReturn(...)` pattern allows you\n to reference any object within the imported namespace. Like with\n `MagicMock`, the static string `ULTI_JOKE` can be referenced.\n* **Step 3**\n * When you use `get_joke()`, be sure to call reference the namespace in the\n same way as to your patch in step 2.\n* **unstub**\n * This step explicitly 'unpatches' `get_joke()`. If you did not `unstub()`,\n the patch to `get_joke()` would persist through the rest of your tests.\n * `mockito` allows you to implicitly `unstub()` by using the context\n manager `with`.\n\n:::\n\n:::\n\n### `monkeypatch()` without OOP\n\nSomething I've noticed about the `pytest` documentation for `monkeypatch`, is\nthat it gets straight into mocking with Object Oriented Programming (OOP).\nWhile this may be a bit more convenient, it is certainly not a requirement of\nusing `monkeypatch` and definitely adds to the cognitive load for new users.\nThis first example will mock the value of `requests.get` without using classes.\n\n::: {#ae5e80c2 .cell execution_count=10}\n``` {.python .cell-code}\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_monkeypatched_no_OOP(monkeypatch, ULTI_JOKE):\n # step 1: Mock the response object\n def _mock_response(*args, **kwargs):\n resp = requests.models.Response()\n resp.status_code = 200\n resp._content = ULTI_JOKE.encode(\"UTF8\")\n resp.headers = {\"Content-Type\": \"text/plain\"}\n return resp\n \n # step 2: Patch requests.get\n monkeypatch.setattr(requests, \"get\", _mock_response)\n # step 3: Use requests.get\n joke = get_joke()\n # step 4: Assert\n assert joke == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{joke}'\"\n # will also work for json format\n joke = get_joke(f=\"application/json\")\n assert joke == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{joke}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * The return value of `requests.get()` will be a response object. We need\n to mock this object with the methods and attributes required by the\n `_handle_response()` function. \n * We need to encode the static joke text as bytes to format the data. \n Response objects encode data as bytes for interoperatability and\n optimisation purposes.\n* **step4**\n * As we have set an appropriate value for the mocked response's `_content`\n attribute, the mocked joke will be returned for both JSON and plain text\n formats - very convenient!\n\n:::\n\n### Condition 1: Test JSON\n\nIn this example, we demonstrate the same functionality as above, but with an\nobject-oriented design pattern. This approach more closely follows that of\nthe `pytest` documentation. As before, `MagicMock` and `mockito` examples will\nbe included. \n\nThe purpose of this test is to test the outcome of `get_joke()` when the user\nspecifies a json format.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#monkey-fixture .cell execution_count=11}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"Return a class instance that will mock all the properties of a response\n object that get_joke needs to work.\n \"\"\"\n HEADERS_MAP = {\n \"text/plain\": {\"Content-Type\": \"text/plain\"},\n \"application/json\": {\"Content-Type\": \"application/json\"},\n \"text/html\": {\"Content-Type\": \"text/html\"},\n }\n\n class MockResponse:\n def __init__(self, f, *args, **kwargs):\n self.ok = True\n self.f = f\n self.headers = HEADERS_MAP[f] # header corresponds to format that\n # the user requested\n self.text = ULTI_JOKE \n\n def json(self):\n if self.f == \"application/json\":\n return {\"joke\": ULTI_JOKE}\n return None\n\n return MockResponse\n\n\ndef test_get_joke_json_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\n\n Test get_joke using the mock class fixture. This approach is the\n implementation suggested in the pytest docs.\n \"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n \"\"\"Return fixtures with the correct header.\n\n If the test uses \"text/plain\" format, we need to return a MockResponse\n class instance with headers attribute equal to\n {\"Content-Type\": \"text/plain\"}, likewise for JSON.\n \"\"\"\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # Step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # Step 3: Use\n j_json = get_joke(f=\"application/json\")\n # Step 4: Assert\n assert j_json == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{j_json}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We define a mocked class instance with the necessary properties expected\n by `_handle_response()`.\n * The mocked response is served to our test as a `pytest` fixture.\n * Within the test, we need another function, which will be able to take\n the arguments passed to `requests.get()`. This will allow our class\n instance to retrieve the appropriate header from the `HEADERS_MAP`\n dictionary.\n\nAs you may appreciate, this does not appear to be the most straight forward\nimplementation, but it will allow us to test when the user asks for JSON,\nplain text or HTML formats. In the above test, we assert against JSON format\nonly.\n\n:::\n\n#### MagicMock\n\n::: {#3ae7ec2f .cell execution_count=12}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_json_magicmocked(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"application/json\"}\n mock_response.json.return_value = {\"joke\": ULTI_JOKE}\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3: Use\n joke = get_joke(f=\"application/json\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * `MagicMock()` can return a mock object with a specification designed to\n mock response objects. Super useful.\n * Our static joke content can be served directly to `MagicMock` without the\n need for an intermediate class.\n * In comparison to the `monkeypatch` approach, this appears to be more\n straight forward and maintainable.\n:::\n\n#### mockito\n\n::: {#b34a65e2 .cell execution_count=13}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_json_mockitoed(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\"\"\"\n # step 1: Mock\n _mock_response = requests.models.Response()\n _mock_response.status_code = 200\n _mock_response._content = b'{\"joke\": \"' + ULTI_JOKE.encode(\"utf-8\") + b'\"}'\n _mock_response.headers = {\"Content-Type\": \"application/json\"}\n # step 2: Patch\n when(requests).get(...).thenReturn(_mock_response)\n # step 3: Use\n joke = example_pkg.only_joking.get_joke(f=\"application/json\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * In order to encode the expected joke for JSON format, we need a\n dictionary encoded within a bytestring. This bit is a little tricky.\n * Alternatively, define the expected dictionary and use the `json`\n package. `json.dumps(dict).encode(\"UTF8\")` will format the content\n dictionary in the required way.\n* **Step 2**\n * `mockito`'s `when()` approach will allow you to access the methods of the\n object that is being patched, in this case `requests`. \n * `mockito` allows you to pass the `...` argument to a patched method, to\n indicate that whatever arguments were passed to `get()`, return the\n specified mock value.\n * Being able to specify values passed in place of `...` will allow you to\n set different return values depending on argument values received by\n `get()`.\n:::\n\n:::\n\n### Condition 2: Test Plain Text\n\nThe purpose of this test is to check the outcome when the user specifies a\nplain/text format while using `get_joke()`.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#c3dd90d2 .cell execution_count=14}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"The same fixture as was used for testing JSON format\"\"\"\n ...\n\n\ndef test_get_joke_text_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # step 3: Use\n j_txt = get_joke(f=\"text/plain\")\n # step 4: Assert\n assert j_txt == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{j_txt}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We can use the same mock class as for testing\n [Condition 1](#monkey-fixture), due to the content of the `HEADERS_MAP`\n dictionary.\n:::\n\n#### MagicMock\n\n::: {#e7771f9f .cell execution_count=15}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_text_magicmocked(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"text/plain\"}\n mock_response.text = ULTI_JOKE\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3: Use\n joke = get_joke(f=\"text/plain\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n```\n:::\n\n\n#### mockito\n\n::: {#c82d0fcd .cell execution_count=16}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\ndef test_get_joke_text_mockitoed(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n mock_response = requests.models.Response()\n mock_response.status_code = 200\n mock_response._content = ULTI_JOKE.encode(\"utf-8\")\n mock_response.headers = {\"Content-Type\": \"text/plain\"}\n # step 2: Patch\n when(requests).get(...).thenReturn(mock_response)\n # step 3: Use\n joke = example_pkg.only_joking.get_joke(f=\"text/plain\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::\n\n### Condition 3: Test Not Implemented\n\nThis test will check the outcome of what happens when the user asks for a\nformat other than text or JSON format. As the webAPI also offers image or HTML\nformats, a response 200 (ok) would be returned from the service. But I was too\nbusy (lazy) to extract the text from those formats. \n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#f8ee246e .cell execution_count=17}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"The same fixture as was used for testing JSON format\"\"\"\n ...\n\n\ndef test_get_joke_not_implemented_monkeypatched(\n monkeypatch, _mock_response):\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"This client accepts 'application/json' or 'text/plain' format\"):\n get_joke(f=\"text/html\")\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We can use the same mock class as for testing\n [Condition 1](#monkey-fixture), due to the content of the `HEADERS_MAP`\n dictionary.\n* **Step 4**\n * We use a context manager (`with pytest.raises`) which catches the raised\n exception and stops it from terminating our `pytest` session. \n * The asserted `match` argument can take a regular expression, so that\n wildcard patterns can be used. This allows matching of part of the\n exception message.\n:::\n\n#### MagicMock\n\n::: {#56d09b58 .cell execution_count=18}\n``` {.python .cell-code}\nimport pytest\nimport requests\nfrom unittest.mock import MagicMock, patch\n\nfrom example_pkg.only_joking import get_joke\ndef test__handle_response_not_implemented_magicmocked():\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"text/html\"}\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"client accepts 'application/json' or 'text/plain' format\"):\n get_joke(f=\"text/html\")\n```\n:::\n\n\n#### mockito\n\n::: {#9965fac2 .cell execution_count=19}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\ndef test_get_joke_not_implemented_mockitoed():\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n mock_response = requests.models.Response()\n mock_response.status_code = 200\n mock_response.headers = {\"Content-Type\": \"text/html\"}\n # step 2: Patch\n when(\n example_pkg.only_joking\n )._query_endpoint(...).thenReturn(mock_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"This client accepts 'application/json' or 'text/plain' format\"):\n example_pkg.only_joking.get_joke(f=\"text/html\")\n unstub()\n```\n:::\n\n\n:::\n\n### Condition 4: Test Bad Response\n\nIn this test, we simulate a bad response from the webAPI, which could arise\nfor a number of reasons:\n\n* The api is unavailable.\n* The request asked for a resource that is not available.\n* Too many requests were made in a short period.\n\nThese conditions are those that we have the least control over and therefore\nhave the greatest need for mocking.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#ed4883c7 .cell execution_count=20}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke, _handle_response\n\n\n@pytest.fixture\ndef _mock_bad_response():\n class MockBadResponse:\n def __init__(self, *args, **kwargs):\n self.ok = False\n self.status_code = 404\n self.reason = \"Not Found\"\n return MockBadResponse\n\n\ndef test_get_joke_http_error_monkeypatched(\n monkeypatch, _mock_bad_response):\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n def _mock_get_bad_response(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_bad_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_bad_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n get_joke()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * This time we need to define a new fixture that returns a bad response.\n * Alternatively, we could have implemented a single fixture for all of our\n tests that dynamically served a good or bad response dependent upon\n arguments passed to `get_joke()`, for example different string values\n passed as the endpoint.\n * In a more thorough implementation of `get_joke()`, you may wish to retry\n the request for certain HTTP error status codes. The ability to provide\n mocked objects that reliably serve those statuses allow you to\n deterministically validate your code's behaviour.\n\n:::\n\n#### MagicMock\n\n::: {#67bda446 .cell execution_count=21}\n``` {.python .cell-code}\nimport pytest\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_http_error_magicmocked():\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n _mock_response = MagicMock(spec=requests.models.Response)\n _mock_response.ok = False\n _mock_response.status_code = 404\n _mock_response.reason = \"Not Found\"\n # step 2: Patch\n with patch(\"requests.get\", return_value=_mock_response):\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n get_joke()\n```\n:::\n\n\n#### mockito\n\n::: {#ed4c1ed4 .cell execution_count=22}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_http_error_mockitoed():\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n _mock_response = requests.models.Response()\n _mock_response.status_code = 404\n _mock_response.reason = \"Not Found\"\n # step 2: Patch\n when(example_pkg.only_joking)._query_endpoint(...).thenReturn(\n _mock_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n example_pkg.only_joking.get_joke()\n unstub()\n```\n:::\n\n\n:::\n\n\n## Summary\n\nWe have thoroughly tested our code using approaches that mock the behaviour of\nan external webAPI. We have also seen how to implement those tests with 3\ndifferent packages.\n\nI hope that this has provided you with enough introductory material to begin\nmocking tests if you have not done so before. If you find that your specific\nuse case for mocking is quite nuanced and fiddly (it's likely to be that way),\nthen the alternative implementations presented here can help you to understand\nhow to solve your specific mocking dilemma. \n\nOne final quote for those developers having their patience tested by errors\nattempting to implement mocking:\n\n> \"He who laughs last, laughs loudest.\"\n\n...or she for that matter: Don't give up!\n\nIf you spot an error with this article, or have a suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Special thanks to Edward\nfor bringing `mockito` to my attention.\n\nThe diagrams used in this article were produced with the excellent\n[Excalidraw](https://excalidraw.com/).\n\n

fin!

\n\n", + "markdown": "---\ntitle: Mocking With Pytest in Plain English\nauthor: Rich Leyshon\ndate: July 14 2024\ndescription: Plain English Comparison of Mocking Approaches in Python\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - mocking\n - pytest-in-plain-english\n - mockito\n - MagicMock\n - monkeypatch\nimage: 'https://i.imgur.com/K0mxjuF.jpeg'\nimage-alt: Soul singer with Joker makeup.\ntoc: true\ncss: /www/13-pytest-parametrize/styles.css\n---\n\n
\n \n
The Joker sings the Green Green Grass of Home.
\n
\n\n> \"A day without laughter is a day wasted.\" Charlie Chaplin\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses the dark art of mocking, why\nyou should do it and the nuts and bolts of implementing mocked tests. This blog\nis the fourth in a series of blogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n### What does Mocking Mean?\n\nCode often has external dependencies:\n\n* Web APIs (as in this article)\n* Websites (if scraping / crawling)\n* External code (importing packages)\n* Data feeds and databases\n* Environment variables\n\nAs developers cannot control the behaviour of those dependencies, they would\nnot write tests dependent upon them. In order to test their source code that\ndepends on these services, developers need to replace the properties of these\nservices when the test suite runs. Injecting replacement values into the code\nat runtime is generally referred to as mocking. Mocking these values means that\ndevelopers can feed dependable results to their code and make reliable\nassertions about the code's behaviour, without changes in the 'outside world'\naffecting outcomes in the system under test.\n\nDevelopers who write unit tests may also mock their own code. The \"unit\" in the\nterm \"unit test\" implies complete isolation from external dependencies. Mocking\nis an indispensible tool in achieving that isolation within a test suite. It\nensures that code can be efficiently verified in any order, without\ndependencies on other elements in your codebase. However, mocking also adds to\ncode complexity, increasing cognitive load and generally making things harder\nto debug.\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\nThe code may not always be optimal, favouring a simplistic approach wherever\npossible.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python, HTTP requests and some\nfamiliarity with `pytest` and packaging. The type of programmer who has\nwondered about how to follow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1 requests mockito`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[mocking branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking)\nof the repo.\n\n## Overview\n\nMocking is one of the trickier elements of testing. It's a bit niche and is\noften perceived to be too hacky to be worth the effort. The options for mocking\nin python are numerous and this adds to the complexity of many example\nimplementations you will find online. \n\nThere is also a compromise in simplicity versus flexibility. Some of the\noptions available are quite involved and can be adapted to the nichest of\ncases, but may not be the best option for those new to mocking. With this in\nmind, I present 3 alternative methods for mocking python source code. So if\nyou'll forgive me, this is the first of the\n[`pytest` in plain English](/blogs/index.qmd#category=pytest-in-plain-english)\nseries where I introduce alternative testing practices from beyond the `pytest`\npackage.\n\n1. [**monkeypatch**][monkeypatch]: The `pytest` fixture designed for mocking. The\norigin of the fixture's name is debated but potentially arose from the term\n'guerrilla patch' which may have been misinterpreted as 'gorilla patch'. This\nis the concept of modifying source code at runtime, which probably sounds a bit\nlike 'monkeying with the code'.\n2. [**MagicMock**][magicmock]: This is the mocking object provided by python3's\nbuiltin `unittest` package.\n3. [**mockito**][mockito]: This package is based upon the popular Java framework\nof the same name. Despite having a user-friendly syntax, `mockito` is robust\nand secure.\n\n:::{.callout-note collapse=\"true\"}\n\n[monkeypatch]: https://docs.pytest.org/en/stable/how-to/monkeypatch.html\n[magicmock]: https://docs.python.org/3/library/unittest.mock.html\n[mockito]: https://mockito-python.readthedocs.io/en/latest/walk-through.html\n\n\n### A note on the language\n\nMocking has a bunch of synonyms & related language which can be a bit\noff-putting. All of the below terms are associated with mocking. Some may be\npreferred to the communities of specific programming frameworks over others.\n\n| Term | Brief Meaning | Frameworks/Libraries |\n|---------------|---------------|----------------------|\n| Mocking | Creating objects that simulate the behaviour of real objects for testing | Mockito (Java), unittest.mock (Python), Jest (JavaScript), Moq (.NET) |\n| Spying | Observing and recording method calls on real objects | Mockito (Java), Sinon (JavaScript), unittest.mock (Python), RSpec (Ruby) |\n| Stubbing | Replacing methods with predefined behaviours or return values | Sinon (JavaScript), RSpec (Ruby), PHPUnit (PHP), unittest.mock (Python) |\n| Patching | Temporarily modifying or replacing parts of code for testing | unittest.mock (Python), pytest-mock (Python), PowerMock (Java) |\n| Faking | Creating simplified implementations of complex dependencies | Faker (multiple languages), Factory Boy (Python), FactoryGirl (Ruby) |\n| Dummy Objects | Placeholder objects passed around but never actually used | Can be created in any testing framework |\n\n:::\n\n## Mocking in Python\n\nThis section will walk through some code that uses HTTP requests to an external\nservice and how we can go about testing the code's behaviour without relying on\nthat service being available. Feel free to clone the repository and check out\nto the\n[example code](https://github.com/r-leyshon/pytest-fiddly-examples/tree/mocking)\nbranch to run the examples.\n\nThe purpose of the code is to retrieve jokes from \nlike so:\n\n\n\n::: {#8c85323a .cell execution_count=2}\n``` {.python .cell-code}\nfor _ in range(3):\n print(get_joke(f=\"application/json\"))\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nWhat did Yoda say when he saw himself in 4K? \"HDMI\"\nThe other day I was listening to a song about superglue, it’s been stuck in my head ever since.\nWhat do you call a careful wolf? Aware wolf.\n```\n:::\n:::\n\n\n::: {.callout-caution}\n\nThe jokes are provided by and are not curated by\nme. In my testing of the service I have found the jokes to be harmless fun, but\nI cannot guarantee that. If an offensive joke is returned, this is\nunintentional but\n[let me know about it](https://github.com/r-leyshon/blogging/issues) and I will\ngenerate new jokes.\n\n:::\n\n### Define the Source Code\n\nThe function `get_joke()` uses 2 internals:\n\n1. `_query_endpoint()` Used to construct the HTTP request with required headers\nand user agent.\n2. `_handle_response()` Used to catch HTTP errors, or to pull the text out of\nthe various response formats.\n\n::: {#050baf3a .cell execution_count=3}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n \"\"\"Utility for formatting query string & requesting endpoint.\"\"\"\n HEADERS = {\n \"User-Agent\": usr_agent,\n \"Accept\": f,\n }\n resp = requests.get(endp, headers=HEADERS)\n return resp\n```\n:::\n\n\nKeeping separate, the part of the codebase that you wish to target for mocking\nis often the simplest way to go about things. The target for our mocking will\nbe the command that integrates with the external service, so `requests.get()`\nhere. \n\nThe use of `requests.get()` in the code above depends on a few things:\n\n1. An endpoint string.\n2. A dictionary with string values for the keys \"User-Agent\" and \"Accept\".\n\nWe'll need to consider those dependencies when mocking. Once we return a\nresponse from the external service, we need a utility to handle the various\nstatuses of that response:\n\n::: {#f0e8226a .cell execution_count=4}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n ...\n\n\ndef _handle_response(r: requests.models.Response) -> str:\n \"\"\"Utility for handling reponse object & returning text content.\n\n Parameters\n ----------\n r : requests.models.Response\n Response returned from webAPI endpoint.\n\n Raises\n ------\n NotImplementedError\n Requested format `f` was not either 'text/plain' or 'application/json'. \n requests.HTTPError\n HTTP error was encountered.\n \"\"\"\n if r.ok:\n c_type = r.headers[\"Content-Type\"]\n if c_type == \"application/json\":\n content = r.json()\n content = content[\"joke\"]\n elif c_type == \"text/plain\":\n content = r.text\n else:\n raise NotImplementedError(\n \"This client accepts 'application/json' or 'text/plain' format\"\n )\n else:\n raise requests.HTTPError(\n f\"{r.status_code}: {r.reason}\"\n )\n return content\n```\n:::\n\n\nOnce `_query_endpoint()` gets us a response, we can feed it into\n`_handle_response()`, where different logic is executed depending on the\nresponse's properties. Specifically, any response we want to mock would need\nthe following:\n\n1. headers, containing a dictionary eg: `{\"content_type\": \"plain/text\"}`\n2. A `json()` method.\n3. `text`, `status_code` and `reason` attributes.\n\nFinally, the above functions get wrapped in the `get_joke()` function below:\n\n::: {#2251c2ee .cell execution_count=5}\n``` {.python .cell-code}\n\"\"\"Retrieve dad jokes available.\"\"\"\nimport requests\n\n\ndef _query_endpoint(\n endp:str, usr_agent:str, f:str,\n ) -> requests.models.Response:\n ...\n\n\ndef _handle_response(r: requests.models.Response) -> str:\n ...\n\n\ndef get_joke(\n endp:str = \"https://icanhazdadjoke.com/\",\n usr_agent:str = \"datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)\", \n f:str = \"text/plain\",\n) -> str:\n \"\"\"Request a joke from icanhazdadjoke.com.\n\n Ask for a joke in either plain text or JSON format. Return the joke text.\n\n Parameters\n ----------\n endp : str, optional\n Endpoint to query, by default \"https://icanhazdadjoke.com/\"\n usr_agent : str, optional\n User agent value, by default\n \"datasavvycorner.com (https://github.com/r-leyshon/pytest-fiddly-examples)\"\n f : str, optional\n Format to request eg \"application.json\", by default \"text/plain\"\n\n Returns\n -------\n str\n Joke text.\n \"\"\"\n r = _query_endpoint(endp=endp, usr_agent=usr_agent, f=f)\n return _handle_response(r)\n```\n:::\n\n\n### Let's Get Testing\n\nThe behaviour in `get_joke()` is summarised in the flowchart below:\n\n\"get_joke()\n\nThere are 4 outcomes to check, coloured red and green in the process chart\nabove.\n\n1. `get_joke()` successfully returns joke text when the user asked for json\nformat.\n2. `get_joke()` successfully returns joke text when the user asked for plain\ntext.\n3. `get_joke()` raises `NotImplementedError` if any other valid format is asked\nfor. Note that the API also accepts HTML and image formats, though parsing the\njoke text out of those is more involved and beyond the scope of this blog.\n4. `get_joke()` raises a `HTTPError` if the response from the API was not ok.\n\nNote that the event that we wish to target for mocking is highlighted in blue - \nwe don't want our tests to execute any real requests.\n\nThe strategy for testing this function without making requests to the web API\nis composed of 4 similar steps, regardless of the package used to implement the\nmocking. \n\n\"4\n\n1. **Mock:** Define the object or property that you wish to use as a\nreplacement. This could be a static value or something a bit more involved,\nlike a mock class that can return dynamic values depending upon the values it\nreceives. \n2. **Patch:** Replace part of the source code with our mock value. \n3. **Use:** Use the source code to return a value.\n4. **Assert:** Check the returned value is what you expect.\n\nIn the examples that follow, I will label the equivalent steps for the various\nmocking implementations.\n\n### The \"Ultimate Joke\"\n\nWhat hard-coded text shall I use for my expected joke? I'll\n[create a fixture](/blogs/11-fiddly-bits-of-pytest.qmd) that will serve\nup this joke text to all of the test modules used below. I'm only going to\ndefine it once and then refer to it throughout several examples below. So it\nneeds to be a pretty memorable, awesome joke.\n\n::: {#9d9d802b .cell execution_count=6}\n``` {.python .cell-code}\nimport pytest\n\n\n@pytest.fixture(scope=\"session\")\ndef ULTI_JOKE():\n return (\"Doc, I can't stop singing 'The Green, Green Grass of Home.' That \"\n \"sounds like Tom Jones Syndrome. Is it common? Well, It's Not Unusual.\")\n```\n:::\n\n\nBeing a Welshman, I may be a bit biased. But that's a pretty memorable dad joke\nin my opinion. This joke will be available to every test within my test suite\nwhen I execute `pytest` from the command line. The assertions that we will\nuse when using `get_joke()` will expect this string to be returned. If some\nother joke is returned, then we have not mocked correctly and an HTTP request\nwas sent to the API.\n\n### Mocking Everything\n\nI'll start with an example of how to mock `get_joke()` completely. This is an\nintentionally bad idea. In doing this, the test won't actually be executing any\nof the code, just returning a hard-coded value for the joke text. All this does\nis prove that the mocking works as expected and has nothing to do with the\nlogic in our source code. \n\nSo why am I doing it? Hopefully I can illustrate the most basic implementation\nof mocking in this way. I'm not having to think about how I can mock a response\nobject with all the required properties. I just need to provide some hard coded\ntext. \n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#200d6afd .cell execution_count=7}\n``` {.python .cell-code}\nimport example_pkg.only_joking\n\n\ndef test_get_joke_monkeypatched_entirely(monkeypatch, ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1\n def _mock_joke():\n \"\"\"Return the joke text.\n\n monkeypatch.setattr expects the value argument to be callable. In plain\n English, a function or class.\"\"\"\n return ULTI_JOKE\n # step 2\n monkeypatch.setattr(\n target=example_pkg.only_joking,\n name=\"get_joke\",\n value=_mock_joke\n )\n # step 3 & 4\n # Use the module's namespace to correspond with the monkeypatch\n assert example_pkg.only_joking.get_joke() == ULTI_JOKE \n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * Step 2 requires the hard coded text to be returned from a callable, like\n a function or class. So we define `_mock_joke` to serve the text in the\n required format.\n* **Step 2**\n * `monkeypatch.setattr()` is able to take the module namespace that we\n imported as the target. This must be the namespace where the function\n (or variable etc) is defined.\n* **Step 3**\n * When invoking the function, be sure to reference the function in the same\n way as it was monkeypatched.\n * Aliases can also be used if preferable\n (eg `import example_pkg.only_joking as jk`). Be sure to update your\n reference to `get_joke()` in step 2 and 3 to match your import statement.\n\n:::\n\n#### MagicMock\n\n::: {#09afd2d8 .cell execution_count=8}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_magicmocked_entirely(ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1\n _mock_joke = MagicMock(return_value=ULTI_JOKE)\n # step 2\n with patch(\"example_pkg.only_joking.get_joke\", _mock_joke):\n # step 3\n joke = example_pkg.only_joking.get_joke()\n # step 4\n assert joke == ULTI_JOKE\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * `MagicMock()` allows us to return static values as mock objects.\n* **Step 3**\n * When you use `get_joke()`, be sure to call reference the namespace in the\n same way as to your patch in step 2.\n\n:::\n\n#### mockito\n\n::: {#fdfe4527 .cell execution_count=9}\n``` {.python .cell-code}\nfrom mockito import when, unstub\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_mockitoed_entirely(ULTI_JOKE):\n \"\"\"Completely replace the entire get_joke return value.\n\n Not a good idea for testing as none of our source code will be tested. But\n this demonstrates how to entirely scrub a function and replace with any\n placeholder value at pytest runtime.\"\"\"\n # step 1 & 2\n when(example_pkg.only_joking).get_joke().thenReturn(ULTI_JOKE)\n # step 3\n joke = example_pkg.only_joking.get_joke()\n # step 4\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1 & 2**\n * `mockito`'s intuitive `when(...).thenReturn(...)` pattern allows you\n to reference any object within the imported namespace. Like with\n `MagicMock`, the static string `ULTI_JOKE` can be referenced.\n* **Step 3**\n * When you use `get_joke()`, be sure to call reference the namespace in the\n same way as to your patch in step 2.\n* **unstub**\n * This step explicitly 'unpatches' `get_joke()`. If you did not `unstub()`,\n the patch to `get_joke()` would persist through the rest of your tests.\n * `mockito` allows you to implicitly `unstub()` by using the context\n manager `with`.\n\n:::\n\n:::\n\n### `monkeypatch()` without OOP\n\nSomething I've noticed about the `pytest` documentation for `monkeypatch`, is\nthat it gets straight into mocking with Object Oriented Programming (OOP).\nWhile this may be a bit more convenient, it is certainly not a requirement of\nusing `monkeypatch` and definitely adds to the cognitive load for new users.\nThis first example will mock the value of `requests.get` without using classes.\n\n::: {#c9ec183d .cell execution_count=10}\n``` {.python .cell-code}\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_monkeypatched_no_OOP(monkeypatch, ULTI_JOKE):\n # step 1: Mock the response object\n def _mock_response(*args, **kwargs):\n resp = requests.models.Response()\n resp.status_code = 200\n resp._content = ULTI_JOKE.encode(\"UTF8\")\n resp.headers = {\"Content-Type\": \"text/plain\"}\n return resp\n \n # step 2: Patch requests.get\n monkeypatch.setattr(requests, \"get\", _mock_response)\n # step 3: Use requests.get\n joke = get_joke()\n # step 4: Assert\n assert joke == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{joke}'\"\n # will also work for json format\n joke = get_joke(f=\"application/json\")\n assert joke == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{joke}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * The return value of `requests.get()` will be a response object. We need\n to mock this object with the methods and attributes required by the\n `_handle_response()` function. \n * We need to encode the static joke text as bytes to format the data. \n Response objects encode data as bytes for interoperatability and\n optimisation purposes.\n* **step4**\n * As we have set an appropriate value for the mocked response's `_content`\n attribute, the mocked joke will be returned for both JSON and plain text\n formats - very convenient!\n\n:::\n\n### Condition 1: Test JSON\n\nIn this example, we demonstrate the same functionality as above, but with an\nobject-oriented design pattern. This approach more closely follows that of\nthe `pytest` documentation. As before, `MagicMock` and `mockito` examples will\nbe included. \n\nThe purpose of this test is to test the outcome of `get_joke()` when the user\nspecifies a json format.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#monkey-fixture .cell execution_count=11}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"Return a class instance that will mock all the properties of a response\n object that get_joke needs to work.\n \"\"\"\n HEADERS_MAP = {\n \"text/plain\": {\"Content-Type\": \"text/plain\"},\n \"application/json\": {\"Content-Type\": \"application/json\"},\n \"text/html\": {\"Content-Type\": \"text/html\"},\n }\n\n class MockResponse:\n def __init__(self, f, *args, **kwargs):\n self.ok = True\n self.f = f\n self.headers = HEADERS_MAP[f] # header corresponds to format that\n # the user requested\n self.text = ULTI_JOKE \n\n def json(self):\n if self.f == \"application/json\":\n return {\"joke\": ULTI_JOKE}\n return None\n\n return MockResponse\n\n\ndef test_get_joke_json_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\n\n Test get_joke using the mock class fixture. This approach is the\n implementation suggested in the pytest docs.\n \"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n \"\"\"Return fixtures with the correct header.\n\n If the test uses \"text/plain\" format, we need to return a MockResponse\n class instance with headers attribute equal to\n {\"Content-Type\": \"text/plain\"}, likewise for JSON.\n \"\"\"\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # Step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # Step 3: Use\n j_json = get_joke(f=\"application/json\")\n # Step 4: Assert\n assert j_json == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{j_json}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We define a mocked class instance with the necessary properties expected\n by `_handle_response()`.\n * The mocked response is served to our test as a `pytest` fixture.\n * Within the test, we need another function, which will be able to take\n the arguments passed to `requests.get()`. This will allow our class\n instance to retrieve the appropriate header from the `HEADERS_MAP`\n dictionary.\n\nAs you may appreciate, this does not appear to be the most straight forward\nimplementation, but it will allow us to test when the user asks for JSON,\nplain text or HTML formats. In the above test, we assert against JSON format\nonly.\n\n:::\n\n#### MagicMock\n\n::: {#6a4a3c0c .cell execution_count=12}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_json_magicmocked(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"application/json\"}\n mock_response.json.return_value = {\"joke\": ULTI_JOKE}\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3: Use\n joke = get_joke(f=\"application/json\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * `MagicMock()` can return a mock object with a specification designed to\n mock response objects. Super useful.\n * Our static joke content can be served directly to `MagicMock` without the\n need for an intermediate class.\n * In comparison to the `monkeypatch` approach, this appears to be more\n straight forward and maintainable.\n:::\n\n#### mockito\n\n::: {#33d981eb .cell execution_count=13}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_json_mockitoed(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for JSON joke.\"\"\"\n # step 1: Mock\n _mock_response = requests.models.Response()\n _mock_response.status_code = 200\n _mock_response._content = b'{\"joke\": \"' + ULTI_JOKE.encode(\"utf-8\") + b'\"}'\n _mock_response.headers = {\"Content-Type\": \"application/json\"}\n # step 2: Patch\n when(requests).get(...).thenReturn(_mock_response)\n # step 3: Use\n joke = example_pkg.only_joking.get_joke(f=\"application/json\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * In order to encode the expected joke for JSON format, we need a\n dictionary encoded within a bytestring. This bit is a little tricky.\n * Alternatively, define the expected dictionary and use the `json`\n package. `json.dumps(dict).encode(\"UTF8\")` will format the content\n dictionary in the required way.\n* **Step 2**\n * `mockito`'s `when()` approach will allow you to access the methods of the\n object that is being patched, in this case `requests`. \n * `mockito` allows you to pass the `...` argument to a patched method, to\n indicate that whatever arguments were passed to `get()`, return the\n specified mock value.\n * Being able to specify values passed in place of `...` will allow you to\n set different return values depending on argument values received by\n `get()`.\n:::\n\n:::\n\n### Condition 2: Test Plain Text\n\nThe purpose of this test is to check the outcome when the user specifies a\nplain/text format while using `get_joke()`.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#90bcbef8 .cell execution_count=14}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"The same fixture as was used for testing JSON format\"\"\"\n ...\n\n\ndef test_get_joke_text_monkeypatched(monkeypatch, _mock_response, ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # step 3: Use\n j_txt = get_joke(f=\"text/plain\")\n # step 4: Assert\n assert j_txt == ULTI_JOKE, f\"Expected:\\n'{ULTI_JOKE}\\nFound:\\n{j_txt}'\"\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We can use the same mock class as for testing\n [Condition 1](#monkey-fixture), due to the content of the `HEADERS_MAP`\n dictionary.\n:::\n\n#### MagicMock\n\n::: {#6aebf8d7 .cell execution_count=15}\n``` {.python .cell-code}\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_text_magicmocked(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"text/plain\"}\n mock_response.text = ULTI_JOKE\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3: Use\n joke = get_joke(f=\"text/plain\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n```\n:::\n\n\n#### mockito\n\n::: {#d07f1a91 .cell execution_count=16}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\ndef test_get_joke_text_mockitoed(ULTI_JOKE):\n \"\"\"Test behaviour when user asked for plain text joke.\"\"\"\n # step 1: Mock\n mock_response = requests.models.Response()\n mock_response.status_code = 200\n mock_response._content = ULTI_JOKE.encode(\"utf-8\")\n mock_response.headers = {\"Content-Type\": \"text/plain\"}\n # step 2: Patch\n when(requests).get(...).thenReturn(mock_response)\n # step 3: Use\n joke = example_pkg.only_joking.get_joke(f=\"text/plain\")\n # step 4: Assert\n assert joke == ULTI_JOKE\n unstub()\n```\n:::\n\n\n:::\n\n### Condition 3: Test Not Implemented\n\nThis test will check the outcome of what happens when the user asks for a\nformat other than text or JSON format. As the webAPI also offers image or HTML\nformats, a response 200 (ok) would be returned from the service. But I was too\nbusy (lazy) to extract the text from those formats. \n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#52bb65a3 .cell execution_count=17}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\n@pytest.fixture\ndef _mock_response(ULTI_JOKE):\n \"\"\"The same fixture as was used for testing JSON format\"\"\"\n ...\n\n\ndef test_get_joke_not_implemented_monkeypatched(\n monkeypatch, _mock_response):\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n def _mock_get_good_resp(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_good_resp)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"This client accepts 'application/json' or 'text/plain' format\"):\n get_joke(f=\"text/html\")\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * We can use the same mock class as for testing\n [Condition 1](#monkey-fixture), due to the content of the `HEADERS_MAP`\n dictionary.\n* **Step 4**\n * We use a context manager (`with pytest.raises`) which catches the raised\n exception and stops it from terminating our `pytest` session. \n * The asserted `match` argument can take a regular expression, so that\n wildcard patterns can be used. This allows matching of part of the\n exception message.\n:::\n\n#### MagicMock\n\n::: {#bdc93567 .cell execution_count=18}\n``` {.python .cell-code}\nimport pytest\nimport requests\nfrom unittest.mock import MagicMock, patch\n\nfrom example_pkg.only_joking import get_joke\ndef test__handle_response_not_implemented_magicmocked():\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n mock_response = MagicMock(spec=requests.models.Response)\n mock_response.ok = True\n mock_response.headers = {\"Content-Type\": \"text/html\"}\n # step 2: Patch\n with patch(\"requests.get\", return_value=mock_response):\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"client accepts 'application/json' or 'text/plain' format\"):\n get_joke(f=\"text/html\")\n```\n:::\n\n\n#### mockito\n\n::: {#f6f4646a .cell execution_count=19}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\ndef test_get_joke_not_implemented_mockitoed():\n \"\"\"Test behaviour when user asked for HTML response.\"\"\"\n # step 1: Mock\n mock_response = requests.models.Response()\n mock_response.status_code = 200\n mock_response.headers = {\"Content-Type\": \"text/html\"}\n # step 2: Patch\n when(\n example_pkg.only_joking\n )._query_endpoint(...).thenReturn(mock_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(\n NotImplementedError,\n match=\"This client accepts 'application/json' or 'text/plain' format\"):\n example_pkg.only_joking.get_joke(f=\"text/html\")\n unstub()\n```\n:::\n\n\n:::\n\n### Condition 4: Test Bad Response\n\nIn this test, we simulate a bad response from the webAPI, which could arise\nfor a number of reasons:\n\n* The api is unavailable.\n* The request asked for a resource that is not available.\n* Too many requests were made in a short period.\n\nThese conditions are those that we have the least control over and therefore\nhave the greatest need for mocking.\n\n:::{.panel-tabset}\n\n#### monkeypatch\n\n::: {#b873cabd .cell execution_count=20}\n``` {.python .cell-code}\nimport pytest\nimport requests\n\nfrom example_pkg.only_joking import get_joke, _handle_response\n\n\n@pytest.fixture\ndef _mock_bad_response():\n class MockBadResponse:\n def __init__(self, *args, **kwargs):\n self.ok = False\n self.status_code = 404\n self.reason = \"Not Found\"\n return MockBadResponse\n\n\ndef test_get_joke_http_error_monkeypatched(\n monkeypatch, _mock_bad_response):\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n def _mock_get_bad_response(*args, **kwargs):\n f = kwargs[\"headers\"][\"Accept\"]\n return _mock_bad_response(f)\n # step 2: Patch\n monkeypatch.setattr(requests, \"get\", _mock_get_bad_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n get_joke()\n```\n:::\n\n\n:::{.callout-note}\n\n##### Notes\n\n* **Step 1**\n * This time we need to define a new fixture that returns a bad response.\n * Alternatively, we could have implemented a single fixture for all of our\n tests that dynamically served a good or bad response dependent upon\n arguments passed to `get_joke()`, for example different string values\n passed as the endpoint.\n * In a more thorough implementation of `get_joke()`, you may wish to retry\n the request for certain HTTP error status codes. The ability to provide\n mocked objects that reliably serve those statuses allow you to\n deterministically validate your code's behaviour.\n\n:::\n\n#### MagicMock\n\n::: {#a1838e9a .cell execution_count=21}\n``` {.python .cell-code}\nimport pytest\nfrom unittest.mock import MagicMock, patch\nimport requests\n\nfrom example_pkg.only_joking import get_joke\n\n\ndef test_get_joke_http_error_magicmocked():\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n _mock_response = MagicMock(spec=requests.models.Response)\n _mock_response.ok = False\n _mock_response.status_code = 404\n _mock_response.reason = \"Not Found\"\n # step 2: Patch\n with patch(\"requests.get\", return_value=_mock_response):\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n get_joke()\n```\n:::\n\n\n#### mockito\n\n::: {#e0779f13 .cell execution_count=22}\n``` {.python .cell-code}\nfrom mockito import when, unstub\nimport requests\n\nimport example_pkg.only_joking\n\n\ndef test_get_joke_http_error_mockitoed():\n \"\"\"Test bad HTTP response.\"\"\"\n # step 1: Mock\n _mock_response = requests.models.Response()\n _mock_response.status_code = 404\n _mock_response.reason = \"Not Found\"\n # step 2: Patch\n when(example_pkg.only_joking)._query_endpoint(...).thenReturn(\n _mock_response)\n # step 3 & 4 Use (try to but exception is raised) & Assert\n with pytest.raises(requests.HTTPError, match=\"404: Not Found\"):\n example_pkg.only_joking.get_joke()\n unstub()\n```\n:::\n\n\n:::\n\n\n## Summary\n\nWe have thoroughly tested our code using approaches that mock the behaviour of\nan external webAPI. We have also seen how to implement those tests with 3\ndifferent packages.\n\nI hope that this has provided you with enough introductory material to begin\nmocking tests if you have not done so before. If you find that your specific\nuse case for mocking is quite nuanced and fiddly (it's likely to be that way),\nthen the alternative implementations presented here can help you to understand\nhow to solve your specific mocking dilemma. \n\nOne final quote for those developers having their patience tested by errors\nattempting to implement mocking:\n\n> \"He who laughs last, laughs loudest.\"\n\n...or she for that matter: Don't give up!\n\nIf you spot an error with this article, or have a suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Special thanks to Edward\nfor bringing `mockito` to my attention.\n\nThe diagrams used in this article were produced with the excellent\n[Excalidraw](https://excalidraw.com/).\n\n

fin!

\n\n", "supporting": [ - "15-pytest-mocking_files" + "15-pytest-mocking_files/figure-html" ], "filters": [], "includes": {} diff --git a/_freeze/blogs/16-pytest-marks/execute-results/html.json b/_freeze/blogs/16-pytest-marks/execute-results/html.json new file mode 100644 index 0000000..b9984bc --- /dev/null +++ b/_freeze/blogs/16-pytest-marks/execute-results/html.json @@ -0,0 +1,12 @@ +{ + "hash": "9c9ae1a9ef6ef4d2987415ff5621d140", + "result": { + "engine": "jupyter", + "markdown": "---\ntitle: Custom Marks With Pytest in Plain English\nauthor: Rich Leyshon\ndate: July 22 2024\ndescription: Selectively running tests with marks\ncategories:\n - Explanation\n - pytest\n - Unit tests\n - marks\n - custom marks\n - markers\n - pytest-in-plain-english\nimage: 'https://i.imgur.com/dCJBh9w.jpeg'\nimage-alt: Planet composed of croissants\ntoc: true\ntoc-depth: 4\ncss: /www/13-pytest-parametrize/styles.css\n---\n\n
\n \n
Planet composed of croissants.
\n
\n\n> \"The only flake I want is that of a croissant.\" Mariel Feldman.\n\n## Introduction\n\n`pytest` is a testing package for the python framework. It is broadly used to\nquality assure code logic. This article discusses custom marks, use cases and\npatterns for selecting and deselecting marked tests. This blog is the fifth and\nfinal in a series of blogs called\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english),\nfavouring accessible language and simple examples to explain the more intricate\nfeatures of the `pytest` package.\n\nFor a wealth of documentation, guides and how-tos, please consult the\n`pytest` documentation.\n\n### What are `pytest` Custom Marks?\n\nMarks are a way to conditionally run specific tests. There are a few marks that\ncome with the `pytest` package. To view these, run `pytest --markers` from the\ncommand line. This will print a list of the pre-registered marks available\nwithin the `pytest` package. However, it is extremely easy to register your own\nmarkers, allowing greater control over which tests get executed.\n\nThis article will cover:\n\n* Reasons for marking tests\n* Registering marks with `pytest`\n* Marking tests\n* Including or excluding markers from the command line\n\n:::{.callout collapse=\"true\"}\n\n### A Note on the Purpose (Click to expand)\n\nThis article intends to discuss clearly. It doesn't aim to be clever or\nimpressive. Its aim is to extend understanding without overwhelming the reader.\nThe code may not always be optimal, favouring a simplistic approach wherever\npossible.\n\n:::\n\n### Intended Audience\n\nProgrammers with a working knowledge of python and some familiarity with\n`pytest` and packaging. The type of programmer who has wondered about how to\nfollow best practice in testing python code.\n\n### What You'll Need:\n\n- [ ] Preferred python environment manager (eg `conda`)\n- [ ] `pip install pytest==8.1.1`\n- [ ] Git\n- [ ] GitHub account\n- [ ] Command line access\n\n### Preparation\n\nThis blog is accompanied by code in\n[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The\nmain branch provides a template with the minimum structure and requirements\nexpected to run a `pytest` suite. The repo branches contain the code used in\nthe examples of the following sections.\n\nFeel free to fork or clone the repo and checkout to the example branches as\nneeded.\n\nThe example code that accompanies this article is available in the\n[marks branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks)\nof the repo.\n\n## Overview\n\nOccasionally, we write tests that are a bit distinct to the rest of our test\nsuite. They could be integration tests, calling on elements of our code from\nmultiple modules. They could be end to end tests, executing a pipeline from\nstart to finish. Or they could be a flaky or brittle sort of test, a test that\nis prone to failure on specific operating systems, architectures or when\nexternal dependencies do not provide reliable inputs. \n\nThere are multiple ways to handle these kinds of tests, including mocking, as\ndiscussed in [my previous blog](/blogs/15-pytest-mocking.qmd). Mocking can\noften take a bit of time, and developers don't always have that precious\ncommodity. So instead, they may mark the test, ensuring that it doesn't get run\non continuous integration (CI) checks. This may involve flagging any flaky test\nas \"technical debt\" to be investigated and fixed later.\n\nIn fact, there are a number of reasons that we may want to selectively run\nelements of a test suite. Here is a selection of scenarios that could benefit\nfrom marking.\n\n:::{.callout-note collapse=\"true\"}\n\n### Flaky Tests: Common Causes\n\n| **Category** | **Cause** | **Explanation** |\n|-----------------------------|-----------------------------|-------------------------------------------------------------------------|\n| External Dependencies | Network | Network latency, outages, or Domain Name System (DNS) issues. |\n| | Web APIs | Breaking changes, rate limits, or outages. |\n| | Databases | Concurrency issues, data changes, or connection problems. |\n| | Timeouts | Hardcoded or too-short timeouts cause failures. |\n| Environment Dependencies | Environment Variables | Incorrectly set environment variables. |\n| | File System | File locks, permissions, or missing files. |\n| | Resource Limits | Insufficient CPU, memory, or disk space. |\n| State Dependencies | Shared State | Interference between tests sharing state. |\n| | Order Dependency | Tests relying on execution order. |\n| Test Data | Random Data | Different results on each run due to random data and seed not set. |\n| Concurrency Issues | Parallel Execution | Tests not designed for parallel execution. |\n| | Locks | Deadlocks or timeouts involving locks or semaphores. |\n| | Race Conditions | Tests depend on the order of execution of threads or processes. |\n| | Async Operations | Improperly awaited asynchronous code. |\n| Hardware and System Issues | Differences in Hardware | Variations in performance across hardware or operating systems. |\n| | System Load | Failures under high system load due to resource contention. |\n| Non-deterministic Inputs | Time | Variations in current time affecting test results. |\n| | User Input | Non-deterministic user input causing flaky behaviour. |\n| | Filepaths | CI runner filepaths may be hard to predict. |\n| Test Implementation Issues | Assertions | Incorrect or overly strict assertions. |\n| | Setup and Teardown | Inconsistent state due to improper setup or teardown. |\n: {.light .hover .responsive-md}\n\n:::\n\nIn the case of integration tests, one approach may be to group them all\ntogether and have them execute within a dedicated CI workflow. This is common\npractice as developers may want to stay alert to problems with external\nresources that their code depends upon, while not failing 'core' CI checks\nabout changes to the source code. If your code relies on a web API; for\ninstance; you're probably less concerned about temporary outages in that\nservice. However, a breaking change to that service would require our source\ncode to be adapted. Once more, life is a compromise.\n\n> \"*Le mieux est l'ennemi du bien.*\" (The best is the enemy of the good),\nVoltaire\n\n## Custom Marks in `pytest`\n\nMarking allows us to have greater control over which of our tests are executed\nwhen we invoke `pytest`. Marking is conveniently implemented in the following\nway (presuming you have already written your source and test code):\n\n1. Register a custom marker\n2. Assign the new marker name to the target test\n3. Invoke `pytest` with the `-m` (MARKEXPR) flag.\n\nThis section uses code available in the\n[marks branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks)\nof the GitHub repository.\n\n### Define the Source Code\n\nI have a motley crew of functions to consider. A sort of homage to Sergio\nLeone's 'The Good, The Bad & the Ugly', although I'll let you figure out which\nis which.\n\n
\n \n
The Flaky, The Slow & The Needy
\n
\n\n\n#### The Flaky Function\n\nHere we define a function that will fail half the time. What a terrible test to\nhave. The root of this unpredictable behaviour should be diagnosed as a\npriority as a matter of sanity.\n\n::: {#e4677b5f .cell execution_count=1}\n``` {.python .cell-code}\nimport random\n\n\ndef croissant():\n \"\"\"A very flaky function.\"\"\"\n if round(random.uniform(0, 1)) == 1:\n return True\n else:\n raise Exception(\"Flaky test detected!\")\n```\n:::\n\n\n#### The Slow Function\n\nThis function is going to be pretty slow. Slow test suites throttle our\nproductivity. Once it finishes waiting for a specified number of seconds, it\nwill return a string.\n\n::: {#e736f07b .cell execution_count=2}\n``` {.python .cell-code}\nimport time\nfrom typing import Union\n\n\ndef take_a_nap(how_many_seconds:Union[int, float]) -> str:\n \"\"\"Mimic a costly function by just doing nothing for a specified time.\"\"\"\n time.sleep(float(how_many_seconds))\n return \"Rise and shine!\"\n```\n:::\n\n\n#### The Needy Function\n\nFinally, the needy function will have an external dependency on a website. This\ntest will simply check whether we get a HTTP status code of 200 (ok) when we\nrequest any URL. \n\n::: {#39c60cc1 .cell execution_count=3}\n``` {.python .cell-code}\nimport requests\n\n\ndef check_site_available(url:str, timeout:int=5) -> bool:\n \"\"\"Checks if a site is available.\"\"\"\n try:\n response = requests.get(url, timeout=timeout)\n return True if response.status_code == 200 else False\n except requests.RequestException:\n return False\n```\n:::\n\n\n#### The Wrapper\n\nFinally, I'll introduce a wrapper that will act as an opportunity for an\nintegration test. This is a bit awkward, as none of the above functions are\nparticularly related to each other. \n\nThis function will execute the `check_site_available()` and `take_a_nap()`\ntogether. A pretty goofy example, I admit. Based on the status of the url\nrequest, a string will be returned.\n\n::: {#712d9659 .cell execution_count=4}\n``` {.python .cell-code}\nimport time\nfrom typing import Union\n\nimport requests\n\ndef goofy_wrapper(url:str, timeout:int=5) -> str:\n \"\"\"Check a site is available, pause for no good reason before summarising\n outcome with a string.\"\"\"\n msg = f\"Napping for {timeout} seconds.\\n\"\n msg = msg + take_a_nap(timeout)\n if check_site_available(url):\n msg = msg + \"\\nYour site is up!\"\n else:\n msg = msg + \"\\nYour site is down!\"\n\n return msg\n```\n:::\n\n\n### Let's Get Testing\n\nInitially, I will define a test that does nothing other than pass. This will\nbe a placeholder, unmarked test. \n\n::: {#d6766c37 .cell execution_count=5}\n``` {.python .cell-code}\ndef test_nothing():\n pass\n```\n:::\n\n\nNext, I import `croissant()` and assert that it returns `True`. As you may\nrecall from above, `croissant()` will do so ~50 % of the time.\n\n::: {#29189e47 .cell execution_count=6}\n``` {.python .cell-code}\nfrom example_pkg.do_something import (\n croissant,\n )\n\n\ndef test_nothing():\n pass\n\n\ndef test_croissant():\n assert croissant()\n```\n:::\n\n\nNow running `pytest -v` will print the test results, reporting test outcomes\nfor each test separately (`-v` means verbose).\n\n```{.abc}\n...% pytest -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 2 items \ntests/test_do_something.py::test_nothing PASSED [ 50%]\ntests/test_do_something.py::test_croissant PASSED [100%]\n============================== 2 passed in 0.05s ==============================\n\n```\n\nBut note that half the time, I will also get the following output:\n\n```{.abc}\n...% pytest -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 2 items \n\ntests/test_do_something.py::test_nothing PASSED [ 50%]\ntests/test_do_something.py::test_croissant FAILED [100%]\n\n================================== FAILURES ==================================\n_______________________________ test_croissant ________________________________\n @pytest.mark.flaky\n def test_croissant():\n> assert croissant()\ntests/test_do_something.py:17: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n def croissant():\n \"\"\"A very flaky function.\"\"\"\n if round(random.uniform(0, 1)) == 1:\n return True\n else:\n> raise Exception(\"Flaky test detected!\")\nE Exception: Flaky test detected!\n\nsrc/example_pkg/do_something.py:13: Exception\n============================short test summary info ===========================\nFAILED ...::test_croissant - Exception: Flaky test detected!\n========================= 1 failed, 1 passed in 0.07s =========================\n```\n\nTo prevent this flaky test from failing the test suite, we can choose to mark\nit as flaky, and optionally skip it when invoking `pytest`. To go about that,\nwe first need to register a new marker. To do that, let's update out project's\n`pyproject.toml` to include additional options for a `flaky` mark:\n\n```{.abc}\n# `pytest` configurations\n[tool.pytest.ini_options]\nmarkers = [\n \"flaky: tests that can randomly fail through no change to the code\",\n]\n```\n\nNote that when registering a marker in this way, text after the colon is an\noptional mark description. Saving the document and running `pytest --markers`\nshould show that a new custom marker is available to our project:\n\n```{.abc}\n... % pytest --markers\n@pytest.mark.flaky: tests that can randomly fail through no change to the code\n...\n\n```\nNow that we have confirmed our marker is available for use, we can use it to\nmark `test_croissant()` as flaky:\n\n::: {#0f3c1eeb .cell execution_count=7}\n``` {.python .cell-code}\nimport pytest\n\nfrom example_pkg.do_something import (\n croissant,\n )\n\n\ndef test_nothing():\n pass\n\n\n@pytest.mark.flaky\ndef test_croissant():\n assert croissant()\n```\n:::\n\n\nNote that we need to import `pytest` to our test module in order to use the\n`pytest.mark.` decorator. \n\n#### Selecting a Single Mark\n\nNow that we have registered and marked a test as `flaky`, we can adapt our\n`pytest` call to execute tests with that mark only. The pattern we will use is:\n\n> `pytest -k -m \"\"`\n\n```{.abc}\n... % pytest -v -m \"flaky\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 2 items / 1 deselected / 1 selected \n\ntests/test_do_something.py::test_croissant PASSED [100%]\n\n======================= 1 passed, 1 deselected in 0.05s =======================\n\n```\n\nNow we see that `test_croissant()` was executed, while the unmarked\n`test_nothing()` was not. \n\n#### Deselecting a Single Mark\n\nMore useful than selectively running a flaky test is to deselect it. In this\nway, it cannot fail our test suite. This is achieved with the following\npattern:\n\n> `pytest -v -m \"not \"`\n\n\n```{.abc}\n... % pytest -v -m \"not flaky\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 2 items / 1 deselected / 1 selected \n\ntests/test_do_something.py::test_nothing PASSED [100%]\n\n======================= 1 passed, 1 deselected in 0.05s =======================\n\n```\n\nNote that this time, `test_flaky()` was not executed. \n\n#### Selecting Multiple Marks\n\nIn this section, we will introduce another, differently marked test to\nillustrate the syntax for running multiple marks. For this example, we'll test\n`take_a_nap()`:\n\n::: {#b75c4842 .cell execution_count=8}\n``` {.python .cell-code}\nimport pytest\n\nfrom example_pkg.do_something import (\n croissant,\n take_a_nap,\n )\n\n\ndef test_nothing():\n pass\n\n\n@pytest.mark.flaky\ndef test_croissant():\n assert croissant()\n\n\n@pytest.mark.slow\ndef test_take_a_nap():\n out = take_a_nap(how_many_seconds=3)\n assert isinstance(out, str), f\"a string was not returned: {type(out)}\"\n assert out == \"Rise and shine!\", f\"unexpected string pattern: {out}\"\n```\n:::\n\n\nOur new test just makes some simple assertions about the string `take_a_nap()`\nreturns after snoozing. But notice what happens when running `pytest -v` now:\n\n```{.abc}\n... % pytest -v\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 3 items \n\ntests/test_do_something.py::test_nothing PASSED [ 33%]\ntests/test_do_something.py::test_croissant PASSED [ 66%]\ntests/test_do_something.py::test_take_a_nap PASSED [100%]\n\n============================== 3 passed in 3.07s ==============================\n\n``` \n\nThe test suite now takes in excess of 3 seconds to execute, as the test\nspecified for `take_a_nap()` to sleep for that period. Let's update our\n`pyproject.toml` and register a new mark:\n\n```{.abc}\n# `pytest` configurations\n[tool.pytest.ini_options]\nmarkers = [\n \"flaky: tests that can randomly fail through no change to the code\",\n \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n]\n```\n\nNote that the nested speech marks within the description of the `slow` mark\nwere escaped. `pytest` would have complained that the toml file was not valid\nunless we ensured it was valid [toml syntax](https://toml.io/en/).\n\nIn order to run tests marked with either `flaky` or `slow`, we can use `or`:\n\n> `pytest -v -m \" or \"`\n\n```{.abc}\n... % pytest -v -m \"flaky or slow\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 3 items / 1 deselected / 2 selected \n\ntests/test_do_something.py::test_croissant PASSED [ 50%]\ntests/test_do_something.py::test_take_a_nap PASSED [100%]\n\n======================= 2 passed, 1 deselected in 3.06s =======================\n\n```\n\nNote that anything not marked with `flaky` or `slow` (eg `test_nothing()`) was\nnot run. Also, `test_croissant()` failed 3 times in a row while I tried to get\na passing run. I didn't want the flaky exception to carry on presenting itself.\nWhile I may be sprinkling glitter, I do not want to misrepresent how\nfrustrating flaky tests can be!\n\n
Glitter GIFfrom Glitter GIFs
\n\n#### Complex Selection Rules\n\nBy adding an additional mark, we can illustrate more complex selection and\ndeselection rules for invoking `pytest`. Let's write an integration test that\nchecks whether the domain for this blog site can be reached.\n\n::: {#77c0b75a .cell execution_count=9}\n``` {.python .cell-code}\nimport pytest\n\nfrom example_pkg.do_something import (\n croissant,\n take_a_nap,\n check_site_available,\n )\n\n\ndef test_nothing():\n pass\n\n\n@pytest.mark.flaky\ndef test_croissant():\n assert croissant()\n\n\n@pytest.mark.slow\ndef test_take_a_nap():\n out = take_a_nap(how_many_seconds=3)\n assert isinstance(out, str), f\"a string was not returned: {type(out)}\"\n assert out == \"Rise and shine!\", f\"unexpected string pattern: {out}\"\n\n\n@pytest.mark.integration\ndef test_check_site_available():\n url = \"https://thedatasavvycorner.com/\"\n assert check_site_available(url), f\"site {url} is down...\"\n```\n:::\n\n\nNow updating our `pyproject.toml` like so:\n\n```{.abc}\n# `pytest` configurations\n[tool.pytest.ini_options]\nmarkers = [\n \"flaky: tests that can randomly fail through no change to the code\",\n \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n \"integration: tests that require external resources\",\n]\n```\n\nNow we can combine `and` and `not` statements when calling `pytest` to execute\njust the tests we need to. In the below, I choose to run the `slow` and\n`integration` tests while excluding that pesky `flaky` test.\n\n```{.abc}\n... % pytest -v -m \"slow or integration and not flaky\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 4 items / 2 deselected / 2 selected \n\ntests/test_do_something.py::test_take_a_nap PASSED [ 50%]\ntests/test_do_something.py::test_check_site_available PASSED [100%]\n\n======================= 2 passed, 2 deselected in 3.29s =======================\n```\n\nNote that both `test_nothing()` (unmarked) and `test_croissant()` (deselected)\nwere not run.\n\n#### Marks and Test Classes\n\nNote that so far, we have applied marks to test functions only. But we can also\napply marks to an entire test class, or even target specific test modules. For\nthis section, I will introduce the wrapper function introduced earlier and use\na test class to group its tests together. I will mark those tests with 2 new\nmarks, `classy` and `subclassy`.\n\n```{.abc}\n# `pytest` configurations\n[tool.pytest.ini_options]\nmarkers = [\n \"flaky: tests that can randomly fail through no change to the code\",\n \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n \"integration: tests that require external resources\",\n \"classy: tests arranged in a class\",\n \"subclassy: test methods\",\n]\n```\n\nUpdating our test module to include `test_goofy_wrapper()`:\n\n::: {#db1a7f0a .cell execution_count=10}\n``` {.python .cell-code}\nimport pytest\n\nfrom example_pkg.do_something import (\n croissant,\n take_a_nap,\n check_site_available,\n goofy_wrapper\n )\n\n\ndef test_nothing():\n pass\n\n\n@pytest.mark.flaky\ndef test_croissant():\n assert croissant()\n\n\n@pytest.mark.slow\ndef test_take_a_nap():\n out = take_a_nap(how_many_seconds=3)\n assert isinstance(out, str), f\"a string was not returned: {type(out)}\"\n assert out == \"Rise and shine!\", f\"unexpected string pattern: {out}\"\n\n\n@pytest.mark.integration\ndef test_check_site_available():\n url = \"https://thedatasavvycorner.com/\"\n assert check_site_available(url), f\"site {url} is down...\"\n\n\n@pytest.mark.classy\nclass TestGoofyWrapper:\n @pytest.mark.subclassy\n def test_goofy_wrapper_url_exists(self):\n assert goofy_wrapper(\n \"https://thedatasavvycorner.com/\", 1\n ).endswith(\"Your site is up!\"), \"The site wasn't up.\"\n @pytest.mark.subclassy\n def test_goofy_wrapper_url_does_not_exist(self):\n assert goofy_wrapper(\n \"https://thegoofycorner.com/\", 1\n ).endswith(\"Your site is down!\"), \"The site wasn't down.\"\n```\n:::\n\n\nNote that targeting either the `classy` or `subclassy` mark results in the\nsame output - all tests within this test class are executed:\n\n```{.abc}\n... % pytest -v -m \"classy\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 6 items / 4 deselected / 2 selected \n\nTestGoofyWrapper::test_goofy_wrapper_url_exists PASSED [ 50%]\nTestGoofyWrapper::test_goofy_wrapper_url_does_not_exist PASSED [100%]\n\n======================= 2 passed, 4 deselected in 2.30s =======================\n\n```\n\nNobody created the domain `https://thegoofycorner.com/` yet, such a shame.\n\n#### Tests with Multiple Marks\n\nNote that we can use multiple marks with any test or test class. Let's update\n`TestGoofyWrapper` to be marked as `integration` & `slow`:\n\n::: {#fc008b5f .cell execution_count=11}\n``` {.python .cell-code}\n@pytest.mark.slow\n@pytest.mark.integration\n@pytest.mark.classy\nclass TestGoofyWrapper:\n @pytest.mark.subclassy\n def test_goofy_wrapper_url_exists(self):\n assert goofy_wrapper(\n \"https://thedatasavvycorner.com/\", 1\n ).endswith(\"Your site is up!\"),\"The site wasn't up.\"\n @pytest.mark.subclassy\n def test_goofy_wrapper_url_does_not_exist(self):\n assert goofy_wrapper(\n \"https://thegoofycorner.com/\", 1\n ).endswith(\"Your site is down!\"), \"The site wasn't down.\"\n```\n:::\n\n\nThis test class can now be exclusively targeted by specifying multiple marks\nwith `and`:\n\n> `pytest -v -m \" and ... and \"`\n\n```{.abc}\n\n... % pytest -v -m \"integration and slow\" \n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 6 items / 4 deselected / 2 selected \n\nTestGoofyWrapper::test_goofy_wrapper_url_exists PASSED [ 50%]\nTestGoofyWrapper::test_goofy_wrapper_url_does_not_exist PASSED [100%]\n\n======================= 2 passed, 4 deselected in 2.30s =======================\n\n```\n\nNote that even though there are other tests marked with `integration` and\n`slow` separately, they are excluded on the basis that `and` expects them to be\nmarked with both.\n\n#### Deselecting All Marks\n\nNow that we have introduced multiple custom markers to our test suite, what if\nwe want to exclude all of these marked tests, just running the 'core' test\nsuite? Unfortunately, there is not a way to specify 'unmarked' tests. There is\nan old `pytest` plugin called\n[`pytest-unmarked`](https://pypi.org/project/pytest-unmarked/) that allowed\nthis functionality. Unfortunately, this plugin is not being actively maintained\nand is not compatible with `pytest` v8.0.0+. You could introduce a 'standard'\nor 'core' marker, but you'd need to remember to mark every unmarked test within\nyour test suite with it.\n\nAlternatively, what we can do is exclude each of the marks that have been\nregistered. There are 2 patterns for achieving this:\n\n> 1. `pytest -v -m \"not ... or not \"`\n> 2. `pytest -v -m \"not ( ... or )\"`\n\n```{.abc}\n... % pytest -v -m \"not (flaky or slow or integration)\"\n============================= test session starts =============================\nplatform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- ...\ncachedir: .pytest_cache\nrootdir: /...\nconfigfile: pyproject.toml\ntestpaths: ./tests\ncollected 6 items / 5 deselected / 1 selected \n\ntests/test_do_something.py::test_nothing PASSED [100%]\n\n======================= 1 passed, 5 deselected in 0.05s =======================\n\n```\nNote that using `or` has greedily excluded any test marked with at least one of\nthe specified marks. \n\n## Summary\n\nRegistering marks with `pytest` is very easy and is useful for controlling\nwhich tests are executed. We have illustrated:\n\n* registering marks\n* marking tests and test classes\n* the use of the `pytest -m` flag\n* selection of multiple marks\n* deselection of multiple marks\n\nOverall, this feature of `pytest` is simple and intuitive. There are more\noptions for marking tests. I recommend reading the\n[`pytest` custom markers](https://docs.pytest.org/en/7.1.x/example/markers.html)\nexamples for more information.\n\nAs mentioned earlier, this is the final in the\n[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english)\nseries. I will be taking a break from blogging about testing for a while. But\ncolleagues have asked about articles on property-based testing and some of the\nmore useful `pytest` plug-ins. I plan to cover these topics at a later date.\n\nIf you spot an error with this article, or have a suggested improvement then\nfeel free to\n[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). \n\nHappy testing!\n\n## Acknowledgements\n\nTo past and present colleagues who have helped to discuss pros and cons,\nestablishing practice and firming-up some opinions. Particularly:\n\n* Ethan\n* Sergio\n\n

fin!

\n\n", + "supporting": [ + "16-pytest-marks_files/figure-html" + ], + "filters": [], + "includes": {} + } +} \ No newline at end of file diff --git a/blogs/11-fiddly-bits-of-pytest.qmd b/blogs/11-fiddly-bits-of-pytest.qmd index 21876e1..3f8381b 100644 --- a/blogs/11-fiddly-bits-of-pytest.qmd +++ b/blogs/11-fiddly-bits-of-pytest.qmd @@ -29,7 +29,7 @@ jupyter: `pytest` is a testing package for the python framework. It is broadly used to quality assure code logic. This article discusses using test data as fixtures with `pytest` and is the first in a series of blogs called -[pytest in plain English](/../index.html#category=pytest-in-plain-english), +[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english), favouring accessible language and simple examples to explain the more intricate features of the `pytest` package. diff --git a/blogs/12-pytest-tmp-path.qmd b/blogs/12-pytest-tmp-path.qmd index dfa6d46..2ecdd3e 100644 --- a/blogs/12-pytest-tmp-path.qmd +++ b/blogs/12-pytest-tmp-path.qmd @@ -32,7 +32,7 @@ jupyter: quality assure code logic. This article discusses why and how we use pytest's temporary fixtures `tmp_path` and `tmp_path_factory`. This blog is the second in a series of blogs called -[pytest in plain English](/../index.html#category=pytest-in-plain-english), +[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english), favouring accessible language and simple examples to explain the more intricate features of the `pytest` package. diff --git a/blogs/13-pytest-parametrize.qmd b/blogs/13-pytest-parametrize.qmd index 9e779ab..06b7ec5 100644 --- a/blogs/13-pytest-parametrize.qmd +++ b/blogs/13-pytest-parametrize.qmd @@ -31,7 +31,7 @@ css: /www/13-pytest-parametrize/styles.css quality assure code logic. This article discusses what parametrized tests mean and how to implement them with `pytest`. This blog is the third in a series of blogs called -[pytest in plain English](/../index.html#category=pytest-in-plain-english), +[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english), favouring accessible language and simple examples to explain the more intricate features of the `pytest` package. diff --git a/blogs/15-pytest-mocking.qmd b/blogs/15-pytest-mocking.qmd index 0549709..edb0750 100644 --- a/blogs/15-pytest-mocking.qmd +++ b/blogs/15-pytest-mocking.qmd @@ -47,7 +47,7 @@ For a wealth of documentation, guides and how-tos, please consult the Code often has external dependencies: -* Web APIs (as in this article) +* Web APIs (as in this article) * Websites (if scraping / crawling) * External code (importing packages) * Data feeds and databases diff --git a/blogs/16-pytest-marks.qmd b/blogs/16-pytest-marks.qmd new file mode 100644 index 0000000..3b9f0b5 --- /dev/null +++ b/blogs/16-pytest-marks.qmd @@ -0,0 +1,855 @@ +--- +title: "Custom Marks With Pytest in Plain English" +author: "Rich Leyshon" +date: "July 22 2024" +description: "Selectively running tests with marks" +categories: + - Explanation + - pytest + - Unit tests + - marks + - custom marks + - markers + - pytest-in-plain-english +image: "https://i.imgur.com/dCJBh9w.jpeg" +image-alt: "Planet composed of croissants" +toc: true +toc-depth: 4 +jupyter: + kernelspec: + name: "conda-env-mocking-env-py" + language: "python" + display_name: "blog-mocking-env" +css: /www/13-pytest-parametrize/styles.css +--- + +
+ +
Planet composed of croissants.
+
+ +> "The only flake I want is that of a croissant." Mariel Feldman. + +## Introduction + +`pytest` is a testing package for the python framework. It is broadly used to +quality assure code logic. This article discusses custom marks, use cases and +patterns for selecting and deselecting marked tests. This blog is the fifth and +final in a series of blogs called +[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english), +favouring accessible language and simple examples to explain the more intricate +features of the `pytest` package. + +For a wealth of documentation, guides and how-tos, please consult the +`pytest` documentation. + +### What are `pytest` Custom Marks? + +Marks are a way to conditionally run specific tests. There are a few marks that +come with the `pytest` package. To view these, run `pytest --markers` from the +command line. This will print a list of the pre-registered marks available +within the `pytest` package. However, it is extremely easy to register your own +markers, allowing greater control over which tests get executed. + +This article will cover: + +* Reasons for marking tests +* Registering marks with `pytest` +* Marking tests +* Including or excluding markers from the command line + +:::{.callout collapse="true"} + +### A Note on the Purpose (Click to expand) + +This article intends to discuss clearly. It doesn't aim to be clever or +impressive. Its aim is to extend understanding without overwhelming the reader. +The code may not always be optimal, favouring a simplistic approach wherever +possible. + +::: + +### Intended Audience + +Programmers with a working knowledge of python and some familiarity with +`pytest` and packaging. The type of programmer who has wondered about how to +follow best practice in testing python code. + +### What You'll Need: + +- [ ] Preferred python environment manager (eg `conda`) +- [ ] `pip install pytest==8.1.1` +- [ ] Git +- [ ] GitHub account +- [ ] Command line access + +### Preparation + +This blog is accompanied by code in +[this repository](https://github.com/r-leyshon/pytest-fiddly-examples). The +main branch provides a template with the minimum structure and requirements +expected to run a `pytest` suite. The repo branches contain the code used in +the examples of the following sections. + +Feel free to fork or clone the repo and checkout to the example branches as +needed. + +The example code that accompanies this article is available in the +[marks branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks) +of the repo. + +## Overview + +Occasionally, we write tests that are a bit distinct to the rest of our test +suite. They could be integration tests, calling on elements of our code from +multiple modules. They could be end to end tests, executing a pipeline from +start to finish. Or they could be a flaky or brittle sort of test, a test that +is prone to failure on specific operating systems, architectures or when +external dependencies do not provide reliable inputs. + +There are multiple ways to handle these kinds of tests, including mocking, as +discussed in [my previous blog](/blogs/15-pytest-mocking.qmd). Mocking can +often take a bit of time, and developers don't always have that precious +commodity. So instead, they may mark the test, ensuring that it doesn't get run +on continuous integration (CI) checks. This may involve flagging any flaky test +as "technical debt" to be investigated and fixed later. + +In fact, there are a number of reasons that we may want to selectively run +elements of a test suite. Here is a selection of scenarios that could benefit +from marking. + +:::{.callout-note collapse="true"} + +### Flaky Tests: Common Causes + +| **Category** | **Cause** | **Explanation** | +|-----------------------------|-----------------------------|-------------------------------------------------------------------------| +| External Dependencies | Network | Network latency, outages, or Domain Name System (DNS) issues. | +| | Web APIs | Breaking changes, rate limits, or outages. | +| | Databases | Concurrency issues, data changes, or connection problems. | +| | Timeouts | Hardcoded or too-short timeouts cause failures. | +| Environment Dependencies | Environment Variables | Incorrectly set environment variables. | +| | File System | File locks, permissions, or missing files. | +| | Resource Limits | Insufficient CPU, memory, or disk space. | +| State Dependencies | Shared State | Interference between tests sharing state. | +| | Order Dependency | Tests relying on execution order. | +| Test Data | Random Data | Different results on each run due to random data and seed not set. | +| Concurrency Issues | Parallel Execution | Tests not designed for parallel execution. | +| | Locks | Deadlocks or timeouts involving locks or semaphores. | +| | Race Conditions | Tests depend on the order of execution of threads or processes. | +| | Async Operations | Improperly awaited asynchronous code. | +| Hardware and System Issues | Differences in Hardware | Variations in performance across hardware or operating systems. | +| | System Load | Failures under high system load due to resource contention. | +| Non-deterministic Inputs | Time | Variations in current time affecting test results. | +| | User Input | Non-deterministic user input causing flaky behaviour. | +| | Filepaths | CI runner filepaths may be hard to predict. | +| Test Implementation Issues | Assertions | Incorrect or overly strict assertions. | +| | Setup and Teardown | Inconsistent state due to improper setup or teardown. | +: {.light .hover .responsive-md} + +::: + +In the case of integration tests, one approach may be to group them all +together and have them execute within a dedicated CI workflow. This is common +practice as developers may want to stay alert to problems with external +resources that their code depends upon, while not failing 'core' CI checks +about changes to the source code. If your code relies on a web API; for +instance; you're probably less concerned about temporary outages in that +service. However, a breaking change to that service would require our source +code to be adapted. Once more, life is a compromise. + +> "*Le mieux est l'ennemi du bien.*" (The best is the enemy of the good), +Voltaire + +## Custom Marks in `pytest` + +Marking allows us to have greater control over which of our tests are executed +when we invoke `pytest`. Marking is conveniently implemented in the following +way (presuming you have already written your source and test code): + +1. Register a custom marker +2. Assign the new marker name to the target test +3. Invoke `pytest` with the `-m` (MARKEXPR) flag. + +This section uses code available in the +[marks branch](https://github.com/r-leyshon/pytest-fiddly-examples/tree/marks) +of the GitHub repository. + +### Define the Source Code + +I have a motley crew of functions to consider. A sort of homage to Sergio +Leone's 'The Good, The Bad & the Ugly', although I'll let you figure out which +is which. + +
+ +
The Flaky, The Slow & The Needy
+
+ + +#### The Flaky Function + +Here we define a function that will fail half the time. What a terrible test to +have. The root of this unpredictable behaviour should be diagnosed as a +priority as a matter of sanity. +```{python filename="src/example_pkg/do_something.py"} +import random + + +def croissant(): + """A very flaky function.""" + if round(random.uniform(0, 1)) == 1: + return True + else: + raise Exception("Flaky test detected!") + +``` + +#### The Slow Function + +This function is going to be pretty slow. Slow test suites throttle our +productivity. Once it finishes waiting for a specified number of seconds, it +will return a string. + +```{python, filename="src/example_pkg/do_something.py"} + +import time +from typing import Union + + +def take_a_nap(how_many_seconds:Union[int, float]) -> str: + """Mimic a costly function by just doing nothing for a specified time.""" + time.sleep(float(how_many_seconds)) + return "Rise and shine!" + +``` + + +#### The Needy Function + +Finally, the needy function will have an external dependency on a website. This +test will simply check whether we get a HTTP status code of 200 (ok) when we +request any URL. + +```{python filename="src/example_pkg/do_something.py"} +import requests + + +def check_site_available(url:str, timeout:int=5) -> bool: + """Checks if a site is available.""" + try: + response = requests.get(url, timeout=timeout) + return True if response.status_code == 200 else False + except requests.RequestException: + return False + +``` + +#### The Wrapper + +Finally, I'll introduce a wrapper that will act as an opportunity for an +integration test. This is a bit awkward, as none of the above functions are +particularly related to each other. + +This function will execute the `check_site_available()` and `take_a_nap()` +together. A pretty goofy example, I admit. Based on the status of the url +request, a string will be returned. + +```{python, filename="src/example_pkg/do_something.py"} +import time +from typing import Union + +import requests + +def goofy_wrapper(url:str, timeout:int=5) -> str: + """Check a site is available, pause for no good reason before summarising + outcome with a string.""" + msg = f"Napping for {timeout} seconds.\n" + msg = msg + take_a_nap(timeout) + if check_site_available(url): + msg = msg + "\nYour site is up!" + else: + msg = msg + "\nYour site is down!" + + return msg +``` + + +### Let's Get Testing + +Initially, I will define a test that does nothing other than pass. This will +be a placeholder, unmarked test. + +```{python, filename="tests/test_do_something.py"} +def test_nothing(): + pass +``` + +Next, I import `croissant()` and assert that it returns `True`. As you may +recall from above, `croissant()` will do so ~50 % of the time. + +```{python, filename="tests/test_do_something.py"} +#| eval: false + +from example_pkg.do_something import ( + croissant, + ) + + +def test_nothing(): + pass + + +def test_croissant(): + assert croissant() + +``` + +Now running `pytest -v` will print the test results, reporting test outcomes +for each test separately (`-v` means verbose). + +```{.abc} +...% pytest -v +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 2 items +tests/test_do_something.py::test_nothing PASSED [ 50%] +tests/test_do_something.py::test_croissant PASSED [100%] +============================== 2 passed in 0.05s ============================== + +``` + +But note that half the time, I will also get the following output: + +```{.abc} +...% pytest -v +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 2 items + +tests/test_do_something.py::test_nothing PASSED [ 50%] +tests/test_do_something.py::test_croissant FAILED [100%] + +================================== FAILURES ================================== +_______________________________ test_croissant ________________________________ + @pytest.mark.flaky + def test_croissant(): +> assert croissant() +tests/test_do_something.py:17: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + def croissant(): + """A very flaky function.""" + if round(random.uniform(0, 1)) == 1: + return True + else: +> raise Exception("Flaky test detected!") +E Exception: Flaky test detected! + +src/example_pkg/do_something.py:13: Exception +============================short test summary info =========================== +FAILED ...::test_croissant - Exception: Flaky test detected! +========================= 1 failed, 1 passed in 0.07s ========================= +``` + +To prevent this flaky test from failing the test suite, we can choose to mark +it as flaky, and optionally skip it when invoking `pytest`. To go about that, +we first need to register a new marker. To do that, let's update out project's +`pyproject.toml` to include additional options for a `flaky` mark: + +```{.abc} +# `pytest` configurations +[tool.pytest.ini_options] +markers = [ + "flaky: tests that can randomly fail through no change to the code", +] +``` + +Note that when registering a marker in this way, text after the colon is an +optional mark description. Saving the document and running `pytest --markers` +should show that a new custom marker is available to our project: + +```{.abc} +... % pytest --markers +@pytest.mark.flaky: tests that can randomly fail through no change to the code +... + +``` +Now that we have confirmed our marker is available for use, we can use it to +mark `test_croissant()` as flaky: + +```{python, filename="tests/test_do_something.py"} +#| eval: false +import pytest + +from example_pkg.do_something import ( + croissant, + ) + + +def test_nothing(): + pass + + +@pytest.mark.flaky +def test_croissant(): + assert croissant() + +``` + +Note that we need to import `pytest` to our test module in order to use the +`pytest.mark.` decorator. + +#### Selecting a Single Mark + +Now that we have registered and marked a test as `flaky`, we can adapt our +`pytest` call to execute tests with that mark only. The pattern we will use is: + +> `pytest -k -m ""` + +```{.abc} +... % pytest -v -m "flaky" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 2 items / 1 deselected / 1 selected + +tests/test_do_something.py::test_croissant PASSED [100%] + +======================= 1 passed, 1 deselected in 0.05s ======================= + +``` + +Now we see that `test_croissant()` was executed, while the unmarked +`test_nothing()` was not. + +#### Deselecting a Single Mark + +More useful than selectively running a flaky test is to deselect it. In this +way, it cannot fail our test suite. This is achieved with the following +pattern: + +> `pytest -v -m "not "` + + +```{.abc} +... % pytest -v -m "not flaky" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 2 items / 1 deselected / 1 selected + +tests/test_do_something.py::test_nothing PASSED [100%] + +======================= 1 passed, 1 deselected in 0.05s ======================= + +``` + +Note that this time, `test_flaky()` was not executed. + +#### Selecting Multiple Marks + +In this section, we will introduce another, differently marked test to +illustrate the syntax for running multiple marks. For this example, we'll test +`take_a_nap()`: + +```{python} +#| eval: false +import pytest + +from example_pkg.do_something import ( + croissant, + take_a_nap, + ) + + +def test_nothing(): + pass + + +@pytest.mark.flaky +def test_croissant(): + assert croissant() + + +@pytest.mark.slow +def test_take_a_nap(): + out = take_a_nap(how_many_seconds=3) + assert isinstance(out, str), f"a string was not returned: {type(out)}" + assert out == "Rise and shine!", f"unexpected string pattern: {out}" + +``` + +Our new test just makes some simple assertions about the string `take_a_nap()` +returns after snoozing. But notice what happens when running `pytest -v` now: + +```{.abc} +... % pytest -v +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 3 items + +tests/test_do_something.py::test_nothing PASSED [ 33%] +tests/test_do_something.py::test_croissant PASSED [ 66%] +tests/test_do_something.py::test_take_a_nap PASSED [100%] + +============================== 3 passed in 3.07s ============================== + +``` + +The test suite now takes in excess of 3 seconds to execute, as the test +specified for `take_a_nap()` to sleep for that period. Let's update our +`pyproject.toml` and register a new mark: + +```{.abc} +# `pytest` configurations +[tool.pytest.ini_options] +markers = [ + "flaky: tests that can randomly fail through no change to the code", + "slow: marks tests as slow (deselect with '-m \"not slow\"')", +] +``` + +Note that the nested speech marks within the description of the `slow` mark +were escaped. `pytest` would have complained that the toml file was not valid +unless we ensured it was valid [toml syntax](https://toml.io/en/). + +In order to run tests marked with either `flaky` or `slow`, we can use `or`: + +> `pytest -v -m " or "` + +```{.abc} +... % pytest -v -m "flaky or slow" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 3 items / 1 deselected / 2 selected + +tests/test_do_something.py::test_croissant PASSED [ 50%] +tests/test_do_something.py::test_take_a_nap PASSED [100%] + +======================= 2 passed, 1 deselected in 3.06s ======================= + +``` + +Note that anything not marked with `flaky` or `slow` (eg `test_nothing()`) was +not run. Also, `test_croissant()` failed 3 times in a row while I tried to get +a passing run. I didn't want the flaky exception to carry on presenting itself. +While I may be sprinkling glitter, I do not want to misrepresent how +frustrating flaky tests can be! + +
Glitter GIFfrom Glitter GIFs
+ +#### Complex Selection Rules + +By adding an additional mark, we can illustrate more complex selection and +deselection rules for invoking `pytest`. Let's write an integration test that +checks whether the domain for this blog site can be reached. + +```{python} +#| eval: false +import pytest + +from example_pkg.do_something import ( + croissant, + take_a_nap, + check_site_available, + ) + + +def test_nothing(): + pass + + +@pytest.mark.flaky +def test_croissant(): + assert croissant() + + +@pytest.mark.slow +def test_take_a_nap(): + out = take_a_nap(how_many_seconds=3) + assert isinstance(out, str), f"a string was not returned: {type(out)}" + assert out == "Rise and shine!", f"unexpected string pattern: {out}" + + +@pytest.mark.integration +def test_check_site_available(): + url = "https://thedatasavvycorner.com/" + assert check_site_available(url), f"site {url} is down..." + +``` + +Now updating our `pyproject.toml` like so: + +```{.abc} +# `pytest` configurations +[tool.pytest.ini_options] +markers = [ + "flaky: tests that can randomly fail through no change to the code", + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "integration: tests that require external resources", +] +``` + +Now we can combine `and` and `not` statements when calling `pytest` to execute +just the tests we need to. In the below, I choose to run the `slow` and +`integration` tests while excluding that pesky `flaky` test. + +```{.abc} +... % pytest -v -m "slow or integration and not flaky" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 4 items / 2 deselected / 2 selected + +tests/test_do_something.py::test_take_a_nap PASSED [ 50%] +tests/test_do_something.py::test_check_site_available PASSED [100%] + +======================= 2 passed, 2 deselected in 3.29s ======================= +``` + +Note that both `test_nothing()` (unmarked) and `test_croissant()` (deselected) +were not run. + +#### Marks and Test Classes + +Note that so far, we have applied marks to test functions only. But we can also +apply marks to an entire test class, or even target specific test modules. For +this section, I will introduce the wrapper function introduced earlier and use +a test class to group its tests together. I will mark those tests with 2 new +marks, `classy` and `subclassy`. + +```{.abc} +# `pytest` configurations +[tool.pytest.ini_options] +markers = [ + "flaky: tests that can randomly fail through no change to the code", + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "integration: tests that require external resources", + "classy: tests arranged in a class", + "subclassy: test methods", +] +``` + +Updating our test module to include `test_goofy_wrapper()`: + +```{python} +#| eval: false + +import pytest + +from example_pkg.do_something import ( + croissant, + take_a_nap, + check_site_available, + goofy_wrapper + ) + + +def test_nothing(): + pass + + +@pytest.mark.flaky +def test_croissant(): + assert croissant() + + +@pytest.mark.slow +def test_take_a_nap(): + out = take_a_nap(how_many_seconds=3) + assert isinstance(out, str), f"a string was not returned: {type(out)}" + assert out == "Rise and shine!", f"unexpected string pattern: {out}" + + +@pytest.mark.integration +def test_check_site_available(): + url = "https://thedatasavvycorner.com/" + assert check_site_available(url), f"site {url} is down..." + + +@pytest.mark.classy +class TestGoofyWrapper: + @pytest.mark.subclassy + def test_goofy_wrapper_url_exists(self): + assert goofy_wrapper( + "https://thedatasavvycorner.com/", 1 + ).endswith("Your site is up!"), "The site wasn't up." + @pytest.mark.subclassy + def test_goofy_wrapper_url_does_not_exist(self): + assert goofy_wrapper( + "https://thegoofycorner.com/", 1 + ).endswith("Your site is down!"), "The site wasn't down." + +``` + +Note that targeting either the `classy` or `subclassy` mark results in the +same output - all tests within this test class are executed: + +```{.abc} +... % pytest -v -m "classy" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 6 items / 4 deselected / 2 selected + +TestGoofyWrapper::test_goofy_wrapper_url_exists PASSED [ 50%] +TestGoofyWrapper::test_goofy_wrapper_url_does_not_exist PASSED [100%] + +======================= 2 passed, 4 deselected in 2.30s ======================= + +``` + +Nobody created the domain `https://thegoofycorner.com/` yet, such a shame. + +#### Tests with Multiple Marks + +Note that we can use multiple marks with any test or test class. Let's update +`TestGoofyWrapper` to be marked as `integration` & `slow`: + +```{python} +#| eval: false +@pytest.mark.slow +@pytest.mark.integration +@pytest.mark.classy +class TestGoofyWrapper: + @pytest.mark.subclassy + def test_goofy_wrapper_url_exists(self): + assert goofy_wrapper( + "https://thedatasavvycorner.com/", 1 + ).endswith("Your site is up!"),"The site wasn't up." + @pytest.mark.subclassy + def test_goofy_wrapper_url_does_not_exist(self): + assert goofy_wrapper( + "https://thegoofycorner.com/", 1 + ).endswith("Your site is down!"), "The site wasn't down." + +``` + +This test class can now be exclusively targeted by specifying multiple marks +with `and`: + +> `pytest -v -m " and ... and "` + +```{.abc} + +... % pytest -v -m "integration and slow" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- /... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 6 items / 4 deselected / 2 selected + +TestGoofyWrapper::test_goofy_wrapper_url_exists PASSED [ 50%] +TestGoofyWrapper::test_goofy_wrapper_url_does_not_exist PASSED [100%] + +======================= 2 passed, 4 deselected in 2.30s ======================= + +``` + +Note that even though there are other tests marked with `integration` and +`slow` separately, they are excluded on the basis that `and` expects them to be +marked with both. + +#### Deselecting All Marks + +Now that we have introduced multiple custom markers to our test suite, what if +we want to exclude all of these marked tests, just running the 'core' test +suite? Unfortunately, there is not a way to specify 'unmarked' tests. There is +an old `pytest` plugin called +[`pytest-unmarked`](https://pypi.org/project/pytest-unmarked/) that allowed +this functionality. Unfortunately, this plugin is not being actively maintained +and is not compatible with `pytest` v8.0.0+. You could introduce a 'standard' +or 'core' marker, but you'd need to remember to mark every unmarked test within +your test suite with it. + +Alternatively, what we can do is exclude each of the marks that have been +registered. There are 2 patterns for achieving this: + +> 1. `pytest -v -m "not ... or not "` +> 2. `pytest -v -m "not ( ... or )"` + +```{.abc} +... % pytest -v -m "not (flaky or slow or integration)" +============================= test session starts ============================= +platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0 -- ... +cachedir: .pytest_cache +rootdir: /... +configfile: pyproject.toml +testpaths: ./tests +collected 6 items / 5 deselected / 1 selected + +tests/test_do_something.py::test_nothing PASSED [100%] + +======================= 1 passed, 5 deselected in 0.05s ======================= + +``` +Note that using `or` has greedily excluded any test marked with at least one of +the specified marks. + +## Summary + +Registering marks with `pytest` is very easy and is useful for controlling +which tests are executed. We have illustrated: + +* registering marks +* marking tests and test classes +* the use of the `pytest -m` flag +* selection of multiple marks +* deselection of multiple marks + +Overall, this feature of `pytest` is simple and intuitive. There are more +options for marking tests. I recommend reading the +[`pytest` custom markers](https://docs.pytest.org/en/7.1.x/example/markers.html) +examples for more information. + +As mentioned earlier, this is the final in the +[pytest in plain English](/blogs/index.qmd#category=pytest-in-plain-english) +series. I will be taking a break from blogging about testing for a while. But +colleagues have asked about articles on property-based testing and some of the +more useful `pytest` plug-ins. I plan to cover these topics at a later date. + +If you spot an error with this article, or have a suggested improvement then +feel free to +[raise an issue on GitHub](https://github.com/r-leyshon/blogging/issues). + +Happy testing! + +## Acknowledgements + +To past and present colleagues who have helped to discuss pros and cons, +establishing practice and firming-up some opinions. Particularly: + +* Ethan +* Sergio + +

fin!

diff --git a/docs/blogs/11-fiddly-bits-of-pytest.html b/docs/blogs/11-fiddly-bits-of-pytest.html index 783bd58..d4a1dac 100644 --- a/docs/blogs/11-fiddly-bits-of-pytest.html +++ b/docs/blogs/11-fiddly-bits-of-pytest.html @@ -236,7 +236,7 @@

Pytest Fixtures in Plain English

Introduction

-

pytest is a testing package for the python framework. It is broadly used to quality assure code logic. This article discusses using test data as fixtures with pytest and is the first in a series of blogs called pytest in plain English, favouring accessible language and simple examples to explain the more intricate features of the pytest package.

+

pytest is a testing package for the python framework. It is broadly used to quality assure code logic. This article discusses using test data as fixtures with pytest and is the first in a series of blogs called pytest in plain English, favouring accessible language and simple examples to explain the more intricate features of the pytest package.

For a wealth of documentation, guides and how-tos, please consult the pytest documentation.