Feature request: Generate "unknown" test results during pytest collection (allure-pytest)

[carlcsaposs-canonical]

I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

allure-pytest generates test results for each test after it finishes

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

What is the expected behavior?

allure-pytest generates "unknown" test result for each test during collection time & replaces that test result if the test completes

What is the motivation / use case for changing the behavior?

Currently, on our CI, if the test step times out (because of GitHub Actions timeout-minutes), tests that did not complete will be missing from the Allure Report.

We also distribute our tests across multiple GitHub Actions runners (to run in parallel), and if the set up of a runner fails, all of its tests will be missing from the Allure Report

Please tell us about your environment:

• Allure version: 2.29.0-1
• Test framework: [email protected]
• Allure adaptor: [email protected]

Other information

We would be happy to open a PR to implement this, but we're not sure how to refactor/structure the code (currently, most of the logic to generate test reports is coupled to the pytest hooks)

There's two use cases here:

1. for normal test execution, generate unknown test results during collection time & replace them during execution time
2. for running tests with collect-only, generate unknown test results
   • this is our use case, since we run pytest collection on one runner & use the results to provision multiple runners. If any of the test execution runners fail to set up, we would still like to show "unknown" results for all tests that were scheduled on that runner

We've actually implemented this already in a pytest plugin that imports from allure-pytest, but would love to upstream this (if that's something you'd want) so we can de-duplicate this code & so that our plugin doesn't break if allure-pytest code internals are modified

• Here's the pytest plugin that generates test results during collection time, if --allure-collection-dir is passed: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/pytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/_plugin.py
• And here's a python script that combines the "default results" (generated during collection time) with the actual results: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/cli/data_platform_workflows_cli/allure_add_default_for_missing_results.py
• full context of how we integrated this into our CI: patch(integration_test_charm.yaml): Add Allure "unknown" status for tests with missing results canonical/data-platform-workflows#183