Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allure.NUnit improvements #460

Merged
merged 10 commits into from
Mar 28, 2024
Merged

Allure.NUnit improvements #460

merged 10 commits into from
Mar 28, 2024

Conversation

delatrie
Copy link
Contributor

@delatrie delatrie commented Mar 27, 2024

Context

This PR improves the user experience with Allure NUnit.

Change of the namespace

The main namespace of the package now is Allure.NUnit. Users should adapt their code accordingly:

  • NUnit.Allure.Core.AllureNUnitAttribute -> Allure.NUnit.AllureNUnitAttribute
  • NUnit.Allure.Attachments -> Allure.NUnit.Attachments
  • NUnit.Allure.Attributes.* -> Allure.NUnit.Attributes.*

The old NUnit.Allure namespace is still in place, but accessing it will trigger a deprecation warning.

Improved error statuses reporting

Fixture and step errors

From a user's perspective, nothing has changed. The decision is not made by the code in Allure.Net.Commons, but the result is the same: exceptions of types AssertionException and NUnitException make the step/fixture failed while exceptions of other types - broken.

Test errors

NUnit considers a test failed in the following cases:

  • a test throws an unhandled exception (including assertion-related exceptions)
  • at least one failed assertion occurred during the test (regardless of whether the exception was handled).
    NUnit doesn't give us an instance of the exception — only the list of assertion results and a final error message.

Previous implementation

An Allure test result is considered broken if the corresponding NUnit test error message contains at least one entry from the brokenTestData configuration property (typically, full names of exception types that should make tests broken). Otherwise, the test result is considered failed.

That requires users to put all possible exception types in the config. Otherwise, test defects are reported as product ones.

New implementation

An Allure test result that corresponds to a failed NUnit test is considered broken in any of the following cases:

  • It doesn't have any assertion, or
  • It contains an invalid assertion (of status AssertionStatus.Error) - that corresponds to an unhandled exception.

Additionally, inconclusive tests and tests with warnings are reported as broken (that stayed the same).

The brokenTestData configuration property is ignored from now on.

Api deprecation and removals

The following API is deprecated now:

  • AllureExtensions:
    • WrapSetUpTearDownParams: it had no effect.
    • WrapInStep (all overloads): use AllureApi.Step instead.

The public API that had been previously deprecated was removed.

  • AllureNUnitHelper:
    • WrapInStep: use [AllureStep] or AllureApi.Step instead.
    • SaveOneTimeResultToContext: hadn't meant to be public.
    • AddOneTimeSetupResult: hadn't meant to be public.
  • StepsHelper: use [AllureStep], AllureApi, or ExtendedApi instead.
  • AllureNUnitAttribute,s constructor with the bool wrapIntoStep parameter (had no effect).
  • AllureExtensions:
    • AddScreenDiff: use AllureApi.AddScreenDiff instead.

The following API that was meant to be internal is now hidden:

  • AllureNUnitHelper.GetNUnitStatus and the entire AllureNUnitHelper class.

Other changes

  • Empty console logs and status details are not included in the report now (fixes Preclude empty console logs and status details from the report produced by allure-nunit #363)
  • Fixed missing container if [AllureAfter] is applied to some but not all [OneTimeTearDown] methods of a test fixture.
  • Fixed IndexOutOfRangeException if [AllureAfter] is applied to a method with arguments (fixes NUnit: IndexOutOfRangeException if AllureAfter is applied on a method with paramters #419).
  • Allure NUnit now uses the common skip reason for tests excluded by a test plan.
  • Ignored tests not in the current test plan are not reported even if [AllureDisplayIgnoredAttribute] is applied.
  • allureConfig.json was removed from Allure.NUnit source code.
  • Now, we don't emit empty containers for ignored tests.
  • An ignored test from Allure.NUnit.Examples was renamed so as not to collide with its non-ignored counterpart.
  • Labeler entry was fixed for Allure.NUnit examples project.

@delatrie delatrie self-assigned this Mar 27, 2024
@delatrie delatrie force-pushed the nunit-fixes-2403 branch 2 times, most recently from 6f53c62 to 6cbced8 Compare March 27, 2024 23:08
@delatrie delatrie added task:improvement Change that improves some user experience but can't be considered a new feature type:improvement and removed task:improvement Change that improves some user experience but can't be considered a new feature labels Mar 27, 2024
@delatrie delatrie force-pushed the commons-fixes-2403 branch from efc568a to 61705d9 Compare March 28, 2024 10:50
@delatrie delatrie force-pushed the commons-fixes-2403 branch from 61705d9 to 0a561ad Compare March 28, 2024 13:20
delatrie added 10 commits March 28, 2024 20:24
  - New algorithm to distinguish between failed/broken tests. Now, a test
    is failed if NUnit shows at least one failed assertion and no
    assertion errors. The 'brokenTestData' config property is removed.
  - Remove empty status details for passed tests (#363)
Additional changes:
  - Remove redundant config from Allure.NUnit project
  - Fix conflicting IgnoreTestWithTestCaseData test names in
    Allure.NUnit.Examples
  - missing container if [AllureAfter] is applied to some, but not all
    OneTimeTearDown methods of a class
  - Fix IndexOutOfRangeException if [AllureAfter] is applied to a method
    with arguments
Every ignored test container contains no fixtures.
Backward compatibility is implemented via OldNamespaceCompatibility
  - AllureNUnitHelper:
      - WrapInStep
      - SaveOneTimeResultToContext
      - AddOneTimeSetupResult
  - StepsHelper
@delatrie delatrie changed the base branch from commons-fixes-2403 to main March 28, 2024 13:24
@delatrie delatrie marked this pull request as ready for review March 28, 2024 13:25
@delatrie delatrie requested a review from epszaw March 28, 2024 13:31
@delatrie delatrie merged commit 39cbb9b into main Mar 28, 2024
1 check passed
@delatrie delatrie deleted the nunit-fixes-2403 branch March 28, 2024 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants