[Bug]: Error Encountered When Generating HTML Accessibility Reports in Storybook: page.evaluate: ReferenceError: __test is not defined error

[aimad-majdou]

Describe the bug

When attempting to automate accessibility tests with the provided code snippet in Storybook's documentation : https://storybook.js.org/docs/writing-tests/accessibility-testing#setup, the test runner encounters errors related to page.evaluate: ReferenceError: __test is not defined. This occurs when attempting to generate an HTML report using axe-playwright or axe-html-reporter.

To Reproduce

1. Use the provided code snippet in Storybook's documentation for automating accessibility tests.
2. Attempt to generate an HTML report using axe-playwright or axe-html-reporter:

await checkA11y(
      page,
      '#storybook-root',
      {
        detailedReport: true,
        detailedReportOptions: {
          html: true,
        },
        verbose: false,
        axeOptions: {
          ...(storyContext.parameters?.a11y?.config?.axeOptions ?? {}),
        },
      },
      false,
      'html',
    );

4. Observe the page.evaluate: ReferenceError: __test is not defined error in the console.

reproduction: https://stackblitz.com/edit/github-dwueer?file=package.json,.storybook%2Fmain.ts&preset=node

System

Storybook Environment Info:
(node:125404) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)

  System:
    OS: Windows 10 10.0.19045
    CPU: (8) x64 Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz      
  Binaries:
    Node: 21.3.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.2.4 - C:\Program Files\nodejs\npm.CMD <----- active
  Browsers:
    Edge: Chromium (122.0.2365.80)
  npmPackages:
    @storybook/addon-a11y: ^8.0.5 => 8.0.5 
    @storybook/addon-coverage: 1.0.1 => 1.0.1 
    @storybook/addon-essentials: 8.0.5 => 8.0.5 
    @storybook/addon-interactions: 8.0.5 => 8.0.5 
    @storybook/addon-links: 8.0.5 => 8.0.5 
    @storybook/addon-themes: 8.0.5 => 8.0.5 
    @storybook/blocks: 8.0.5 => 8.0.5 
    @storybook/react: 8.0.5 => 8.0.5 
    @storybook/react-vite: 8.0.5 => 8.0.5
    @storybook/test: 8.0.5 => 8.0.5
    @storybook/test-runner: 0.17.0 => 0.17.0
    eslint-plugin-storybook: 0.8.0 => 0.8.0
    storybook: 8.0.5 => 8.0.5

Additional context

No response

[unional]

dup of storybookjs/test-runner#68

[shilman]

Thanks @unional. There is already a lot of discussion on storybookjs/test-runner#68 and work in progress, so I'm closing this as a duplicate.

[toBeOfUse]

if this issue is about errors that happen when using axe-playwright or axe-html-reporter, then it is not a duplicate of the linked issue in the test runner repository. the linked issue has the same error message as a symptom, but it occurs in the absence of those accessibility tools, so it likely has a different cause