Feature: stream Cucumber json report events

[Sanorf]

Hello,

Is it possible with the json reporter to access the results datas after each scenario execution? Currently the json file is generated only at the end of all tests execution, I would like to have more granularity to send the results more frequently to my third party test management tool.

[vitalets]

Hi,

If you mean Cucumber json results, they are available only at the reporter end.
If you mean Playwright's json results, they are available in [onTestEnd(https://playwright.dev/docs/api/class-reporter#reporter-on-step-end) callbacks of the custom reporter API.

[Sanorf]

Sorry I didn't mention, I mean Cucumber json results. I would like if it is possible to not wait the end of all tests execution to get the report. This way project stakeholders can have first executed tests feedbacks more fast in their test management tool

[vitalets]

For now I'd suggest to try to use Playwright results for that task.
Which fields do you need? Playwright results contain most of the data, e.g. scenario name, passed/failing status, etc.

[Sanorf]

Results are sent to Xray API (Jira plugin): https://docs.getxray.app/display/XRAY/Import+Execution+Results+-+REST#ImportExecutionResultsREST-CucumberJSONresultsMultipart
As tests are Gherkin scenarios json results follow Cucumber json scheme

[vitalets]

The API accepts an array of items, where each item corresponds to a single feature, e.g.:

[
  {
    "keyword": "Feature",
    "name": "Feature 1",
    // ...
  },
  {
    "keyword": "Feature",
    "name": "Feature 2",
    // ...
  }
]

On the first glance, the granularity level is features. Or maybe it's possible to send a feature json with only some of the scenarios (in elements array)? Could you check that?

Anyway, currently in playwright-bdd it's not possible to stream parts of Cucumber json. But I agree this is useful feature to add.

[Sanorf]

Yes it's possible to only send json corresponding to a single feature or even a single scenario (elements level), as long as the scheme is respected.
Unfortunately i don't know how to access these stream datas on flight during the execution. I suppose they are stored somewhere in memory as all is injected in the report.json at the end.