Reports in json

[martotkd1]

Hello @bencompton ,

I am really interested in jest-cucumber. I compared it to CucumberJS and let's say it covers the must have features.

I saw there are already issues for asking if jest-cucumber can generate reports.json and i think jest-junit does not help me since it can't display the scenario steps.

Can you please let me know if you have something implemented in this direction ?

Regards,
Martin Bogdanov

[bencompton]

Agreed that jest-junit and other reporters aren't really sufficient. I am planning to look into creating a plug-in that will output Cucumber JSON when I get some time.

[UsharaniA]

Any tentative date on #27 enhancement?

[hemanta8841]

@martotkd1 Appreciate if you could provide any ETA on the reporting front ?

[bdjos1]

Yes I agree it would be great if there could be an ETA provided for this enhancement

[bencompton]

Improved reporting is the most commonly requested feature at this point, so this will definitely be next on the list, and I have already started some initial prototyping.

[bencompton]

I have been working on this here and there as I have spare time, and at this point, I have published an early preview version (2.0.12). It has still has some known issues I'm working to resolve, but it at least shows off what it can do.

The default setup in your Jest configuration looks like this:

"reporters": [
  "./node_modules/jest-cucumber/dist/src/reporter"
],

With the defaults, it will output report.json in the same directory as your package.json.

A custom output path can be specified like so:

"reporters": [
  "default",
  ["./node_modules/jest-cucumber/dist/src/reporter", {
    "formatter": "json",
    "path": "./reports/test-report.json"
  }]
],

The reporter works by using the Cucumber Event Protocol and formatters from Cucumber.js. This means that it is technically possible to use any formatter from Cucumber.js, but not all of them make sense with Jest. In addition to the JSON formatter, the progress and summary formatters are also supported.

The summary formatter outputs to the console at the end of a test run and shows which specific steps failed (if any), along with the number of scenarios / steps and the run-time.

"reporters": [
  "default",
  ["./node_modules/jest-cucumber/dist/src/reporter", {
    "formatter": "summary"
  }]
],

The progress formatter does everything the summary formatter does, but it shows the progress of your steps as well with colored dots for passing steps, "F" for skipped steps, "S" for skipped steps, etc. It is recommended that this be used without the default Jest reporter, since its output will get mixed in with the default reporter output otherwise:

"reporters": [
  ["./node_modules/jest-cucumber/dist/src/reporter", {
    "formatter": "progress"
  }]
],

Known issues:

• Need to find a cleaner way to reference it as a reporter without digging into node_modules and the package internals. Jest's module resolution for reports is not very well-documented and is kind of finicky.

• There are errors when normal Jest tests are executed with this reporter

• The time reported is not accurate

• It currently does not report declaration errors, and only reports step / test failures. For example, when step definitions are missing.

• Skipped steps are not yet implemented

• Long feature file names cause errors

• Issue when using with Angular, reported in TypeError: Converting circular structure to JSON #73

[s2oBCN]

Hi,
This will be great to have the JSON file with the annotations related to the scenario+feature.

I downloaded the code from the branch "reporting", to see if I can help you, but I'm sorry, I'm a newbie with TypeScript/nodeJS.

I see, that you are generating events to cucumber.js. Can you have a look at the serenity-js
https://github.com/jan-molak/serenity-js/tree/master/packages/serenity-js/src/serenity-cucumber. This code integrates Cucumber+Protractor with SerenityJS.

Maybe it is possible to integrate your solution with SerenityJS? They have pretty and detailed reports.

Thank you for your work!

[bencompton]

@s2oBCN - In addition to having the few built-in formatters listed above, I was also considering allowing custom formatter plug-ins. I haven't worked with SerenityJS, but looking at the code, it appears that serenity-cucumber listens for Cucumber events and calls the Serenity API appropriately. Something similar could probably be accomplished with a custom formatter plug-in.

[s2oBCN]

Do you think is it possible to have the tags in the json file?

[bencompton]

@s2oBCN - Any tags in your feature files currently will output to the JSON file as expected. Is that what you were asking?

For example, consider this feature file with these tags:

@feature-test
Feature: Rocket Launching

@scenario-test
Scenario: Launching a SpaceX rocket
...

The following JSON will be generated:

  {
    "keyword": "Feature",
    "name": "Rocket Launching",
    "line": 2,
    "id": "rocket-launching",
    "tags": [
      {
        "name": "@feature-test",
        "line": 1
      }
    ],
    "uri": "/git/jest-cucumber/examples/typescript/specs/features/basic-scenarios.feature",
    "elements": [
      {
        "id": "rocket-launching;launching-a-spacex-rocket",
        "keyword": "Scenario",
        "line": 5,
        "name": "Launching a SpaceX rocket",
        "tags": [
          {
            "name": "@feature-test",
            "line": 1
          },
          {
            "name": "@scenario-test",
            "line": 4
          }
        ],
        "type": "scenario",
        "steps": [
...

[s2oBCN]

Yes, I'm sorry, I executed some tests but I didn't find the tags in the json file. I suppose I was wrong, sorry.

So, when do you think we will have this code in a release? :)

[bdjos1]

Hi @bencompton

This looks great in terms of the output report it produces and the flexibility if offers in terms of format.

I just wanted to give some feedback in relation to minor issue I saw.

So the only thing I noticed is that there is now a restriction in relation to the length of Scenario name, that I don't think was there previously (or at least it has reduced). I got about 4 failure in relation to that but the fix was obvious by just decreasing the length of scenario name.

Not sure exactly what the threshold for the length is, as I just chopped off 20(ish) characters to make it pass.. Here is one of ones I had that caused the failure: 'Sign up confirmation banners displayed on the top of Electronic Signature page when user select's Sign up option on Login page when User select to sign up before submitting application'

Just wondering about the correct fix. Would it be for me to shorten the offending Scenario names OR would a resolution to first Known issue (cleaner reference to jest-cucumber-reporting node_module) you have specified solve it?

Will provide screenshot with error

[bdjos1]

[s2oBCN]

Great news!!!!

About the error, I see, there's a file per scenario, I suppose this is the only way. In another approach (in a Java project), I generated a file per feature (with the name of the feature), but I suppose this is not possible here.
Another solution that I did before, It was to generate a random number (UUID) after cut the initial name of the file...
Anyway, I can live with this restriction :)

Can you push the code in order to test it?

[s2oBCN]

I see an issue, when executing "Scenario Outline", if you change the sample:

Scenario Outline: Selling an <Item>

By:
Scenario Outline: Selling something

(So we will have allways the same title)

Then, the JSON report will only show result of the last Example of the Scenatio Outline.

Another thing, in other frameworks I can see the JSON when "Scenario Outline" with:
"keyword":"Scenario Outline",

I can see in the Cucumber.js project in the "keyword":"Scenario" is hardcoded, this is different from the implementation in Java. But, of course, this is not related to this project.

[bencompton]

@bdjos1 and @s2oBCN - thanks for the feedback! Yes, unfortunately, it has to write info about each scenario to a file because there isn't really another option with the way Jest's parallel execution works. I looked into hooking into the event mechanism that Jest itself uses to communicate between worker processes, but it turned out that would interfere with Jest. Could try another method of interprocess communication, or the file name could be a hash or something at least.

The scenario outline only showing the last example title is definitely an issue. Not sure why the Cucumber.js formatter splits up scenario outlines into separate scenarios, or whether there is a way to fix it. For people using the JSON to generate a living documentation website, this would be annoying to have the report not match the feature file.

Can you push the code in order to test it?

Sorry, not sure I follow. It's already published on NPM v2.0.12, or are you referring to something else?

[s2oBCN]

Ops! Yes, I see that is in v2.0.12. As I could not see the code in the master branch, I didn't realize that there was a release
Thank you for your work!!

Now, I will try to find the way distinguish both cases "Scenario Outline"/"Scenario", because I have a process that generates the test cases in Jira, from the execution report, and we use the tags to create fields like "data creation"/"Reviewer".... And we treat as only one testCase all the samples from a Scenario Outline.

Meanwhile, I deepened in your code, and I like it very much. I think it's a very smart and simple solution, the way of interact Jest with Cucumber.

[bencompton]

@s2oBCN - glad the project is useful to someone! Yeah, definitely need to see if there is a way to make it not split up scenario outlines into separate scenarios.

[7agustibm]

Is someone working on creating the report? It is to start creating it or helping it.

[yellowbrickc]

Hi @bencompton,

I'm using the whole lib (thanks for open sourcing it!), the reporting feature included and the only disturbing issue what I have at the moment is with skipping tests. This means, I cannot use any tags at the moment.

  ● Test suite failed to run

    TypeError: Cannot read property 'steps' of undefined

      at TestCaseEventGenerator.generateTestCasePreparedEvent (node_modules/jest-cucumber/src/reporting/report-event-generation/TestCaseEventGenerator.ts:22:19)
      at node_modules/jest-cucumber/src/reporting/report-event-generation/ReportEventGenerator.ts:34:43

I saw that

Skipped steps are not yet implemented

Can I help here somehow?

I would like to be able to include the skipped scenarios in the report, I think the information that specs are skipped belong in every report. This is why I would like to help to get this working.
Otherwise I could configure everything with the already existent flags.

As I see the error thrown by skipped scenarios occurs because the eventDataCollector.pickleMap does not contain anything about the skipped steps. I think if every skipped step would be still saved to the map with default values like startTime = endTime = 0, error = null, (maybe a state "skipped" ??) the problem could disappear.
What do you think?

[cappelaere]

Hey @bencompton,

I am using 3.0 and having difficulties generating a json file. Have the reporting changes been incorporated in master? Or 2.0.12 is still the only one that works?
Many thanks,
Pat

[bencompton]

Reporting was an experimental feature that unfortunately hasn't panned out so far and currently only works in 2.0.12, though it's buggy. I really wanted to finish up reporting for 3.0, but I ran into some issues that made it not so feasible:

• 3.0 now uses the latest Gherkin version from Cucumber.js, but the Cucumber.js code has undergone drastic changes that broke the reporting in this library.

• Reporting in this library also has some issues around scenario outlines--it outputs them as separate scenarios because it is missing the identifiers required to group them, which would require having access to the Cucumber Pickles data from the original feature file parsing to group them, which in turn would require writing all of that out to disk because Jest runs tests in parallel in separate processes / threads with no ability to share state in memory. I don't really like that the performance implications of writing that much out to disk, but not many alternatives come to mind. In addition, this library would probably need to use the Cucumber Pickles format instead of its own data model for parsed features.

• The Cucumber team has deprecated the JSON format in favor of a binary format

• The only halfway decent living documentation generator (that is open source and not paid) I am aware of (Pickles) is no longer maintained.

Long story short, reporting has fatal flaws and would need to be completely rewritten. My main motivation for JSON reporting is living documentation, and I started messing around building an open source living documentation and example mapping tool a while back, but it's hard to find time to devote to such endeavors.

[mainfraame]

@yellowbrickc @martotkd1 @cappelaere @s2oBCN @hemantaK @7agustibm you should check out cucumber-jest, it's a jest transformer that allows you to use cucumber's api for defining steps, hooks, world, etc. The output includes example tables (scenario outlines), tables, and doc strings.

[yellowbrickc]

Thanks @mentierd for the suggestion

[yellowbrickc]

Hi @mainfraame
As I see it, the package you have suggested does not support any reports at all. Or am I overseeing something?

[nicodinh]

Hi @mainfraame
As I see it, the package you have suggested does not support any reports at all. Or am I overseeing something?

it's a spam

[yellowbrickc]

Thanks @nicodinh
I couldn't even imagine that there is spam on GitHub, I don't see the reason, but making others to waste time. But I was too naive obviously.

[mattwynne]

Hi @bencompton, Cucumber maintainer here 👋. Thanks for all your work bringing Cucumber love to the Jest community!

If there's anything we can do from our end to make Cucumber-JS or its internal libraries easier to work with, please come over to https://cucumber.io/community#slack or https://github.com/cucumber/cucumber-js and we'll see how we can help. I know those bits have been through some pretty drastic changes of late, and I know how frustrating it can be trying to maintain tooling against libraries that change so much. Sorry! FWIW I think that change has probably settled down now, and we're back into a state of incremental evolution rather than revolution.

We do have an online service now for publishing Cucumber reports: https://reports.cucumber.io/ which works with the new message protocol (They're ndjson in fact, not binary). It's not open source but it is free.

I'm curious to hear what people are doing with that legacy JSON report format. We thought it was pretty hideous, but I think we've been surprised by the amount of tooling people had been building on top of it. For people in that situation, we've built a CLI tool which can take a stream of the new ndjson messages and pump out a legacy json report. Maybe that could help?

[aslakhellesoy]

Another Cucumber maintainer here. FWIW, we're phasing out the JSON report format and replacing it with cucumber messages. You can read more about why on that page.

The Cucumber team has deprecated the JSON format in favor of a binary format

This is incorrect. The format isn't binary, it's NDJSON. The current implementation uses protobuf's JSON format, but we're removing that - see cucumber/common#1386

[bencompton]

@mattwynne and @aslakhellesoy, thanks for reaching out! Migrating reporting has been on the backlog for a while mainly due to lack of bandwidth on my part, though if I do find any ways to improve the Cucumber libraries, I'll certainly be in contact. It's good to know that the dust has settled. I suppose the legacy JSON format sufficiently covered a lot of use cases, but the additional capabilities in the new format are quite valuable to the community. The CLI converter definitely helps so that libraries like this one won't have to output both formats.

This is incorrect. The format isn't binary, it's NDJSON. The current implementation uses protobuf's JSON format, but we're removing that - see cucumber/common#1386

Thanks, appreciate the clarification! Either way, good to know that it's a more inspectable, textual format. It seems I never noticed the update clarifying the format, and stuck with my invalid assumption that it was binary because it uses protobufs, so my apologies for spreading misinformation.

[rkrisztian]

So what can we do today to have the cucumber.json reporting working? Any temporary solution ideas? Just as cypress-cucumber-preprocessor can do it, I need something for Jest too. And I cannot afford to use a cloud service for that.

[elsasslegend]

What about this issue ? Not being able to generate .json reports from jest-cucumber, makes it useless for us when doing CI/CD with Jenkins.
We need a way to consume a cucumber generated report and send it back to JIRA for instance.
(Edit: had to switch to cucumber.js unfortunately)

[astrit-cdl]

anyone have an ideas on how to get screenshots embedded to failed steps for reporting? According to cucumberjs docs I should use an After hook like so:

var {After, Status} = require('cucumber');

After(function (testCase) {
  var world = this;
  if (testCase.result.status === Status.FAILED) {
    return webDriver.takeScreenshot().then(function(screenShot) {
      // screenShot is a base-64 encoded PNG
      world.attach(screenShot, 'image/png');
    });
  }
});

but every time I run the test I get hit with this error:

 FAIL  src/_tests/login_page/steps/login.steps.ts
  ● Test suite failed to run

    TypeError [ERR_INVALID_ARG_TYPE]: The "from" argument must be of type string. Received type undefined

      19 |   })
      20 |
    > 21 |   After(async (testCase) => {
         |        ^
      22 |     var world = require("cucumber").World;
      23 |     if (testCase.result.status === Status.FAILED) {
      24 |       return driver.takeScreenshot().then(function(screenShot) {

      at getDefinitionLineAndUri (node_modules/cucumber/lib/support_code_library_builder/build_helpers.js:184:27)
      at buildTestCaseHookDefinition (node_modules/cucumber/lib/support_code_library_builder/build_helpers.js:53:7)
      at node_modules/cucumber/lib/support_code_library_builder/index.js:62:77
      at src/_tests/login_page/steps/login.steps.ts:21:8
      at node_modules/jest-cucumber/src/feature-definition-creation.ts:314:9
      at defineFeature (node_modules/jest-cucumber/src/feature-definition-creation.ts:313:5)
      at Object.<anonymous> (src/_tests/login_page/steps/login.steps.ts:11:14)

[Pravallikaps]

Hey @bencompton,

"reporters": [
  "default",
  ["./node_modules/jest-cucumber/dist/src/reporter", {
    "formatter": "json",
    "path": "./reports/test-report.json"
  }]
],

This does not work now. I am using nodejs v14.x.x and npm v6.x.x. [email protected]

Error: An error occurred while adding the reporter at path "C:\Users\...\node_modules\jest-cucumber\dist\src\reporter.js".
Cannot find module 'cucumber'
Require stack:
- C:\Users\...\node_modules\jest-cucumber\dist\src\reporting\report-event-generation\ReportEventGenerator.js
- C:\Users\...\node_modules\jest-cucumber\dist\src\reporting\index.js      
- C:\Users\...\node_modules\jest-cucumber\dist\src\reporter.js
- C:\Users\...\node_modules\jest-util\build\requireOrImportModule.js       
- C:\Users\...\node_modules\jest-util\build\index.js
- C:\Users\...\node_modules\jest-config\build\getCacheDirectory.js
- C:\Users\...\node_modules\jest-config\build\Defaults.js
- C:\Users\...\node_modules\jest-config\build\normalize.js
- C:\Users\...\node_modules\jest-config\build\index.js
- C:\Users\...\node_modules\jest\node_modules\jest-cli\build\init\index.js 
- C:\Users\...\node_modules\jest\node_modules\jest-cli\build\cli\index.js  
- C:\Users\...\node_modules\jest\node_modules\jest-cli\build\index.js      
- C:\Users\...\node_modules\jest\node_modules\jest-cli\bin\jest.js
- C:\Users\...\node_modules\jest\bin\jest.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (C:\Users\...\node_modules\jest-cucumber\dist\src\reporting\report-event-generation\ReportEventGenerator.js:6:26)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)

any other way to generate json reports for jest-cucumber?

[PriyankaKoparkar]

@s2oBCN - Any tags in your feature files currently will output to the JSON file as expected. Is that what you were asking?

For example, consider this feature file with these tags:

@feature-test
Feature: Rocket Launching

@scenario-test
Scenario: Launching a SpaceX rocket
...

The following JSON will be generated:

  {
    "keyword": "Feature",
    "name": "Rocket Launching",
    "line": 2,
    "id": "rocket-launching",
    "tags": [
      {
        "name": "@feature-test",
        "line": 1
      }
    ],
    "uri": "/git/jest-cucumber/examples/typescript/specs/features/basic-scenarios.feature",
    "elements": [
      {
        "id": "rocket-launching;launching-a-spacex-rocket",
        "keyword": "Scenario",
        "line": 5,
        "name": "Launching a SpaceX rocket",
        "tags": [
          {
            "name": "@feature-test",
            "line": 1
          },
          {
            "name": "@scenario-test",
            "line": 4
          }
        ],
        "type": "scenario",
        "steps": [
...

How can I get this type of report using jest-cucumber?
I am looking for tags in the report.