Test request data population (body & query string)

[noahdietz]

I am beginning work in my own fork on creating options for specifying the values of generated test request parameters in the body and query string, following a similar style to #91.

The options will take the form of JSON objects with values provided on a path-by-path, operation-by-operation basis. See the example below.

For a path specified as:

"/user": {
  "post": {
    "parameters": [
      {
        "name": "name",
        "in": "body",
        "description": "name of a new User",
        "required": true,
        "type": "string"
      }
     ]
    }
  }

the corresponding parameter data would be passed to the function call options as:

"bodyParams": {
  "/user": {
    "post": {
      "name": "John Doe"
    }
  }
}

[noahdietz]

Now have data population for:

• path params (from a separate branch of work)
• query string params
• body params

Will submit a PR from my fork.

[mm-gmbd]

Hey @noahdietz -- I'm coming back around to using STT and am in need of the exact same feature. Any chance this PR is ready, or can I jump in to help complete?

[mm-gmbd]

Actually, I just glanced at your branch and saw your implementation of adding optional parameters bodyParams and queryParams.

A couple of notes:

• Is there a way to provide different sets of inputs for requests that should succeed vs. fail? It looks like they are only definable by route (rather than by route and success/failure)
• Using tools like json-schema-faker, STT could automatically generate both pass/fail response bodies. This is more work, but definitely the version I'd like to see.

[noahdietz]

@mm-gmbd thanks for revisiting the project

• no, there is no way to differentiate data between passing & failing, although that would be much more desirable...I did this in my own fork as a sort of proof of concept for a paper i was writing. Wasn't planning on merging it because I am the only active owner of this repo, belonging to my employer's org.
• json-schema-faker looks pretty cool to me, how do you propose specifying successful/failure? An idea that just came to me was you use a handlebars helper, that takes the expected status code (2XX for success, everything else failure) and the validation schema of the request, then based on the status code, either the schema is used to generate a successful body, or ignore the schema and generate a generally failing body (unless you want to use the schema and fudge it, but thatd be more work)...thoughts?

[mm-gmbd]

I'm working on your second bullet right now 😄 .... unsuccessfully 😢

This requires somewhat of a fundamental shift in the way that STT approaches templating now. Currently, it generates templates by checking things in the following order:

1. Path (testGenPath)
2. Operation (testGenOperation)
3. Responses (testGenContentTypes --> testGenResponse)
4. EDIT: Added: Get Data for Response (getData)
5. For each response, check bodyVals

EDIT (add): I think the workflow for 4 and 5 needs to be changed. Need to do call getData, and then based on the data in data.bodyVals, generate responses.
EDIT: I think 3 and 4 need to be flipped. Also, as I'm writing this I'm starting to feel like this may not be so bad :)

Here is some sample input I am giving for a particular path, so you can see where my head is at:

var config = {
  assertionModule:
  testModule:
  ...
  bodyVals: {}
}

//This is where you could have a codegen add to `bodyVals` based on responses/schemas
config.bodyVals["a/path"] = [
  {
    "statusCode": 200,
    "inputs": [
       //acceptable input 1
       //acceptable input 2
     ]
  },
  {
    "statusCode": 400,
    "inputs": [
       //invalid input 1
       //invalid input 2
     ] 
  }
]

BTW -- I have made the decision that, for my API, an error code of 400 means that request validation has failed. All other failures will take on different HTTP error codes -- this means that for using STT, I know that my array of inputs for statusCode 400 means that they should all be invalid based on the JSON schema in the API.

EDIT (add): Another note is that, in my project currently, all request validation will be done on data contained in the request body, but obviously there should likely be validation done on the data in either the path or query parameters. I'm starting from the body, but bodyParams should probably be named something else... Also, we may want to create tests that have a valid query parameter, but an invalid body, or vice-versa, so maybe it should be something like the following:

config.inputTesting = {};
config.inputTesting = [
  {
    "statusCode": 200,
    "tests": [
       {
         "bodyVals": //acceptable request body input, if applicable
         "queryVals: //acceptable query vals, if applicable,
         "pathVals": //acceptable path parameters, if applicable
       }
     ]
  },
  {
    "statusCode": 400,
    "tests": [
      {
        "bodyVals": //invalid request body input, if applicable
        "queryVals": //invalid query vals, if applicable
        "pathVals": //invalid path parameters, if applicable
      }
    ]
  }

Thoughts?

[noahdietz]

I would like to avoid changing the traversal hierarchy if possible...my suggestion:

We take the syntactic approach you've suggested in the STT config, allowing specification of values based on status code, stored in an array keyed to the path....however we do the evaluation of the request parameters (i.e. a look up in the bodyVals, based on the status code && path being evaluated in the template) in the handlebar helper function bodify. This will also be a lot less work, less diffs and an easy PR.

I know I keep suggesting the handlebars helper, but they are the most configurable aspect of the module. The traversal should stay the same, but the information being passed on to the Handlebars compilers can be changed (with changes to the templates too), with the brunt of the data generation logic/work being handled by the helpers

Does this make sense? What do you think @mm-gmbd

[mm-gmbd]

Hmm, I understand what you're saying, but I'm not sure how to realize the goal of generation of multiple test cases per statuscode in the current structure, unless the templates were more heavily modified.

And actually, maybe it's not as impossible as I'm thinking...

For example, let's look at /templates/supertest/put/put.handebars. Right now, it generates a single test, starting as follows:

  it('should respond with {{length description}}', function(done) {
      {{#validateResponse returnType noSchema}}
      /*eslint-disable*/
      {{> schema-partial this}}

      /*eslint-enable*/
      {{/validateResponse}}
      api.put({{pathify path pathParams}})
      ...
      {{#is contentType 'application/json'}}
      .send({
        {{#each bodyParameters}}
        {{this.name}}: {{bodify ../../path this.name ../../bodyVals}}{{#unless @last}},{{/unless}}
        {{/each}}
      })
      {{/is}}
      ...
  })

Rather than passing bodyVals through the bodify helper, the template could be modified to immediately loop through the config.inputTesting for that particular path, something like:

1. Define inputTesting like the following: UPDATE: Added operation verbs

    config.inputTesting["a/path"] = {
      "PUT": {
        "200": [
          {
            "bodyVals": //good body
            "pathVals": //good path
            "queryVals": //good query
          }
        ],
        "400": [
          {
            "bodyVals": //bad body 1
          },
          {
            "bodyVals": //bad body 2
          },
        ]
      }
    }

1. Modify index.js to get the inputTesting parameters for that particular operation/statusCode combination

    var tempPath = (((swagger.basePath !== undefined) && (swagger.basePath !== '/'))
        ? swagger.basePath : '') + path;

    if (config.inputTesting && 
        config.inputTesting[tempPath] && 
        config.inputTesting[tempPath][operation.toUpperCase()] &&
        config.inputTesting[tempPath][operation.toUpperCase()][response]) {

      console.log("InputTesting was defined for path "+operation+":"+response+":"+tempPath);
      data.inputTesting = config.inputTesting[tempPath][operation.toUpperCase()][response];
    }

1. Modify template to look like the following:

    {{#if inputTesting}} //see if inputTesting is defined for this particular operation/statusCode combination
      {{#each inputTesting}}
        //Do standard template, but replace the request body with the provided bodyVals by passing them to `bodify`
      {{/each}}
    {{#else}}
      //Do the standard template, and indicate that the user needs to fill out the request body
    {{/if}

[Remco75]

Hi all, any progress on this? I am currently in the process of implementing the some kinda functionality using the swagMock module for data generation. Will push it soon and make a PR if anyones interested that is ;-)

[mm-gmbd]

I have implemented a version of this but haven't submitted a PR, and I think the version I ended up with is closer to the way @noahdietz was hoping it would be.

Rather than the way I wrote it in my last comment (modifying the template to include a big if/else), I slightly altered the codegen (but I think it still fits the current paradigm) to generate a test that has placeholders for user input or utilize the input testing, depending on if test data is provided for that path/method.

I can probably submit a PR in the near future (today/tomorrow).

[Remco75]

ha, cool!
Would lik to see the PR. Is it in a branch on github already? then I can check it out

[mm-gmbd]

Opened a PR and included a few comments: #107

Only has body parameter input testing implemented, not query params.