Validate example values for eachLike, eachKey, eachValue

[tienvx]

Because example values are not validated against matchers, I can define consumer test like this:

Consumer test (with invalid example value)

describe('Test', () => {
  test('User has full information', async () => {
      const user = {
          id: uuid(id),
          data: {
              gender: regex('(M|F|O)', gender),
              'first name': like(given),
              'last name': like(surname),
              birthday: regex(ISO8601_DATETIME_FORMAT, '2021-06-06T15:50:47+07:00'),
              avatar: url([])
          },
          rels: {
              spouses: {
                  value: [uuid(spouse)],
                  getValue: () => [uuid(spouse)],
                  'pact:matcher:type': 'type',
                  min: 0,
              },
              father: uuid(father),
              mother: uuid(mother),
              children: {
                  value: [uuid(child)],
                  getValue: () => [uuid(child)],
                  'pact:matcher:type': 'type',
                  min: 0,
              }
          }
      };
      pact.addInteraction({
          states: [{ description: 'Users with full information' }],
          uponReceiving: 'get list users with full information',
          withRequest: {
              method: 'GET',
              path: '/tree',
              headers: {
                  Accept: '*/*',
              }
          },
          willRespondWith: {
              status: 200,
              headers: {
                  'Content-Type': 'application/json',
              },
              body: {
                  min: 1,
                  'pact:matcher:type': 'type',
                  value: [
                      user,
                      {
                          id: father,
                          data: {
                              gender: 'M',
                              'first name': 'Father',
                          },
                          rels: {
                              spouses: [mother],
                              children: [id]
                          }
                      },
                      {
                          id: mother,
                          data: {
                              gender: 'F',
                              'first name': 'Mother',
                          },
                          rels: {
                              spouses: [father],
                              children: [id]
                          }
                      },
                      {
                          id: child,
                          data: {
                              gender: 'M',
                              'first name': 'Child',
                          },
                          rels: {
                              mother: id,
                              father: spouse
                          }
                      },
                      {
                          id: spouse,
                          data: {
                              gender: 'M',
                              'first name': 'Spouse',
                          },
                          rels: {
                              spouses: [id],
                          }
                      }
                  ],
              },
          },
      });

      await pact.executeTest(async (mockserver: V3MockServer) => {
          await waitPort({
              host: '127.0.0.1',
              port: mockserver.port,
          });
          await fetch(`${mockserver.url}/tree`);
      });
  });
});

NOTE: user is valid example value, but father, mother, child and spouse are invalid, because they are missing "father" and "mother" fields. The field "birthday" probably also has the same problem, but let ignore it for now.

Run this test and I got this pact file:

Pact (with invalid example value)

pact-with-invalid-example-values.json.txt

Provider response data like this:

Provider response body

[
{
  "data": {
    "avatar": "https://via.placeholder.com/640x480.png/0033ee?text=ab",
    "birthday": "1975-11-15T09:27:47+00:00",
    "first name": "Jalyn",
    "gender": "M",
    "last name": "Price"
  },
  "id": "79c4b8d7-5224-32cf-8ff1-ef0bcaedc1e3",
  "rels": {
    "children": [],
    "father": "f897f1b7-3867-318c-9f34-06480ffd0fab",
    "mother": "7973a31e-856e-3940-acbb-9e21e7f627bf",
    "spouses": []
  }
},
{
  "data": {
    "avatar": "https://via.placeholder.com/640x480.png/0011ee?text=maxime",
    "birthday": "1996-02-07T01:13:50+00:00",
    "first name": "Idella",
    "gender": "F",
    "last name": "Upton"
  },
  "id": "7973a31e-856e-3940-acbb-9e21e7f627bf",
  "rels": {
    "children": [
      "79c4b8d7-5224-32cf-8ff1-ef0bcaedc1e3"
    ],
    "father": null,
    "mother": null,
    "spouses": [
      "f897f1b7-3867-318c-9f34-06480ffd0fab"
    ]
  }
},
{
  "data": {
    "avatar": "https://via.placeholder.com/640x480.png/00dddd?text=aut",
    "birthday": "2017-07-24T14:35:27+00:00",
    "first name": "Jacklyn",
    "gender": "M",
    "last name": "Hahn"
  },
  "id": "f897f1b7-3867-318c-9f34-06480ffd0fab",
  "rels": {
    "children": [
      "79c4b8d7-5224-32cf-8ff1-ef0bcaedc1e3"
    ],
    "father": null,
    "mother": null,
    "spouses": [
      "7973a31e-856e-3940-acbb-9e21e7f627bf"
    ]
  }
}
]

Provider verification passed with this log (it should failed instead):

Provider verification log

provider verification log

Now removed all invalid example values from consumer test:

Consumer test (without invalid example values)

describe('Test', () => {
  test('User has full information', async () => {
      const user = {
          id: uuid(id),
          data: {
              gender: regex('(M|F|O)', gender),
              'first name': like(given),
              'last name': like(surname),
              birthday: regex(ISO8601_DATETIME_FORMAT, '2021-06-06T15:50:47+07:00'),
              avatar: url([])
          },
          rels: {
              spouses: {
                  value: [uuid(spouse)],
                  getValue: () => [uuid(spouse)],
                  'pact:matcher:type': 'type',
                  min: 0,
              },
              father: uuid(father),
              mother: uuid(mother),
              children: {
                  value: [uuid(child)],
                  getValue: () => [uuid(child)],
                  'pact:matcher:type': 'type',
                  min: 0,
              }
          }
      };
      pact.addInteraction({
          states: [{ description: 'Users with full information' }],
          uponReceiving: 'get list users with full information',
          withRequest: {
              method: 'GET',
              path: '/tree',
              headers: {
                  Accept: '*/*',
              }
          },
          willRespondWith: {
              status: 200,
              headers: {
                  'Content-Type': 'application/json',
              },
              body: eachLike(user),
          },
      });

      await pact.executeTest(async (mockserver: V3MockServer) => {
          await waitPort({
              host: '127.0.0.1',
              port: mockserver.port,
          });
          await fetch(`${mockserver.url}/tree`);
      });
  });
});

New pact generated (exactly the same, but without invalid example values):

Pact (without invalid example values)

pact-without-invalid-example-values.json.txt

Provider verification will failed as expected:

Provider verification log: expected

1) Verifying a pact between admin and relationship Given Users with full information - get list users with full information
  1.1) has a matching body
         $[1].rels.mother -> Expected '' to match '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
         $[1].rels.father -> Expected '' to match '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
         $[2].rels.father -> Expected '' to match '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
         $[2].rels.mother -> Expected '' to match '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'

Link to Slack discussion https://pact-foundation.slack.com/archives/C9VBGLUM9/p1690472580471449

[rholshausen]

Not sure I really understand, but what I assume is happening:

You set up a type matcher, with five examples, the first one has fields for "father" and "mother" while the other four do not.

This will not fail validation, because the other four values will be matched without the definition of those fields, so will ignore them.

I.e., this is setting up a template with five values, and those will be be applied to the actual five values, while the example that does fail has a template of one value, and that gets applied to all values.

So is this issue asking to reject any configuration where there is more than one example?

[tienvx]

You set up a type matcher, with five examples, the first one has fields for "father" and "mother" while the other four do not.

yes

This will not fail validation, because the other four values will be matched without the definition of those fields, so will ignore them.

It was fine.
I was trying to set the matchers once at the first example, and I don't have to re-defined matchers again for the remaining examples.
It worked as I expected, and I am happy about it.
It's my fault to provide invalid examples (missing the field father and mother). I thought it should not affect provider verification, but it affected. I believe this is a bug (see the Slack's discussion )
I was told to create this ticket instead. So because of that, I don't have to create another issue for that bug.

I.e., this is setting up a template with five values, and those will be be applied to the actual five values, while the example that does fail has a template of one value, and that gets applied to all values.

I'm not clear about this one yet.

So is this issue asking to reject any configuration where there is more than one example?

no, this issue is not about rejecting if there are more than 1 example. This issue is about rejecting if other 4 examples are incompatible with the first example. The first example defined father and mother must be uuid, but other examples are missing those fields, so they are not compatible with the first one, so they are should be rejected.

[rholshausen]

I'll try to explain what is going on, as this is not so much a bug as a deficiency in the FFI interface.

By default, when you say eachLike(A), that sets up A as a template to be applied to all items of the array. So for the second example, where you have [A, B, C, D, E] that fails are you expect as B, C, D and E are missing the required fields.

However, when you use eachLike([A, B, C, D, E]) that sets all those up as the template, so A -> A, B -> B, etc. is matched, and as B to E do not define those fields in the template, when they are compared to the actual values they will pass. I.e. B to E is not compared with A, which is what you were expecting.

However, the issue is the eachLike matcher has a dual purpose. It defines a template, but also creates the examples to use in the consumer test. I.e., eachLike(A) is really eachLike(T, E) where T is the template for A, and E is the example derived from A. What I think you meant to specify is eachLike(A, [A, B, C, D, E]) where A is the template, and the others are the examples. And this should fail when defined, as B to E do not match A. But I don't think you can express this with the FFI interface.