Merge pull request #32 from superfaceai/poc/monitoring-provider-changes

PoC: Monitoring provider changes

CHANGELOG.md

@@ -11,16 +11,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - New module for preparing files necessary for `perform` (SuperJson, ProfileAST, MapAST, ProviderJson)
 - New module for mocking necessary files for `perform`
 - Support hiding of credentials used with new security scheme Digest
+- New parameter `fullError` in method `run()` to enable returning whole `PerformError` instead of string
+- New static function `report` in `SuperfaceTest` to report found provider changes
+- Module `matcher` for comparing old and new HTTP traffic
+- Module `analyzer` for determining impact of provider changes
+- Module `reporter` for reporting provider changes throughout tests
+- Class `ErrorCollector` for collecting errors in `matcher`
+- Environment variable `UPDATE_TRAFFIC` to replace old traffic with new, if present
+- Environment variable `DISABLE_PROVIDER_CHANGES_COVERAGE` to disable collecting of test reports
+- Environment variable `USE_NEW_TRAFFIC` to test with newly recorded traffic
+- Errors for module `matcher`
+- Error `CoverageFileNotFoundError` for correct reporting
 
 ### Removed
 - Parameter `client` from constructor and method `run`
+- Function for omitting timestamp from perform error `removeTimestamp`
 
 ### Changed
 - **BREAKING CHANGE:** Updated One-SDK to [v2.0.0](https://github.com/superfaceai/one-sdk-js/releases/tag/v2.0.0)
 - **BREAKING CHANGE:** Use `BoundProfileProvider` instead of using client and use-case to run `perform` -> Local use only
 - Move functions used for recording in `SuperfaceTest` to seperate module
 - Use `SecurityConfiguration` (containing merged `SecurityValue` and `SecurityScheme` interfaces) instead of using them separately
 - Move parameter `testInstance` from superface components to second parameter in constructor
+- Return value from method `run` to `PerformError | string`
+- Does not overwrite HTTP traffic recording when in record mode, instead save new one next to old one with suffix `-new`
 
 ## [2.0.3] - 2022-02-15
 ### Changed

README.md

@@ -89,15 +89,16 @@ const superface = new SuperfaceTest(
     path: 'nock-recordings',
     fixture: 'my-recording',
     enableReqheadersRecording: true,
+    testInstance: expect
   }
 );
 ```
 
-Given nock configuration is also stored in class. Property `path` and `fixture` is used to configure location of recordings and property `enableReqheadersRecording` is used to enable/disable recording of request headers (This is turned off by default).
+Since it uses `nock` to record HTTP traffic during perform, second parameter in SuperfaceTest constructor is nock configuration containing `path` and `fixture` to configure location of recordings, property `enableReqheadersRecording` to enable/disable recording of request headers (This is turned off by default) and also property `testInstance` to enable testing library accessing current test names to generate unique hashes for recordings (currently supported only Jest and Mocha).
 
 ### Running
 
-To test your capabilities, use method `run()`, which encapsulates nock recording and UseCase perform. It expects test configuration (similar to initializing `SuperfaceTest` class) and input. You don't need to specify `profile`, `provider` or `useCase` if you already specified them when initializing `SuperfaceTest` class.
+To test your capabilities, use method `run()`, which encapsulates `nock` recording and `BoundProfileProvider` perform. It expects test configuration (similar to initializing `SuperfaceTest` class) and input. You don't need to specify `profile`, `provider` or `useCase` if you already specified them when initializing `SuperfaceTest` class.
 
 ```typescript
 import { SuperfaceTest } from '@superfaceai/testing';
@@ -123,16 +124,17 @@ import { SuperfaceTest } from '@superfaceai/testing';
 describe('test', () => {
   let superface: SuperfaceTest;
 
-  afterEach(() => {
-    superface = new SuperfaceTest();
+  beforeAll(() => {
+    superface = new SuperfaceTest({
+      profile: 'profile',
+      provider: 'provider',
+      useCase: 'useCase',
+    });
   });
 
   it('performs corretly', async () => {
     await expect(
       superface.run({
-        profile: 'profile',
-        provider: 'provider',
-        useCase: 'useCase',
         input: {
           some: 'input',
         },
@@ -142,13 +144,34 @@ describe('test', () => {
 });
 ```
 
-Method `run()` will initialize Superface client, transform all components that are represented by string to corresponding instances, check whether map is locally present based on super.json, runs perform for given usecase and returns **result** or **error** value from perform (More about perform in [One-SDK docs](https://github.com/superfaceai/one-sdk-js#performing-the-use-case)).
+Method `run()` initializes `BoundProfileProvider` class, runs perform for given usecase and returns **result** or **error** value from perform (More about perform in [One-SDK docs](https://github.com/superfaceai/one-sdk-js#performing-the-use-case)). Since testing library don't use `SuperfaceClient` anymore, it is **limited to local use only**. 
+
+[OneSDK 2.0](https://github.com/superfaceai/one-sdk-js/releases/tag/v2.0.0) does not contain parser anymore, so it looks for compiled files `.ast.json` next to original ones. To support this, parser was added to testing library and can be used to parse files when no AST is found.
+
+Method `run` also have second parameter, containing parameters to setup processing of recordings described bellow in [Recording](#recording).
+Only one parameter from this group processes result of method `run` and that is `fullError`. It enables method `run` to return full error from OneSDK instead of string.
+
+```typescript
+superface.run(
+  {
+    profile: 'profile',
+    provider: 'provider',
+    useCase: 'useCase',
+    input: {
+      some: 'input',
+    },
+  },
+  {
+    fullError: true,
+  }
+);
+```
 
 You can then use this return value to test your capabilities (We recommend you to use jest [snapshot testing](https://jestjs.io/docs/snapshot-testing) as seen in example above).
 
 ### Recording
 
-Method `run()` also records HTTP traffic with `nock` library during UseCase perform and saves recorded traffic to json file. Before perform, library will decide to record HTTP traffic based on environmental variable `SUPERFACE_LIVE_API` and current test configuration.
+Method `run()` also records HTTP traffic as we mentioned above and saves recorded traffic to json file. Before perform, library will decide to record HTTP traffic based on environmental variable `SUPERFACE_LIVE_API` and current test configuration.
 
 Variable `SUPERFACE_LIVE_API` specifies configuration which needs to be matched to record HTTP traffic.
 
@@ -216,7 +239,7 @@ superface.run(
 );
 ```
 
-You can also enter your own processing functions along side `processRecordings` parameter. Both have same function signature and are called either before load or before save of recordings.
+You can also enter your own processing functions along side `processRecordings` parameter. Both have same function signature and are called either before load or before save of recordings (see [this sequence diagram](./docs/sequence_diagram.png))
 
 ```typescript
 import { RecordingDefinitions, SuperfaceTest } from '@superfaceai/testing';
@@ -252,25 +275,63 @@ superface.run(
 );
 ```
 
-## Debug
+## Continuous testing
 
-You can use enviroment variable `DEBUG` to enable logging throughout testing process.
+Testing library supports continuous testing with live provider's traffic. This means that you can run testing library in record mode without worrying that old recording of traffic gets rewritten. Testing library compares old recording with new one and determines changes. If it find changes, it will save new traffic next to old one with suffix `-new`. 
+
+This recording represents new traffic and you can test your capabilities with it. First time it records new traffic, it also uses it for map and therefore you can see if map works with it, but we can also setup environment variable `USE_NEW_TRAFFIC=true` to mock new traffic instead of old one when not in record mode (it looks for recording with suffix `-new` next to old one).
+
+When you think the new recording is safe to use for testing, you can set it up as default with env variable `UPDATE_TRAFFIC=true`.
 
-`DEBUG="superface:testing*"` will enable all logging
+## Reporting
 
-`DEBUG="superface:testing"` will enable logging in `SuperfaceTest` class, its methods and utility functions
+To report found changes in traffic, you can implement your own function for reporting and pass it to `SuperfaceTest.report()`. It's signiture should be:
 
-`DEBUG="superface:testing:recordings"` will enable logging of processing sensitive information in recordings
+```typescript
+type TestReport = {
+  impact: MatchImpact;
+  profileId: string;
+  providerName: string;
+  useCaseName: string;
+  recordingPath: string;
+  input: NonPrimitive;
+  result: TestingReturn;
+  errors: ErrorCollection<string>;
+}[];
+
+type AlertFunction = (report: TestReport) => unknown | Promise<unknown>;
+```
+
+To disable collecting and also reporting these information, you can setup environment variable `DISABLE_PROVIDER_CHANGES_COVERAGE=true`.
+
+## Debug
+
+You can use enviroment variable `DEBUG` to enable logging throughout testing process.
 
-`DEBUG="superface:testing:recordings*"` or `DEBUG="superface:testing:recordings:sensitive"` will enable logging of replacing actual credentials
+- `DEBUG="superface:testing*"` will enable all logging
+- `DEBUG="superface:testing"` will enable logging of: 
+  - perform results
+  - start and end of recording/mocking HTTP traffic
+  - start of `beforeRecordingLoad` and `beforeRecordingSave` functions
+- `DEBUG=superface:testing:setup*` will enable logging of:
+  - setup of recording paths and superface components (profile, provider, usecase)
+  - setup of super.json and local map
+- `DEBUG=superface:testing:hash*` will enable logging of hashing recordings
+- `DEBUG="superface:testing:recordings"` will enable logging of processing sensitive information in recordings
+- `DEBUG="superface:testing:recordings*"` or `DEBUG="superface:testing:recordings:sensitive"` will also enable logging of replacing actual credentials
+- `DEBUG=superface:testing:matching*` enables logging of matching recordings
+- `DEBUG=superface:testing:reporter*` enables logging of reporting
+
+You can encounter `NetworkError` or `SdkExecutionError` during testing with mocked traffic, it usually means that request didn’t get through. If nock (used for loading mocked traffic) can’t match recording, request is denied. You can debug nock matching of recordings with `DEBUG=nock*` to see what went wrong.
 
 ## Known Limitations
 
 ### Multiple matching requests for the same use case
 
 Recordings make it possible to run tests without calling the live API. This works by trying to match a request to the requests in the existing recordings. If a match is found, the recorded response is returned. However, since the testing client saves recording for each test run in a single file, it means multiple matching requests for the same use-case and input will overwrite each other.
 
-A workaround is to use different inputs for each each test.
+To solve this, you can enter test instance (`expect` from jest) or specify custom hash phrase to differentiate between runs.
+Also a workaround is to use different inputs for each each test.
 
 ## Support
 

package.json

@@ -44,7 +44,10 @@
   },
   "dependencies": {
     "@superfaceai/ast": "^1.2.0",
+    "ajv": "^8.11.0",
     "debug": "^4.3.2",
+    "genson-js": "^0.0.8",
+    "http-encoding": "^1.5.1",
     "nock": "^13.1.3"
   },
   "peerDependencies": {

src/common/errors.test.ts

@@ -1,11 +1,14 @@
 import {
   assertIsIOError,
+  BaseURLNotFoundError,
   ComponentUndefinedError,
+  CoverageFileNotFoundError,
   InstanceMissingError,
   MapUndefinedError,
   ProfileUndefinedError,
   ProviderJsonUndefinedError,
   RecordingPathUndefinedError,
+  RecordingsNotFoundError,
   SuperJsonNotFoundError,
   UnexpectedError,
 } from './errors';
@@ -227,6 +230,59 @@ describe('errors', () => {
     });
   });
 
+  describe('when throwing RecordingsNotFoundError', () => {
+    const error = new RecordingsNotFoundError('path/to/recording.json');
+
+    it('throws in correct format', () => {
+      expect(() => {
+        throw error;
+      }).toThrow(
+        'Recordings could not be found for running mocked tests at "path/to/recording.json".\nYou must call the live API first to record API traffic.\nUse the environment variable SUPERFACE_LIVE_API to call the API and record traffic.\nSee https://github.com/superfaceai/testing#recording to learn more.'
+      );
+    });
+
+    it('returns correct format', () => {
+      expect(error.toString()).toEqual(
+        'RecordingsNotFoundError: Recordings could not be found for running mocked tests at "path/to/recording.json".\nYou must call the live API first to record API traffic.\nUse the environment variable SUPERFACE_LIVE_API to call the API and record traffic.\nSee https://github.com/superfaceai/testing#recording to learn more.'
+      );
+    });
+  });
+
+  describe('when throwing BaseURLNotFoundError', () => {
+    const error = new BaseURLNotFoundError('provider');
+
+    it('throws in correct format', () => {
+      expect(() => {
+        throw error;
+      }).toThrow(
+        'No base URL was found for provider "provider", configure a service in provider.json.'
+      );
+    });
+
+    it('returns correct format', () => {
+      expect(error.toString()).toEqual(
+        'BaseURLNotFoundError: No base URL was found for provider "provider", configure a service in provider.json.'
+      );
+    });
+  });
+
+  describe('when throwing CoverageFileNotFoundError', () => {
+    const samplePath = 'path/to/coverage.json';
+    const error = new CoverageFileNotFoundError(samplePath);
+
+    it('throws in correct format', () => {
+      expect(() => {
+        throw error;
+      }).toThrow(`No coverage file at path "${samplePath}" found.`);
+    });
+
+    it('returns correct format', () => {
+      expect(error.toString()).toEqual(
+        `CoverageFileNotFoundError: No coverage file at path "${samplePath}" found.`
+      );
+    });
+  });
+
   describe('when asserting error is IO error', () => {
     it('throws developer error correctly', async () => {
       expect(() => assertIsIOError(null)).toThrow(new UnexpectedError('null'));

src/common/errors.ts

@@ -1,20 +1,25 @@
 import { SDKExecutionError } from '@superfaceai/one-sdk';
 import { inspect } from 'util';
 
-class ErrorBase extends Error {
-  constructor(public kind: string, public override message: string) {
+export class ErrorBase extends Error {
+  constructor(kind: string, message: string) {
     super(message);
-    this.name = kind;
 
     Object.setPrototypeOf(this, ErrorBase.prototype);
+
+    this.name = kind;
   }
 
-  get [Symbol.toStringTag](): string {
-    return this.kind;
+  public get [Symbol.toStringTag](): string {
+    return this.name;
   }
 
-  override toString(): string {
-    return `${this.kind}: ${this.message}`;
+  public get kind(): string {
+    return this.name;
+  }
+
+  public override toString(): string {
+    return `${this.name}: ${this.message}`;
   }
 }
 
@@ -107,10 +112,10 @@ export class SuperJsonLoadingFailedError extends ErrorBase {
 }
 
 export class RecordingsNotFoundError extends ErrorBase {
-  constructor() {
+  constructor(path: string) {
     super(
       'RecordingsNotFoundError',
-      'Recordings could not be found for running mocked tests.\nYou must call the live API first to record API traffic.\nUse the environment variable SUPERFACE_LIVE_API to call the API and record traffic.\nSee https://github.com/superfaceai/testing#recording to learn more.'
+      `Recordings could not be found for running mocked tests at "${path}".\nYou must call the live API first to record API traffic.\nUse the environment variable SUPERFACE_LIVE_API to call the API and record traffic.\nSee https://github.com/superfaceai/testing#recording to learn more.`
     );
   }
 }
@@ -124,6 +129,15 @@ export class BaseURLNotFoundError extends ErrorBase {
   }
 }
 
+export class CoverageFileNotFoundError extends ErrorBase {
+  constructor(path: string) {
+    super(
+      'CoverageFileNotFoundError',
+      `No coverage file at path "${path}" found.`
+    );
+  }
+}
+
 export function assertIsIOError(
   error: unknown
 ): asserts error is { code: string } {

src/common/format.ts

@@ -1,11 +1,5 @@
 import { join as joinPath } from 'path';
 
-const ISO_DATE_REGEX =
-  /(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}(?:\.\d*)?)((-(\d{2}):(\d{2})|Z)?)/gm;
-
-export const removeTimestamp = (payload: string): string =>
-  payload.replace(ISO_DATE_REGEX, '');
-
 export function getFixtureName(
   profileId: string,
   providerName: string,