Allow API to serve delayed data (#34)

* Save all beacon signed data, not just the freshest

* Implement delayed endpoint

* Remove .DS_Store

* Improve error messages

* Use cache value from config

* Implement in-memory-api tests

* Add better error messages

* Verify that express json body parser works

* Fix docker compose, edit documentation

* Remove unused dependency

* Make each endpoint unique, document configuration

* Fix tests

.dockerignore

@@ -17,3 +17,5 @@
 **/coverage
 **/pusher.json
 **/secrets.env
+**/signed-api.json
+**/.DS_Store

.eslintrc.js

@@ -65,5 +65,8 @@ module.exports = {
     'no-useless-escape': 'off',
     semi: 'error',
     eqeqeq: ['error', 'smart'],
+
+    // Jest
+    'jest/valid-title': 'off', // Prevents using "<function-name>.name" as a test name
   },
 };

.gitignore

@@ -11,3 +11,5 @@ node_modules
 coverage
 pusher.json
 secrets.env
+signed-api.json
+.DS_Store

jest.config.js

@@ -3,7 +3,7 @@
  * https://jestjs.io/docs/configuration
  */
 module.exports = {
-  clearMocks: true,
+  restoreMocks: true,
   collectCoverage: true,
   coverageDirectory: 'coverage',
   coverageProvider: 'v8',

packages/api/README.md

@@ -4,21 +4,52 @@ A service for storing and accessing signed data. It provides endpoints to handle
 
 ## Local development
 
-1. `cp .env.example .env` - To copy `.env` from the `example.env` file. Optionally change the defaults.
-2. `pnpm run dev` - To start the API server. The port number can be configured in the `.env` file.
+1. `cp config/signed-api.example.json config/signed-api.json` - To create a config file from the example one. Optionally
+   change the defaults.
+2. `pnpm run dev` - To start the API server. The port number can be configured in the configuration file.
 
 ## Deployment
 
 TODO: Write example how to deploy on AWS (and maybe other cloud providers as well).
 
+## Configuration
+
+The API is configured via `signed-api.json`. You can use this file to specify the port of the server, configure cache
+header longevity or maximum batch size the API accepts.
+
+### Configuring endpoints
+
+The API needs to be configured with endpoints to be served. This is done via the `endpoints` section. For example:
+
+```json
+  "endpoints": [
+    {
+      "urlPath": "/real-time",
+      "delaySeconds": 0
+    },
+    {
+      "urlPath": "/delayed",
+      "delaySeconds": 15
+    }
+  ],
+```
+
+defines two endpoints. The `/real-time` serves the non-delayed data, the latter (`/delayed`) ignores all signed data
+that has bee pushed in the last 15 seconds (configured by `delaySeconds` parameter). You can define multiple endpoints
+as long as the `urlPath` is unique.
+
 ## Usage
 
 The API provides the following endpoints:
 
-- `PUT /`: Upsert single signed data.
-- `POST /`: Upsert batch of signed data.
-- `GET /{airnode}`: Retrieve signed data for the airnode.
-- `GET /`: Retrieve list of all available airnode address.
+- `POST /`: Insert a batch of signed data.
+  - The batch is validated for consistency and data integrity errors. If there is any issue during this step, the whole
+    batch is rejected. Otherwise the batch is accepted.
+- `GET /{endpoint-name}/{airnode}`: Retrieve signed data for the Airnode respecting the endpoint configuration.
+  - Only returns the freshest signed data available for the given Airnode, respecting the configured endpoint delay.
+- `GET /`: Retrieve list of all available Airnode address.
+  - Returns all Airnode addresses for which there is signed data. It is possible that this data cannot be shown by the
+    delayed endpoints (in case the data is too fresh and there is not an older alternative).
 
 ## Local development
 
@@ -30,34 +61,27 @@ pnpm run dev
 
 ## Docker
 
-The API is also dockerized. In order to run the API from a docker, run:
+The API is also dockerized. Docker needs to publish the port of the server (running inside the docker) to the port on
+the host machine. By default, it expects the server is running on port `8090` and publishes it to the `8090` on the
+host. To change this, you need to modify both `signed-api.json` configuration and use `PORT` environment variable with
+the same value.
+
+In order to run the API from a docker, run:
 
 ```bash
-# starts the API on port 4000
+# Starts the API on port 8090
 pnpm run docker:start
-# or in a detached mode
+# Or in a detached mode
 pnpm run docker:detach:start
-# optionally specify port
+# Optionally specify port (also make sure the same port is specified inside `signed-api.json`)
 PORT=5123 pnpm run docker:start
 ```
 
 ### Examples
 
-Here are some examples of how to use the API with `curl`. Note, the port may differ based on the `.env` value.
+Here are some examples of how to use the API with `curl`. Note, the port may differ based on the configuration.
 
 ```bash
-# Upsert signed data (HTTP PUT)
-curl --location --request PUT 'http://localhost:8090' \
---header 'Content-Type: application/json' \
---data '{
-    "airnode": "0xc52EeA00154B4fF1EbbF8Ba39FDe37F1AC3B9Fd4",
-    "beaconId": "0x70601427c8ff03560563917eed9837651ad9d6eb3414e46e8f96302c6f0aefcd",
-    "templateId": "0x8f255387c5fdb03117d82372b8fa5c7813881fd9a8202b7cc373f1a5868496b2",
-    "timestamp": "1694644051",
-    "encodedValue": "0x000000000000000000000000000000000000000000000002eb268c108b0b1da0",
-    "signature": "0x8e540abb31f6ef161153c508b9cc3909dcc3cf6596deff88ed4f9f2226fa28c61b8c23078373f64a7125035d1f70fd3befa6dfc48a31e7e15cc23133331ed9221b"
-  }'
-
 # Upsert batch of signed data (HTTP POST)
 curl --location 'http://localhost:8090' \
 --header 'Content-Type: application/json' \
@@ -78,7 +102,7 @@ curl --location 'http://localhost:8090' \
   }]'
 
 # Get data for the airnode address (HTTP GET)
-curl --location 'http://localhost:8090/0xc52EeA00154B4fF1EbbF8Ba39FDe37F1AC3B9Fd4' \
+curl --location 'http://localhost:8090/real-time/0xc52EeA00154B4fF1EbbF8Ba39FDe37F1AC3B9Fd4' \
 --header 'Content-Type: application/json'
 
 # List available airnode addresses (HTTP GET)

packages/api/config/signed-api.example.json

@@ -0,0 +1,17 @@
+{
+  "endpoints": [
+    {
+      "urlPath": "/real-time",
+      "delaySeconds": 0
+    },
+    {
+      "urlPath": "/delayed",
+      "delaySeconds": 15
+    }
+  ],
+  "maxBatchSize": 10,
+  "port": 8090,
+  "cache": {
+    "maxAgeSeconds": 300
+  }
+}

packages/api/docker/docker-compose.yml

@@ -6,7 +6,18 @@ services:
       context: ../../../
       dockerfile: ./packages/api/docker/Dockerfile
     ports:
-      - '${PORT:-4000}:${PORT:-4000}'
+      - '${PORT-8090}:${PORT:-8090}'
     environment:
       - NODE_ENV=production
-      - PORT=${PORT:-4000}
+    volumes:
+      # Mount the config file path to the container /dist/config folder where the API expects it to be.
+      #
+      # Environment variables specified in the docker-compose.yml file are applied to all operations that affect the
+      # service defined in the compose file, including the build process ("docker compose build"). Docker Compose will
+      # expect that variable to be defined in the environment where it's being run, and it will throw an error if it is
+      # not. For this reason we provide a default value.
+      #
+      # Docker Compose doesn't allow setting default values based on the current shell's environment, so we can't access
+      # current working directory and default to "$(pwd)/config". For this reason we default to the config folder
+      # relative to the docker-compose.yml file.
+      - ${CONFIG_PATH:-../config}:/usr/src/app/packages/api/dist/config

packages/api/package.json

@@ -18,7 +18,7 @@
     "eslint:fix": "eslint . --ext .js,.ts --fix",
     "prettier:check": "prettier --check \"./**/*.{js,ts,md,json}\"",
     "prettier:fix": "prettier --write \"./**/*.{js,ts,md,json}\"",
-    "test": "jest --passWithNoTests"
+    "test": "jest"
   },
   "license": "MIT",
   "devDependencies": {
@@ -28,7 +28,6 @@
   },
   "dependencies": {
     "@api3/promise-utils": "0.4.0",
-    "dotenv": "^16.3.1",
     "ethers": "^5.7.2",
     "express": "^4.18.2",
     "lodash": "^4.17.21",

packages/api/src/cache.ts

@@ -0,0 +1,18 @@
+import { SignedData } from './schema';
+
+type SignedDataCache = Record<
+  string, // Airnode ID.
+  Record<
+    string, // Template ID.
+    SignedData[] // Signed data is ordered by timestamp (oldest first).
+  >
+>;
+
+let signedDataCache: SignedDataCache = {};
+
+// Making this a getter function makes it easier to mock the cache in storage.
+export const getCache = () => signedDataCache;
+
+export const setCache = (cache: SignedDataCache) => {
+  signedDataCache = cache;
+};

packages/api/src/constants.ts

@@ -1,5 +1,3 @@
-export const MAX_BATCH_SIZE = parseInt(process.env.MAX_BATCH_SIZE as string);
-
 export const COMMON_HEADERS = {
   'content-type': 'application/json',
   'access-control-allow-origin': '*',

packages/api/src/evm.ts

@@ -1,5 +1,5 @@
 import { ethers } from 'ethers';
-import { SignedData } from './types';
+import { SignedData } from './schema';
 
 export const decodeData = (data: string) => ethers.utils.defaultAbiCoder.decode(['int256'], data);
 

packages/api/src/handlers.test.ts

@@ -0,0 +1,165 @@
+import { readFileSync } from 'fs';
+import { join } from 'path';
+import { omit } from 'lodash';
+import * as cacheModule from './cache';
+import * as utilsModule from './utils';
+import { batchInsertData, getData, listAirnodeAddresses } from './handlers';
+import { createSignedData, generateRandomWallet } from '../test/utils';
+
+afterEach(() => {
+  cacheModule.setCache({});
+});
+
+beforeEach(() => {
+  jest
+    .spyOn(utilsModule, 'getConfig')
+    .mockImplementation(() => JSON.parse(readFileSync(join(__dirname, '../config/signed-api.example.json'), 'utf8')));
+});
+
+describe(batchInsertData.name, () => {
+  it('drops the batch if it is invalid', async () => {
+    const invalidData = await createSignedData({ signature: '0xInvalid' });
+    const batchData = [await createSignedData(), invalidData];
+
+    const result = await batchInsertData(batchData);
+
+    expect(result).toEqual({
+      body: JSON.stringify({
+        message: 'Unable to recover signer address',
+        detail:
+          'signature missing v and recoveryParam (argument="signature", value="0xInvalid", code=INVALID_ARGUMENT, version=bytes/5.7.0)',
+        extra: invalidData,
+      }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'content-type': 'application/json',
+      },
+      statusCode: 400,
+    });
+    expect(cacheModule.getCache()).toEqual({});
+  });
+
+  it('inserts the batch if data is valid', async () => {
+    const batchData = [await createSignedData(), await createSignedData()];
+
+    const result = await batchInsertData(batchData);
+
+    expect(result).toEqual({
+      body: JSON.stringify({ count: 2 }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'content-type': 'application/json',
+      },
+      statusCode: 201,
+    });
+    expect(cacheModule.getCache()).toEqual({
+      [batchData[0]!.airnode]: {
+        [batchData[0]!.templateId]: [batchData[0]],
+      },
+      [batchData[1]!.airnode]: {
+        [batchData[1]!.templateId]: [batchData[1]],
+      },
+    });
+  });
+});
+
+describe(getData.name, () => {
+  it('drops the request if the airnode address is invalid', async () => {
+    const batchData = [await createSignedData(), await createSignedData()];
+    await batchInsertData(batchData);
+
+    const result = await getData('0xInvalid', 0);
+
+    expect(result).toEqual({
+      body: JSON.stringify({ message: 'Invalid request, airnode address must be an EVM address' }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'content-type': 'application/json',
+      },
+      statusCode: 400,
+    });
+  });
+
+  it('returns the live data', async () => {
+    const airnodeWallet = generateRandomWallet();
+    const batchData = [await createSignedData({ airnodeWallet }), await createSignedData({ airnodeWallet })];
+    await batchInsertData(batchData);
+
+    const result = await getData(airnodeWallet.address, 0);
+
+    expect(result).toEqual({
+      body: JSON.stringify({
+        count: 2,
+        data: {
+          [batchData[0]!.beaconId]: omit(batchData[0], 'beaconId'),
+          [batchData[1]!.beaconId]: omit(batchData[1], 'beaconId'),
+        },
+      }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'cache-control': 'no-store',
+        'cdn-cache-control': 'max-age=10',
+        'content-type': 'application/json',
+      },
+      statusCode: 200,
+    });
+  });
+
+  it('returns the delayed data', async () => {
+    const airnodeWallet = generateRandomWallet();
+    const delayTimestamp = (Math.floor(Date.now() / 1000) - 60).toString(); // Delayed by 60 seconds
+    const batchData = [
+      await createSignedData({ airnodeWallet, timestamp: delayTimestamp }),
+      await createSignedData({ airnodeWallet }),
+    ];
+    await batchInsertData(batchData);
+
+    const result = await getData(airnodeWallet.address, 30);
+
+    expect(result).toEqual({
+      body: JSON.stringify({
+        count: 1,
+        data: {
+          [batchData[0]!.beaconId]: omit(batchData[0], 'beaconId'),
+        },
+      }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'cache-control': 'no-store',
+        'cdn-cache-control': 'max-age=10',
+        'content-type': 'application/json',
+      },
+      statusCode: 200,
+    });
+  });
+});
+
+describe(listAirnodeAddresses.name, () => {
+  it('returns the list of airnode addresses', async () => {
+    const airnodeWallet = generateRandomWallet();
+    const batchData = [await createSignedData({ airnodeWallet }), await createSignedData({ airnodeWallet })];
+    await batchInsertData(batchData);
+
+    const result = await listAirnodeAddresses();
+
+    expect(result).toEqual({
+      body: JSON.stringify({
+        count: 1,
+        'available-airnodes': [airnodeWallet.address],
+      }),
+      headers: {
+        'access-control-allow-methods': '*',
+        'access-control-allow-origin': '*',
+        'cache-control': 'no-store',
+        'cdn-cache-control': 'max-age=300',
+        'content-type': 'application/json',
+      },
+      statusCode: 200,
+    });
+  });
+});