Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename Pusher to Airnode feed #147

Merged
merged 1 commit into from
Dec 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
**/dist
**/node_modules
**/coverage
**/pusher.json
**/airnode-feed.json
**/secrets.env
**/signed-api.json
**/.DS_Store
6 changes: 3 additions & 3 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,14 @@ jobs:
run: pnpm run build
- name: Build Docker images
run: pnpm run docker:build
- name: Copy pusher secrets
run: cd packages/e2e/src && cp pusher/secrets.example.env pusher/secrets.env
- name: Copy Airnode feed secrets
run: cd packages/e2e/src && cp airnode-feed/secrets.example.env airnode-feed/secrets.env
- name: Start services
# Start the e2e services in the background and wait a small amount of time for them to start.
run: |
pnpm run --recursive --filter e2e start:data-provider-api &
pnpm run --recursive --filter e2e start:ci:signed-api &
pnpm run --recursive --filter e2e start:ci:pusher &
pnpm run --recursive --filter e2e start:ci:airnode-feed &
sleep 5
- name: Run e2e tests
run: pnpm run --recursive --filter e2e test:e2e
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ build
dist
node_modules
coverage
pusher.json
airnode-feed.json
secrets.env
signed-api.json
.DS_Store

# Do not ignore e2e config files. These don't need to be listed on Dockerignore because e2e package is not dockerized.
!packages/e2e/**/pusher.json
!packages/e2e/**/airnode-feed.json
!packages/e2e/**/.env
!packages/e2e/**/signed-api.json
28 changes: 14 additions & 14 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
# Note, that Dockerfile assumes the context (path) is the root of the monorepo in order to generate the common "build"
# stage.
#
# Debugging tips (assuming CWD = ./packages/pusher):
# 1. Build: docker build --target pusher --tag api3/pusher:latest ../../
# 2. Inspect: docker run -it --init -v $(pwd)/config:/app/config --env-file .env --entrypoint /bin/sh api3/pusher:latest
# Debugging tips (assuming CWD = ./packages/airnode-feed):
# 1. Build: docker build --target airnode-feed --tag api3/airnode-feed:latest ../../
# 2. Inspect: docker run -it --init -v $(pwd)/config:/app/config --env-file .env --entrypoint /bin/sh api3/airnode-feed:latest
# The above commands will allow you to inspect the output of the build stage. You can change the target to debug other
# stages and verify that the image is correct.

Expand All @@ -32,23 +32,23 @@ RUN pnpm install --recursive --prefer-offline
# Build all packages in the monorepo.
RUN pnpm run --recursive build

# Create a separate stage for pusher package. We create a temporary stage for deployment and then copy the result into
# the final stage. Only the production dependencies and package implementation is part of this last stage.
LABEL application="deployed-pusher" description="Deployed Pusher container"
# Create a separate stage for Airnode feed package. We create a temporary stage for deployment and then copy the result
# into the final stage. Only the production dependencies and package implementation is part of this last stage.
LABEL application="deployed-airnode-feed" description="Deployed Airnode feed container"

FROM build AS deployed-pusher
FROM build AS deployed-airnode-feed

RUN pnpm --filter=pusher --prod deploy deployed-pusher
FROM node:18-alpine as pusher
RUN pnpm --filter=airnode-feed --prod deploy deployed-airnode-feed
FROM node:18-alpine as airnode-feed
WORKDIR /app
ENV NODE_ENV=production

RUN addgroup -S deployed-pusher && \
adduser -h /app -s /bin/false -S -D -H -G deployed-pusher deployed-pusher && \
chown -R deployed-pusher /app
USER deployed-pusher
RUN addgroup -S deployed-airnode-feed && \
adduser -h /app -s /bin/false -S -D -H -G deployed-airnode-feed deployed-airnode-feed && \
chown -R deployed-airnode-feed /app
USER deployed-airnode-feed

COPY --chown=deployed-pusher:deployed-pusher --from=deployed-pusher /app/deployed-pusher .
COPY --chown=deployed-airnode-feed:deployed-airnode-feed --from=deployed-airnode-feed /app/deployed-airnode-feed .
ENTRYPOINT ["node", "dist/src/index.js"]

# Create a separate stage for api package. We create a temporary stage for deployment and then copy the result into
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ A monorepo for managing signed data. Consists of:

- [api](./packages/api/README.md) - A service for storing and accessing signed data. It provides endpoints to handle
signed data for a specific airnode.
- [pusher](./packages/pusher/README.md) - A service for pushing data provider signed data.
- [e2e](./packages/e2e/README.md) - End to end test utilizing Mock API, pusher and signed API.
- [airnode-feed](./packages/airnode-feed/README.md) - A service for pushing data provider signed data.
- [e2e](./packages/e2e/README.md) - End to end test utilizing Mock API, Airnode feed and signed API.

Read the
[specification](https://docs.google.com/document/d/1-kUPIXSD4ZW1SGs_P8HsejC9k9aHB-NXs9_6-OclnmE/edit#heading=h.i307237rdfda)
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
89 changes: 45 additions & 44 deletions packages/pusher/README.md → packages/airnode-feed/README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
# pusher
# airnode-feed

> A service for storing and accessing signed data.

Pusher is a Node.js service, dockerized and deployable on any cloud provider or hostable on premise. It is continuously
running two core loops:
Airnode feed is a Node.js service, dockerized and deployable on any cloud provider or hostable on premise. It is
continuously running two core loops:

1. `Fetch beacon data` - Each `triggers.signedApiUpdates` entry defines a group of templates. Pusher makes a template
request to the API specified in the OIS to get the template data. Pusher's wallet is used to sign the responses and
these are then saved to in-memory storage.
1. `Fetch beacon data` - Each `triggers.signedApiUpdates` entry defines a group of templates. Airnode feed makes a
template request to the API specified in the OIS to get the template data. Airnode feed's wallet is used to sign the
responses and these are then saved to in-memory storage.
2. `Push signed beacon data to signed API` - For each `triggers.signedApiUpdates`, periodically checks the in-memory
storage and pushes the signed data to the configured API.

## Local development

The pusher needs a configuration in order to run. The `config` folder contains example configuration which uses:
The Airnode feed needs a configuration in order to run. The `config` folder contains example configuration which uses:

- [Nodary](https://nodary.io/) as the data provider, from which the data is fetched.
- Signed API running on `http://localhost:8090` where the data is pushed.

To start the the pusher in dev mode run the following:
To start the the Airnode feed in dev mode run the following:

1. `cp config/pusher.example.json config/pusher.json` - To copy the pusher configuration from the example. Note, the
`pusher.json` file is ignored by git. If you are using Docker Desktop, you need to change the URL from localhost to
`host.docker.internal`. For example:
1. `cp config/airnode-feed.example.json config/airnode-feed.json` - To copy the Airnode feed configuration from the
example. Note, the `airnode-feed.json` file is ignored by git. If you are using Docker Desktop, you need to change
the URL from localhost to `host.docker.internal`. For example:

```jsonc
"url": "http://host.docker.internal:8090"
Expand All @@ -32,8 +32,8 @@ To start the the pusher in dev mode run the following:
is also ignored by git.
3. Set the `NODARY_API_KEY` inside the secrets file. Ask someone from development team for the key.
4. `cp .env.example .env` - To copy the example environment variables. Optionally change the defaults.
5. `pnpm run dev` - To run the pusher. This step assumes already running signed API as specified in the `pusher.json`
configuration.
5. `pnpm run dev` - To run the Airnode feed. This step assumes already running signed API as specified in the
`airnode-feed.json` configuration.

### Testing

Expand All @@ -59,7 +59,7 @@ pnpm run docker:run

## Configuration

Pusher can be configured via a combination of [environment variables](#environment-variables) and
Airnode feed can be configured via a combination of [environment variables](#environment-variables) and
[configuration files](#configuration-files).

### Environment variables
Expand Down Expand Up @@ -131,9 +131,9 @@ Default: `info`.

### Configuration files

Pusher needs two configuration files, `pusher.json` and `secrets.env`. All expressions of a form `${SECRET_NAME}` are
referring to values from secrets and are interpolated inside the `pusher.json` at runtime. You are advised to put
sensitive information inside secrets.
Airnode feed needs two configuration files, `airnode-feed.json` and `secrets.env`. All expressions of a form
`${SECRET_NAME}` are referring to values from secrets and are interpolated inside the `airnode-feed.json` at runtime.
You are advised to put sensitive information inside secrets.

You can also refer to the [example configuration](./config).

Expand Down Expand Up @@ -252,14 +252,14 @@ For example:

##### `triggers.signedApiUpdates[n]`

Configuration for one of the signed API update triggers. Pusher periodically pushes the data to the signed API. The
period is `2.5` seconds.
Configuration for one of the signed API update triggers. Airnode feed periodically pushes the data to the signed API.
The period is `2.5` seconds.

Pusher only makes a single template request independently of the number of template IDs specified. This is to reduce the
number of data provider calls. This implies that all of the templates in the trigger must use the same endpoint and
parameters. You can use [OIS processing](https://dapi-docs.api3.org/reference/ois/latest/processing.html) to remove the
parameters before making the request (using pre-processing) and later get the corresponding template value based on the
endpoint parameters (using-processing). Refer to the [example configuration](./config) for details.
Airnode feed only makes a single template request independently of the number of template IDs specified. This is to
reduce the number of data provider calls. This implies that all of the templates in the trigger must use the same
endpoint and parameters. You can use [OIS processing](https://dapi-docs.api3.org/reference/ois/latest/processing.html)
to remove the parameters before making the request (using pre-processing) and later get the corresponding template value
based on the endpoint parameters (using-processing). Refer to the [example configuration](./config) for details.

###### `signedApiName`

Expand Down Expand Up @@ -321,11 +321,12 @@ Refer to Airnode's

#### `nodeSettings`

Contains general deployment parameters of the pusher.
Contains general deployment parameters of the Airnode feed.

##### `nodeVersion`

The version of the pusher. The version specified in the config must match the version of the pusher at deployment time.
The version of the Airnode feed. The version specified in the config must match the version of the Airnode feed at
deployment time.

##### `airnodeWalletMnemonic`

Expand All @@ -339,47 +340,47 @@ secrets. For example:

##### `stage`

An identifier of the deployment stage. This is used to distinguish between different deployments of pusher, for example
`dev`, `staging` or `production`. The stage value can have 256 characters at maximum and can only include lowercase
alphanumeric characters and hyphens.
An identifier of the deployment stage. This is used to distinguish between different deployments of Airnode feed, for
example `dev`, `staging` or `production`. The stage value can have 256 characters at maximum and can only include
lowercase alphanumeric characters and hyphens.

## Versioning and release

Pusher uses [semantic versioning](https://semver.org/). The version is specified in the `package.json` file. The package
is not published to NPM, but instead dockerized and published to Docker Hub. The image is called
[api3/pusher](https://hub.docker.com/r/api3/pusher).
Airnode feed uses [semantic versioning](https://semver.org/). The version is specified in the `package.json` file. The
package is not published to NPM, but instead dockerized and published to Docker Hub. The image is called
[api3/airnode-feed](https://hub.docker.com/r/api3/airnode-feed).

To release a new version:

1. `git checkout main` - Always version from `main` branch. Also, ensure that the working directory is clean (has no
uncommitted changes).
2. `cd packages/pusher` - Navigate to the pusher package.
2. `cd packages/airnode-feed` - Navigate to the Airnode feed package.
3. `pnpm version [major|minor|patch]` - Choose the right version bump. This will bump the version, create a git tag and
commit it.
4. `pnpm run docker:build` - Build the docker image with tag `api3/pusher:latest`.
5. `docker tag api3/pusher:latest api3/pusher:<MAJOR.MINOR.PATCH>` - Tag the image with the version. Replace the
`<MAJOR.MINOR.PATCH>` with the version you just bumped (copy it from `package.json`).
6. `docker push api3/pusher:latest && docker push api3/pusher:<MAJOR.MINOR.PATCH>` - Push the image upstream. Both the
latest and the versioned tag should be published.
4. `pnpm run docker:build` - Build the docker image with tag `api3/airnode-feed:latest`.
5. `docker tag api3/airnode-feed:latest api3/airnode-feed:<MAJOR.MINOR.PATCH>` - Tag the image with the version. Replace
the `<MAJOR.MINOR.PATCH>` with the version you just bumped (copy it from `package.json`).
6. `docker push api3/airnode-feed:latest && docker push api3/airnode-feed:<MAJOR.MINOR.PATCH>` - Push the image
upstream. Both the latest and the versioned tag should be published.
7. `git push --follow-tags` - Push the tagged commit upstream.

## Deployment

<!-- markdown-link-check-disable -->

To deploy pusher on AWS you can use the Cloud Formation template created by the API integrations team. The template can
be found in the private api-integrations repository
To deploy Airnode feed on AWS you can use the Cloud Formation template created by the API integrations team. The
template can be found in the private api-integrations repository
[here](https://github.com/api3dao/api-integrations/blob/main/data/cloudformation-template.json).

<!-- markdown-link-check-enable -->

To deploy on premise you can use the Docker image by reading the instructions below.

### Run pusher with Docker
### Run Airnode feed with Docker

To run the pusher docker image you need to:
To run the Airnode feed docker image you need to:

1. Mount config folder to `/app/config`. The folder should contain the `pusher.json` and `secrets.env` files.
1. Mount config folder to `/app/config`. The folder should contain the `airnode-feed.json` and `secrets.env` files.
2. Pass the `-it --init` flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See
[this](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals) for details.
3. Specify the `--env-file` with the path to the `.env` file containing the [ENV configuration](#environment-variables).
Expand All @@ -393,5 +394,5 @@ For example:

```sh
# Assuming the current folder contains the "config" folder and ".env" file.
docker run -it --init --volume $(pwd)/config:/app/config --env-file .env --rm api3/pusher:latest
docker run -it --init --volume $(pwd)/config:/app/config --env-file .env --rm api3/airnode-feed:latest
```
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "pusher",
"name": "airnode-feed",
"version": "0.1.0",
"engines": {
"node": "^18.18.2",
Expand All @@ -9,8 +9,8 @@
"build": "tsc --project tsconfig.build.json",
"clean": "rm -rf coverage dist",
"dev": "nodemon --ext ts,js,json,env --exec \"pnpm ts-node src/index.ts\"",
"docker:build": "docker build --target pusher --tag api3/pusher:latest ../../",
"docker:run": "docker run -it --init --volume $(pwd)/config:/app/config --network host --env-file .env --rm api3/pusher:latest",
"docker:build": "docker build --target airnode-feed --tag api3/airnode-feed:latest ../../",
"docker:run": "docker run -it --init --volume $(pwd)/config:/app/config --network host --env-file .env --rm api3/airnode-feed:latest",
"eslint:check": "eslint . --ext .js,.ts --max-warnings 0",
"eslint:fix": "eslint . --ext .js,.ts --fix",
"prettier:check": "prettier --check \"./**/*.{js,ts,md,yml,json}\"",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ describe(logHeartbeat.name, () => {
signature:
'0xd0f84a62705585e03c14ebfda2688a4e4c733aa42a3d13f98d4fe390d482fd1c3e698e91939df28b9565a9de82f3495b053824ffe61239fdd4fa0e4a970523a81b',
};
const rawConfig = JSON.parse(readFileSync(join(__dirname, '../../config/pusher.example.json'), 'utf8'));
const rawConfig = JSON.parse(readFileSync(join(__dirname, '../../config/airnode-feed.example.json'), 'utf8'));
jest.spyOn(configModule, 'loadRawConfig').mockReturnValue(rawConfig);
const state = stateModule.getInitialState(config);
jest.spyOn(stateModule, 'getState').mockReturnValue(state);
Expand Down Expand Up @@ -65,7 +65,7 @@ describe(verifyHeartbeatLog.name, () => {
};
// The config hash is taken from config with all spaces removed.
const rawConfig = JSON.stringify(
JSON.parse(readFileSync(join(__dirname, '../../config/pusher.example.json'), 'utf8'))
JSON.parse(readFileSync(join(__dirname, '../../config/airnode-feed.example.json'), 'utf8'))
);

expect(() => verifyHeartbeatLog(jsonLog.context, rawConfig)).not.toThrow();
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ import dotenv from 'dotenv';
import { configSchema } from './schema';
import { interpolateSecrets, parseSecrets } from './utils';

// When pusher is built the "/dist" file contains "src" folder and "package.json" and the config is expected to be
// When Airnode feed is built the "/dist" file contains "src" folder and "package.json" and the config is expected to be
// located next to the "/dist" folder. When run in development, the config is expected to be located next to the "src"
// folder (one less import level). We resolve the config by CWD as a workaround. Since the pusher is dockerized, this
// is hidden from the user.
// folder (one less import level). We resolve the config by CWD as a workaround. Since the Airnode feed is dockerized,
// this is hidden from the user.
const getConfigPath = () => join(cwd(), './config');

export const loadRawConfig = () => JSON.parse(fs.readFileSync(join(getConfigPath(), 'pusher.json'), 'utf8'));
export const loadRawConfig = () => JSON.parse(fs.readFileSync(join(getConfigPath(), 'airnode-feed.json'), 'utf8'));

export const loadConfig = async () => {
const goLoadConfig = await go(async () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { type Config, configSchema, signedApisSchema } from './schema';
import { interpolateSecrets } from './utils';

test('validates example config', async () => {
const exampleConfig = JSON.parse(readFileSync(join(__dirname, '../../config/pusher.example.json'), 'utf8'));
const exampleConfig = JSON.parse(readFileSync(join(__dirname, '../../config/airnode-feed.example.json'), 'utf8'));

// The mnemonic is not interpolated (and thus invalid).
await expect(configSchema.parseAsync(exampleConfig)).rejects.toStrictEqual(
Expand All @@ -29,7 +29,7 @@ test('validates example config', async () => {
);
});

test('ensures nodeVersion matches pusher version', async () => {
test('ensures nodeVersion matches Airnode feed version', async () => {
const invalidConfig: Config = {
...config,
nodeSettings: {
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion packages/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ AWS_S3_BUCKET_NAME=my-config-bucket
AWS_S3_BUCKET_PATH=configs/my-app/signed-api.json
```

<!-- NOTE: Keep the logger configuration in-sync with pusher. -->
<!-- NOTE: Keep the logger configuration in-sync with Airnode feed package. -->

#### `LOGGER_ENABLED` _(optional)_

Expand Down
2 changes: 1 addition & 1 deletion packages/api/src/handlers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ export const batchInsertData = async (requestBody: unknown): Promise<ApiResponse
if (firstError) return firstError;

const newSignedData: SignedData[] = [];
// Because pushers do not keep track of the last timestamp they pushed, they may push the same data twice, which
// Because Airnode feed does not keep track of the last timestamp they pushed, it may push the same data twice, which
// is acceptable, but we only want to store one data for each timestamp.
for (const signedData of batchSignedData) {
const requestTimestamp = Number.parseInt(signedData.timestamp, 10);
Expand Down
2 changes: 1 addition & 1 deletion packages/e2e/.eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ build
dist
node_modules
coverage
pusher.json
airnode-feed.json
secrets.env
signed-api.json
.DS_Store
Loading