Skip to content

Commit

Permalink
Merge branch 'main' into metamorph-cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
@nozim
nozim committed Nov 16, 2023
2 parents 454dd36 + 9fe7f55 commit 1804d3b
Show file tree
Hide file tree
Showing 100 changed files with 3,568 additions and 597 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/go.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ jobs:
- name: Check generate api
run: make api && git diff --exit-code

- name: Launch test database
run: docker compose -f docker-compose.itest.yml up -d

- name: Run tests
run: go test -race -vet=off ./...

Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/static-analysis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ jobs:
with:
go-version-file: "./go.mod"

- name: Launch test database
run: docker compose -f docker-compose.itest.yml up -d

- name: Run unit Tests
continue-on-error: true
run: |
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,6 @@ cov.out
gosec-report.json
.scannerwork
gotest.out
report.xml
report.xml
init_env.sh
background_jobs/config.yaml
53 changes: 53 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# CHANGELOG

All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Table of Contents

- [Unreleased](#unreleased)
- [1.0.0 - YYYY-MM-DD](#100---yyyy-mm-dd)

## [Unreleased]

### Changed
- BREAKING CHANGE: The Merkle Path Binary format previously calculated and stored in BlockTx has been updated to a new standard encoding format referred to as BUMP and detailed here: [BRC-74](https://brc.dev/74). This means that the BlockTx database ought to be dumped prior to updating to using this version, since the structs are incompatible.

### Deprecated
- This has the effect of deprecating the previously used Merkle Path Binary format detailed here: [BRC-71](https://brc.dev/71) which is not used anywhere else in the ecosystem to our knowledge.

---

## [1.0.0] - YYYY-MM-DD

### Added
- Initial release

---

### Template for New Releases:

Replace `X.X.X` with the new version number and `YYYY-MM-DD` with the release date:

```
## [X.X.X] - YYYY-MM-DD
### Added
-
### Changed
-
### Deprecated
-
### Removed
-
### Fixed
-
### Security
-
```

Use this template as the starting point for each new version. Always update the "Unreleased" section with changes as they're implemented, and then move them under the new version header when that version is released.
13 changes: 8 additions & 5 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ clean_e2e_tests:
# Remove containers and images; avoid failure if the image doesn't exist
docker container prune -f
docker rmi test-tests || true

docker rmi test-arc || true

.PHONY: build_release
build_release:
Expand All @@ -25,8 +25,8 @@ build_release:

.PHONY: run_e2e_tests
run_e2e_tests:
cd ./test && docker-compose up -d node1 node2 node3 arc
cd ./test && docker-compose up --exit-code-from tests tests
cd ./test && docker-compose up -d node1 node2 node3
cd ./test && docker-compose up --exit-code-from tests tests arc
cd ./test && docker-compose down

.PHONY: test
Expand Down Expand Up @@ -122,8 +122,11 @@ api:
.PHONY: clean_restart_e2e_test
clean_restart_e2e_test: clean_e2e_tests build_release run_e2e_tests

migrate_postgres:
migrate -database "postgres://arcuser:arcpass@localhost:5432/arcdb?sslmode=disable" -path database/migrations/postgres up
migrate_postgres_blocktx:
migrate -database "postgres://${DB_USERNAME}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/blocktx_dev?sslmode=disable" -path database/migrations/blocktx/postgres up

migrate_postgres_metamorph:
migrate -database "postgres://${DB_USERNAME}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/metamorph_dev?sslmode=disable" -path database/migrations/metamorph/postgres up

migrate_sqlite:
migrate -path database/migrations/sqlite -database "sqlite3://data/sqlite/arcdb.sqlite3" up
60 changes: 41 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ ARC is a transaction processor for Bitcoin that keeps track of the life cycle of
Settings for ARC are defined in a configuration file. The default configuration file is `config.yaml` in the root directory. Each setting is documented in the file itself.
If you want to load `config.yaml` from a different location, you can specify it on the command line using the `-config=<path>` flag.

Each setting in the file `config.yaml` can be overridden with an environment variable. The environment variable needs to have this prefix `ARC_`. A sub setting will be separated using an underscore character. For example the following config setting could be overridden by the environment variable `ARC_METAMORPH_LISTENADDR`
```yaml
metamorph:
listenAddr:
```
## Microservices
To run all the microservices in one process (during development), use the `main.go` file in the root directory.
Expand Down Expand Up @@ -37,7 +43,7 @@ where options are:
-tracer=<true|false>
whether to start the Jaeger tracer (default=false)
-config=<path>
path to config file (default='')
```
Expand Down Expand Up @@ -144,20 +150,20 @@ The only difference between the two is that the generic `main.go` starts the Go
Metamorph keeps track of the lifecycle of a transaction, and assigns it a status. The following statuses are
available:

| Code | Status | Description |
|-----|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. |
| 1 | `QUEUED` | The transaction has been queued for processing. |
| 2 | `RECEIVED` | The transaction has been properly received by the metamorph processor. |
| 3 | `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. |
| 4 | `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. |
| 5 | `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. |
| 6 | `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. |
| 7 | `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZQM, this status will never by set. |
| 8 | `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. |
| 9 | `MINED` | The transaction has been mined into a block by a mining node. |
| 108 | `CONFIRMED` | The transaction is marked as confirmed when it is in a block with 100 blocks built on top of that block. |
| 109 | `REJECTED` | The transaction has been rejected by the Bitcoin network. |
| Code | Status | Description |
|-----|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | `UNKNOWN` | The transaction has been sent to metamorph, but no processing has taken place. This should never be the case, unless something goes wrong. |
| 1 | `QUEUED` | The transaction has been queued for processing. |
| 2 | `RECEIVED` | The transaction has been properly received by the metamorph processor. |
| 3 | `STORED` | The transaction has been stored in the metamorph store. This should ensure the transaction will be processed and retried if not picked up immediately by a mining node. |
| 4 | `ANNOUNCED_TO_NETWORK` | The transaction has been announced (INV message) to the Bitcoin network. |
| 5 | `REQUESTED_BY_NETWORK` | The transaction has been requested from metamorph by a Bitcoin node. |
| 6 | `SENT_TO_NETWORK` | The transaction has been sent to at least 1 Bitcoin node. |
| 7 | `ACCEPTED_BY_NETWORK` | The transaction has been accepted by a connected Bitcoin node on the ZMQ interface. If metamorph is not connected to ZMQ, this status will never by set. |
| 8 | `SEEN_ON_NETWORK` | The transaction has been seen on the Bitcoin network and propagated to other nodes. This status is set when metamorph receives an INV message for the transaction from another node than it was sent to. |
| 9 | `MINED` | The transaction has been mined into a block by a mining node. |
| 108 | `CONFIRMED` | The transaction is marked as confirmed when it is in a block with 100 blocks built on top of that block. |
| 109 | `REJECTED` | The transaction has been rejected by the Bitcoin network. |

This status is returned in the `txStatus` field whenever the transaction is queried.

Expand Down Expand Up @@ -273,11 +279,27 @@ go run cmd/broadcaster/main.go -api=false -consolidate -keyfile=./cmd/broadcaste
Detailed information about flags can is displayed by running `go run cmd/broadcaster/main.go`.

## Tests
### Unit tests
In order to run the unit tests do the following
```
make test
```
The end-to-end tests are located in the folder `test`. They can be run locally together with arc and 3 nodes using the provided docker-compose file. In order to run them do the following
### Integration tests
Integration tests of DynamoDB need docker installed to run them. If colima implementation of Docker is being used on Mac OS, the `DOCKER_HOST` environment variable may need to be given as follows
```bash
DOCKER_HOST=unix:///Users/<username>/.colima/default/docker.sock make test
```
These integration tests can be excluded from execution with `go test ./...` by adding the `-short` flag like this `go test -short ./...`.

### end-to-end tests
The end-to-end tests are located in the folder `test`. Docker needs to be installed in order to run them. End-to-end tests can be run locally together with arc and 3 nodes using the provided docker-compose file.
The tests can be executed like this:
```
make build_release
cd ./test
docker compose up --abort-on-container-exit --exit-code-from tests
make clean_restart_e2e_test
```


## Background jobs

See [here](https://github.com/bitcoin-sv/arc/tree/arc-background-jobs/background_jobs/README.md)
Loading

0 comments on commit 1804d3b

Please sign in to comment.