Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TOOLS-1469 #27

Open
wants to merge 4 commits into
base: rosetta-zen
Choose a base branch
from
Open

TOOLS-1469 #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
82018b4
Use constant value instead of hardcoded value in test
Oct 4, 2022
81cfa46
Added changes from rosetta-bitcoin
Oct 5, 2022
f67f9c7
Added LogLevel control system
Oct 7, 2022
5ef4283
Added LogLevel endpoints
Oct 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ rosetta-zen
zen-data
zen-testnet-data
cli-data
/http/
2 changes: 1 addition & 1 deletion .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ os: linux
dist: bionic
language: go
go:
- "1.17.2"
- "1.19.1"
services:
- docker
addons:
Expand Down
18 changes: 9 additions & 9 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,18 @@
# limitations under the License.

## Build zend
FROM ubuntu:18.04 as zend-builder
FROM ubuntu:20.04 as zend-builder

MAINTAINER [email protected]

SHELL ["/bin/bash", "-c"]

# Latest release zen 3.2.0
ARG ZEN_COMMITTISH=v3.2.0
ARG ZEN_COMMITTISH
ARG IS_RELEASE=false
# cronic <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/219F55740BBF7A1CE368BA45FB7053CE4991B669
# Luigi Varriale <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C
ARG MAINTAINER_KEYS="219F55740BBF7A1CE368BA45FB7053CE4991B669 FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C"
# otoumas <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/2BBE2AA1A641F6147B58450BE3527B60DAACA1D8
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too much info in the comments = security risk?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well these are our public keys, so I don't think it is a great risk

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that MAINTAINER_KEYS here is used to check who is allowed to sign the release tag on the zend repository. Not rosetta-zen.

Including PaoloT in this list would make more sense.

ARG MAINTAINER_KEYS="219F55740BBF7A1CE368BA45FB7053CE4991B669 FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C 2BBE2AA1A641F6147B58450BE3527B60DAACA1D8"

RUN set -euxo pipefail \
&& export DEBIAN_FRONTEND=noninteractive \
Expand Down Expand Up @@ -53,18 +53,18 @@ RUN set -euxo pipefail \
&& ( gpgconf --kill dirmngr || true ) \
&& ( gpgconf --kill gpg-agent || true ); \
fi \
&& export MAKEFLAGS="-j $(($(nproc)+1))" && ./zcutil/build.sh $MAKEFLAGS
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer making this a build time check, as legacy-cpu has a performance penalty. Any modern platform (cloud servers where Coinbase and we run rosetta-zen) supports the needed CPU flag. The only use case I see for building legacy-cpu images is on developer machines.

For detecting CPU support while running docker build an approach similar to this https://github.com/HorizenOfficial/zen-node-docker/blob/84c213dec9636d616721c971b74214ab75a66916/entrypoint.sh#L6 should work. Or https://manpages.ubuntu.com/manpages/focal/man1/lscpu.1.html can be used.

&& export MAKEFLAGS="-j $(($(nproc)+1))" && ./zcutil/build.sh --legacy-cpu $MAKEFLAGS


## Build Rosetta Server Components
FROM ubuntu:18.04 as rosetta-builder
FROM ubuntu:20.04 as rosetta-builder

MAINTAINER [email protected]

SHELL ["/bin/bash", "-c"]

ARG GOLANG_VERSION=1.17.2
ARG GOLANG_DOWNLOAD_SHA256=f242a9db6a0ad1846de7b6d94d507915d14062660616a61ef7c808a76e4f1676
ARG GOLANG_VERSION=1.19.1
ARG GOLANG_DOWNLOAD_SHA256=acc512fbab4f716a8f97a8b3fbaa9ddd39606a28be6c2515ef7c6c6311acffde
ARG GOLANG_DOWNLOAD_URL="https://golang.org/dl/go${GOLANG_VERSION}.linux-amd64.tar.gz"

COPY . /go/src
Expand All @@ -84,7 +84,7 @@ RUN set -euxo pipefail \


## Build Final Image
FROM ubuntu:18.04
FROM ubuntu:20.04

MAINTAINER [email protected]

Expand Down
49 changes: 37 additions & 12 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,44 @@
run-testnet-offline check-comments add-license check-license shorten-lines test \
coverage spellcheck salus build-local coverage-local format check-format

ADDLICENSE_CMD=go run github.com/google/addlicense
ADDLICENCE_SCRIPT=${ADDLICENSE_CMD} -c "Coinbase, Inc." -l "apache" -v
ADDLICENSE_INSTALL=go install github.com/google/addlicense@latest
ADDLICENSE_CMD=addlicense
ADDLICENSE_IGNORE=-ignore ".github/**/*" -ignore ".idea/**/*"
ADDLICENCE_SCRIPT=${ADDLICENSE_CMD} -c "Coinbase, Inc." -l "apache" -v ${ADDLICENSE_IGNORE}
SPELLCHECK_CMD=go run github.com/client9/misspell/cmd/misspell
GOLINES_CMD=go run github.com/segmentio/golines
GOLINT_CMD=go run golang.org/x/lint/golint
GOVERALLS_CMD=go run github.com/mattn/goveralls
GOLINES_INSTALL=go install github.com/segmentio/golines@latest
GOLINES_CMD=golines
GOLINT_INSTALL=go get golang.org/x/lint/golint
GOLINT_CMD=golint
GOVERALLS_INSTALL=go install github.com/mattn/goveralls@latest
GOVERALLS_CMD=goveralls
GOIMPORTS_CMD=go run golang.org/x/tools/cmd/goimports
GO_PACKAGES=./services/... ./indexer/... ./zen/... ./zend/... ./zenutil/... ./configuration/...
GO_FOLDERS=$(shell echo ${GO_PACKAGES} | sed -e "s/\.\///g" | sed -e "s/\/\.\.\.//g")
TEST_SCRIPT=go test ${GO_PACKAGES}
TEST_SCRIPT=go test ${GO_PACKAGES} -buildmode=pie
LINT_SETTINGS=golint,misspell,gocyclo,gocritic,whitespace,goconst,gocognit,bodyclose,unconvert,lll,unparam
PWD=$(shell pwd)
GZIP_CMD=$(shell command -v pigz || echo gzip)
NOFILE=100000
ZEND_VERSION=v3.2.0

deps:
go get ./...

build:
docker build --pull -t rosetta-zen:latest https://github.com/HorizenOfficial/rosetta-zen
docker build --pull --build-arg "ZEN_COMMITTISH=${ZEND_VERSION}" -t rosetta-zen:latest https://github.com/HorizenOfficial/rosetta-zen

build-local:
docker build --pull -t rosetta-zen:latest .
docker build --pull --build-arg "ZEN_COMMITTISH=${ZEND_VERSION}" -t rosetta-zen:latest .

build-release:
# make sure to always set version with vX.X.X
docker build --pull --no-cache --build-arg IS_RELEASE=true -t rosetta-zen:$(version) .;
docker build --pull --no-cache --build-arg IS_RELEASE=true --build-arg "ZEN_COMMITTISH=${ZEND_VERSION}" -t rosetta-zen:$(version) .;
docker save rosetta-zen:$(version) | ${GZIP_CMD} > rosetta-zen-$(version).tar.gz;

run-mainnet-online:
docker container rm rosetta-zen-mainnet-online || true
docker run --rm -v "${PWD}/zen-data:/data" ubuntu:18.04 bash -c 'chown -R nobody:nogroup /data';
docker run --rm -v "${PWD}/zen-data:/data" ubuntu:20.04 bash -c 'chown -R nobody:nogroup /data';
docker run -d --name=rosetta-zen-mainnet-online --ulimit "nofile=${NOFILE}:${NOFILE}" -v "${PWD}/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest;

run-mainnet-offline:
Expand All @@ -42,29 +48,46 @@ run-mainnet-offline:

run-testnet-online:
docker container rm rosetta-zen-testnet-online || true
docker run --rm -v "${PWD}/zen-data:/data" ubuntu:18.04 bash -c 'chown -R nobody:nogroup /data';
docker run -d --name=rosetta-zen-testnet-online --ulimit "nofile=${NOFILE}:${NOFILE}" -v "${PWD}/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 19033:19033 rosetta-zen:latest;
docker run --rm -v "${PWD}/zen-data:/data" ubuntu:20.04 bash -c 'chown -R nobody:nogroup /data';
docker run -d --name=rosetta-zen-testnet-online --ulimit "nofile=${NOFILE}:${NOFILE}" -v "${PWD}/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 19033:19033 -p 18231:18231 rosetta-zen:latest;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RPC ports like -p 18231:18231 should never be exposed to the internet, which -p does if the instance rosetta is running on has a publicly routable IP. The RPC interface is unencrypted, and even though there is authentication, there is no rate limiting against brute force attacks. --expose 18231 can be used instead which exposes the port to other containers on the same docker network, or -p 127.0.0.1:18231 to only expose the port on localhost.


run-testnet-offline:
docker container rm rosetta-zen-testnet-offline || true
docker run -d --name=rosetta-zen-testnet-offline -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 rosetta-zen:latest

stop-mainnet-online:
docker container stop rosetta-zen-mainnet-online

stop-mainnet-offline:
docker container stop rosetta-zen-mainnet-offline

stop-testnet-online:
docker container stop rosetta-zen-testnet-online

stop-testnet-offline:
docker container stop rosetta-zen-testnet-offline

train:
./zstd-train.sh $(network) transaction $(data-directory)

check-comments:
${GOLINT_INSTALL}
${GOLINT_CMD} -set_exit_status ${GO_FOLDERS} .
go mod tidy

lint: | check-comments
golangci-lint run --timeout 2m0s -v -E ${LINT_SETTINGS},gomnd

add-license:
${ADDLICENSE_INSTALL}
${ADDLICENCE_SCRIPT} .

check-license:
${ADDLICENSE_INSTALL}
${ADDLICENCE_SCRIPT} -check .

shorten-lines:
${GOLINES_INSTALL}
${GOLINES_CMD} -w --shorten-comments ${GO_FOLDERS} .

format:
Expand All @@ -79,6 +102,7 @@ test:
${TEST_SCRIPT}

coverage:
${GOVERALLS_INSTALL}
if [ "${COVERALLS_TOKEN}" ]; then ${TEST_SCRIPT} -coverprofile=c.out -covermode=count; ${GOVERALLS_CMD} -coverprofile=c.out -repotoken ${COVERALLS_TOKEN}; fi

coverage-local:
Expand All @@ -94,4 +118,5 @@ mocks:
rm -rf mocks;
mockery --dir indexer --all --case underscore --outpkg indexer --output mocks/indexer;
mockery --dir services --all --case underscore --outpkg services --output mocks/services;
${ADDLICENSE_INSTALL}
${ADDLICENCE_SCRIPT} .;
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ You can also use a named volume which will be created with the correct ownership
```text
# create <working directory>/zen-data with correct ownership
docker run --rm -v "$(pwd)/zen-data:/data" ubuntu:18.04 bash -c 'chown -R nobody:nogroup /data'
# start rosetta-zen
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest
# start rosetta-zen. Zend version needs to be specified (ex. v3.2.0)
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to specify NODE_VERSION? I can't see it being used anywhere. Zend version could also be detected on the fly in entypoint.sh by running zend -version | head -n1 | cut -d " " -f 4, or storing the var from the Makefile via ENV NODE_VERSION=$ARG in Dockerfile.

```
_If you cloned the repository, you can run `make run-mainnet-online`._

Expand All @@ -64,7 +64,7 @@ The zend configuration file can be extended by setting the docker command to /ap
and using the optional -extend-zen-conf="" switch. The value of -extend-zen-conf="" will be
appended to /app/zen-${NETWORK}.conf, newlines can be set as "\n".
```text
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest /app/rosetta-zen -extend-zen-conf="reindexfast=1\ndebug=rpc\ndebug=net"
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest /app/rosetta-zen -extend-zen-conf="reindexfast=1\ndebug=rpc\ndebug=net"
# this command line would append the following to /app/zen-mainnet.conf
reindexfast=1
debug=rpc
Expand All @@ -73,7 +73,7 @@ debug=net

#### Mainnet:Offline
```text
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 rosetta-zen:latest
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8081" -p 8081:8081 rosetta-zen:latest
```
_If you cloned the repository, you can run `make run-mainnet-offline`._

Expand All @@ -82,13 +82,13 @@ _If you cloned the repository, you can run `make run-mainnet-offline`._
# create <working directory>/zen-data with correct ownership
docker run --rm -v "$(pwd)/zen-data:/data" ubuntu:18.04 bash -c 'chown -R nobody:nogroup /data'
# start rosetta-zen
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 19033:19033 rosetta-zen:latest
docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8080" -p 8080:8080 -p 19033:19033 rosetta-zen:latest
```
_If you cloned the repository, you can run `make run-testnet-online`._

#### Testnet:Offline
```text
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 rosetta-zen:latest
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8081" -p 8081:8081 rosetta-zen:latest
```
_If you cloned the repository, you can run `make run-testnet-offline`._

Expand Down
15 changes: 8 additions & 7 deletions configuration/configuration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
"github.com/HorizenOfficial/rosetta-zen/zen"

"github.com/HorizenOfficial/rosetta-zen/zend/chaincfg"
"github.com/coinbase/rosetta-sdk-go/storage"
"github.com/coinbase/rosetta-sdk-go/storage/encoder"
"github.com/coinbase/rosetta-sdk-go/types"
)

Expand Down Expand Up @@ -88,8 +88,8 @@ const (
// persistent data.
DataDirectory = "/data"

zendPath = ".zen"
indexerPath = "indexer"
zendPath = ".zen"
indexerPath = "indexer"

// allFilePermissions specifies anyone can do anything
// to the file.
Expand Down Expand Up @@ -130,7 +130,8 @@ type Configuration struct {
Pruning *PruningConfiguration
IndexerPath string
ZendPath string
Compressors []*storage.CompressorEntry
ZendVersion string
Compressors []*encoder.CompressorEntry
}

// LoadConfiguration attempts to create a new Configuration
Expand Down Expand Up @@ -176,7 +177,7 @@ func LoadConfiguration(baseDirectory string) (*Configuration, error) {
config.Currency = zen.MainnetCurrency
config.ConfigPath = mainnetConfigPath
config.RPCPort = mainnetRPCPort
config.Compressors = []*storage.CompressorEntry{
config.Compressors = []*encoder.CompressorEntry{
{
Namespace: transactionNamespace,
DictionaryPath: mainnetTransactionDictionary,
Expand All @@ -192,7 +193,7 @@ func LoadConfiguration(baseDirectory string) (*Configuration, error) {
config.Currency = zen.TestnetCurrency
config.ConfigPath = testnetConfigPath
config.RPCPort = testnetRPCPort
config.Compressors = []*storage.CompressorEntry{
config.Compressors = []*encoder.CompressorEntry{
{
Namespace: transactionNamespace,
DictionaryPath: testnetTransactionDictionary,
Expand All @@ -208,7 +209,7 @@ func LoadConfiguration(baseDirectory string) (*Configuration, error) {
config.Currency = zen.TestnetCurrency
config.ConfigPath = regtestConfigPath
config.RPCPort = regtestRPCPort
config.Compressors = []*storage.CompressorEntry{
config.Compressors = []*encoder.CompressorEntry{
{
Namespace: transactionNamespace,
DictionaryPath: testnetTransactionDictionary,
Expand Down
19 changes: 10 additions & 9 deletions configuration/configuration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (

"github.com/HorizenOfficial/rosetta-zen/zen"

"github.com/coinbase/rosetta-sdk-go/storage"
"github.com/coinbase/rosetta-sdk-go/storage/encoder"
"github.com/coinbase/rosetta-sdk-go/types"
"github.com/coinbase/rosetta-sdk-go/utils"
"github.com/stretchr/testify/assert"
Expand All @@ -33,9 +33,8 @@ func TestLoadConfiguration(t *testing.T) {
Mode string
Network string
Port string

cfg *Configuration
err error
cfg *Configuration
err error
}{
"no envs set": {
err: errors.New("MODE must be populated"),
Expand Down Expand Up @@ -70,7 +69,8 @@ func TestLoadConfiguration(t *testing.T) {
Depth: pruneDepth,
MinHeight: minPruneHeight,
},
Compressors: []*storage.CompressorEntry{
ZendVersion: "",
Compressors: []*encoder.CompressorEntry{
{
Namespace: transactionNamespace,
DictionaryPath: mainnetTransactionDictionary,
Expand Down Expand Up @@ -99,7 +99,8 @@ func TestLoadConfiguration(t *testing.T) {
Depth: pruneDepth,
MinHeight: minPruneHeight,
},
Compressors: []*storage.CompressorEntry{
ZendVersion: "",
Compressors: []*encoder.CompressorEntry{
{
Namespace: transactionNamespace,
DictionaryPath: testnetTransactionDictionary,
Expand Down Expand Up @@ -133,9 +134,9 @@ func TestLoadConfiguration(t *testing.T) {
assert.NoError(t, err)
defer utils.RemoveTempDir(newDir)

os.Setenv(ModeEnv, test.Mode)
os.Setenv(NetworkEnv, test.Network)
os.Setenv(PortEnv, test.Port)
_ = os.Setenv(ModeEnv, test.Mode)
_ = os.Setenv(NetworkEnv, test.Network)
_ = os.Setenv(PortEnv, test.Port)

cfg, err := LoadConfiguration(newDir)
if test.err != nil {
Expand Down
Loading