TOOLS-1469 #27

otoumas · 2022-10-07T15:11:02Z

In this PR:

Included latest changes form rosetta-bitcoin see TOOLS-1469
Added log level endpoints
Removed hardcoded zend version numbers (now we only need to update it in make file)

Notes

We will use this same repository to implement the indexer for mainchain, first with a different branch and then with flags

Modified the way Zend node version is retrieved at run time

voyenavion · 2022-10-24T14:08:13Z

Dockerfile

 ARG IS_RELEASE=false
 # cronic <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/219F55740BBF7A1CE368BA45FB7053CE4991B669
 # Luigi Varriale <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C
-ARG MAINTAINER_KEYS="219F55740BBF7A1CE368BA45FB7053CE4991B669 FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C"
+# otoumas <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/2BBE2AA1A641F6147B58450BE3527B60DAACA1D8


too much info in the comments = security risk?

well these are our public keys, so I don't think it is a great risk

Note that MAINTAINER_KEYS here is used to check who is allowed to sign the release tag on the zend repository. Not rosetta-zen.

Including PaoloT in this list would make more sense.

voyenavion · 2022-10-24T14:16:22Z

indexer/indexer.go

+	// we multiply runtime.NumCPU by to determine
+	// how many goroutines we should
+	// spwan to handle block data sequencing.
+	overclockMultiplier = 16


I wonder if there would be a way to make this value a function of the underlying machine's capacity instead of a fixed hard-coded value? seems like a potential risk for running out of threads or memory, but I could be wrong. Is it always the case that 16 routines to one cpu is a good fit?

Agree, but we can come back here and improve this afterwards. At the moment just aligning with rosetta-bitcoin

voyenavion · 2022-10-24T14:23:55Z

indexer/indexer.go

-	opts.TableLoadingMode = options.MemoryMap
-	opts.ValueLogLoadingMode = options.MemoryMap
+	opts.TableLoadingMode = options.FileIO
+	opts.ValueLogLoadingMode = options.FileIO


Has something changed here to warrant a change of the comment? "FileIO" seems different and more also more vague than "MemoryMap"

This is a very good question @voyenavion , this is something I wanted @cronicc 's opinion on

I don't know about the reasoning for switching to FileIO here, it should be slower than MemoryMap, but have less memory requirements. Maybe there was a reason for the change provided in the upstream commit message?

cronicc · 2023-03-22T18:50:33Z

Dockerfile

 ARG IS_RELEASE=false
 # cronic <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/219F55740BBF7A1CE368BA45FB7053CE4991B669
 # Luigi Varriale <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C
-ARG MAINTAINER_KEYS="219F55740BBF7A1CE368BA45FB7053CE4991B669 FC3388A460ACFAB04E8328C07BB2A1D2CFDFCD2C"
+# otoumas <[email protected]> https://keys.openpgp.org/vks/v1/by-fingerprint/2BBE2AA1A641F6147B58450BE3527B60DAACA1D8


Note that MAINTAINER_KEYS here is used to check who is allowed to sign the release tag on the zend repository. Not rosetta-zen.

Including PaoloT in this list would make more sense.

cronicc · 2023-03-22T18:54:47Z

Dockerfile

@@ -53,18 +53,18 @@ RUN set -euxo pipefail \
      && ( gpgconf --kill dirmngr || true ) \
      && ( gpgconf --kill gpg-agent || true ); \
    fi \
-    && export MAKEFLAGS="-j $(($(nproc)+1))" && ./zcutil/build.sh $MAKEFLAGS


I would prefer making this a build time check, as legacy-cpu has a performance penalty. Any modern platform (cloud servers where Coinbase and we run rosetta-zen) supports the needed CPU flag. The only use case I see for building legacy-cpu images is on developer machines.

For detecting CPU support while running docker build an approach similar to this https://github.com/HorizenOfficial/zen-node-docker/blob/84c213dec9636d616721c971b74214ab75a66916/entrypoint.sh#L6 should work. Or https://manpages.ubuntu.com/manpages/focal/man1/lscpu.1.html can be used.

cronicc · 2023-03-22T18:59:01Z

Makefile

-	docker run --rm -v "${PWD}/zen-data:/data" ubuntu:18.04 bash -c 'chown -R nobody:nogroup /data';
-	docker run -d --name=rosetta-zen-testnet-online --ulimit "nofile=${NOFILE}:${NOFILE}" -v "${PWD}/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 19033:19033 rosetta-zen:latest;
+	docker run --rm -v "${PWD}/zen-data:/data" ubuntu:20.04 bash -c 'chown -R nobody:nogroup /data';
+	docker run -d --name=rosetta-zen-testnet-online --ulimit "nofile=${NOFILE}:${NOFILE}" -v "${PWD}/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 19033:19033 -p 18231:18231 rosetta-zen:latest;


RPC ports like -p 18231:18231 should never be exposed to the internet, which -p does if the instance rosetta is running on has a publicly routable IP. The RPC interface is unencrypted, and even though there is authentication, there is no rate limiting against brute force attacks. --expose 18231 can be used instead which exposes the port to other containers on the same docker network, or -p 127.0.0.1:18231 to only expose the port on localhost.

cronicc · 2023-03-22T19:06:24Z

README.md

-# start rosetta-zen
-docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest
+# start rosetta-zen. Zend version needs to be specified (ex. v3.2.0)
+docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/zen-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "NODE_VERSION=${ZEND_VERSION}" -e "PORT=8080" -p 8080:8080 -p 9033:9033 rosetta-zen:latest


Why do we need to specify NODE_VERSION? I can't see it being used anywhere. Zend version could also be detected on the fly in entypoint.sh by running zend -version | head -n1 | cut -d " " -f 4, or storing the var from the Makefile via ENV NODE_VERSION=$ARG in Dockerfile.

cronicc · 2023-03-22T19:11:18Z

indexer/indexer.go

-	opts.TableLoadingMode = options.MemoryMap
-	opts.ValueLogLoadingMode = options.MemoryMap
+	opts.TableLoadingMode = options.FileIO
+	opts.ValueLogLoadingMode = options.FileIO


I don't know about the reasoning for switching to FileIO here, it should be slower than MemoryMap, but have less memory requirements. Maybe there was a reason for the change provided in the upstream commit message?

cronicc · 2023-03-22T19:12:36Z

indexer/indexer.go

@@ -154,6 +184,13 @@ func defaultBadgerOptions(
 	// filters will be immediately discarded from the cache).
 	opts.LoadBloomsOnOpen = false

+	// BlockSize sets the size of any block in SSTable. SSTable is divided into multiple blocks
+	// internally. We set each block to 1MB.
+	opts.BlockSize = 1 << blockLeftShift // 1MB


If this has any relationship to the size of blocks in our mainchain, 1MB would not be enough as we support larger blocks.

cronicc · 2023-03-22T19:20:51Z

services/account_service_test.go

@@ -161,3 +141,61 @@ func TestAccountBalance_Online_Historical(t *testing.T) {

 	mockIndexer.AssertExpectations(t)
 }
+
+func TestAccountCoins_Online(t *testing.T) {


Accounts are deprecated in zend, all accounts are hard coded to empty string "" in RPC commands.

otoumas added 4 commits October 4, 2022 11:27

Use constant value instead of hardcoded value in test

82018b4

Added changes from rosetta-bitcoin

81cfa46

Modified the way Zend node version is retrieved at run time

Added LogLevel control system

f67f9c7

Added LogLevel endpoints

5ef4283

otoumas requested review from yurikoinaba and cronicc October 7, 2022 15:11

otoumas requested a review from voyenavion October 17, 2022 15:22

carlosper88 approved these changes Oct 24, 2022

View reviewed changes

voyenavion approved these changes Oct 24, 2022

View reviewed changes

yurikoinaba approved these changes Oct 26, 2022

View reviewed changes

cronicc reviewed Mar 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TOOLS-1469 #27

TOOLS-1469 #27

otoumas commented Oct 7, 2022 •

edited by jira bot

Loading

voyenavion Oct 24, 2022

otoumas Oct 27, 2022

cronicc Mar 22, 2023

voyenavion Oct 24, 2022

otoumas Oct 27, 2022

voyenavion Oct 24, 2022

otoumas Oct 27, 2022

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

cronicc Mar 22, 2023

TOOLS-1469 #27

Are you sure you want to change the base?

TOOLS-1469 #27

Conversation

otoumas commented Oct 7, 2022 • edited by jira bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

otoumas commented Oct 7, 2022 •

edited by jira bot

Loading