Skip to content

Commit

Permalink
Server 0.10.0.3 support (#15)
Browse files Browse the repository at this point in the history
Server 0.10.0.3
Rest 2.1.0
Renamed daemon with detached.
The workaround of log files directory in server
Updated optin test preset.
Added --service to allow starting just a docker service by name.

Co-authored-by: wayonb <[email protected]>
  • Loading branch information
fboucquez and Wayonb authored Sep 26, 2020
1 parent cb0f7b1 commit e2bb4a1
Show file tree
Hide file tree
Showing 65 changed files with 474 additions and 263 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file.

The changelog format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [0.1.0] - 26-Sep-2020

**Milestone**: Hippopotamus(0.10.0.3)

Package | Version | Link
---|---|---
Symbol Bootstrap | v0.1.0 | [symbol-bootstrap](https://www.npmjs.com/package/symbol-bootstrap)

- 0.10.0.3 catapult server support.
- 2.1.0 rest server support.
- Improved logging configuration.
- Allowing custom user when running config time docker images.
- Renamed param from `--daemon` to `--detached` to keep it in line with docker compose.
- Added `--service (-s)` to allow starting just one docker service by name.

## [0.0.0] - 14-Sep-2020

**Milestone**: Gorilla.1(0.9.6.4)
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ $ npm install -g symbol-bootstrap
$ symbol-bootstrap COMMAND
running command...
$ symbol-bootstrap (-v|--version|version)
symbol-bootstrap/0.0.0 linux-x64 node-v12.18.3
symbol-bootstrap/0.1.0 linux-x64 node-v14.8.0
$ symbol-bootstrap --help [COMMAND]
USAGE
$ symbol-bootstrap COMMAND
Expand Down Expand Up @@ -222,12 +222,12 @@ sudo rm -rf ./target
One use case of this CLI is client E2E testing support. If you are coding a Symbol client, you (Travis or Jenkins) can run e2e tests like:

```
symbol-bootstrap start -p bootstrap --daemon
symbol-bootstrap start -p bootstrap --detached
YOUR TEST (e.g: npm run test, gradle test, selenium etc.)
symbol-bootstrap stop
```

`--daemon` starts the server waiting until it is up (by polling the network http://localhost:3000/node/health). The command will fail if the components are not up in 30 seconds.
`--detached` starts the server waiting until it is up (by polling the network http://localhost:3000/node/health). The command will fail if the components are not up in 30 seconds.

You can also provide your own custom preset (`-c`) if you want your e2e test to start with a specific state (specific balances addresses, mosaics, namespaces, generation hash seed, etc.)

Expand All @@ -247,7 +247,7 @@ Your `package.json` can look like this:
scripts": {
...
"clean-network": "symbol-bootstrap clean",
"run-network": "symbol-bootstrap start -c ./output/my_custom_preset.yml --daemon",
"run-network": "symbol-bootstrap start -c ./output/my_custom_preset.yml --detached",
"run-stop": "symbol-bootstrap stop",
"integration-test": "....some mocha/jest/etc tests running against localhost:3000 network....",
"e2e": "npm run clean-network && npm run run-network && npm run integration-test && npm run stop-network",
Expand Down Expand Up @@ -276,7 +276,7 @@ it('Bootstrap e2e test', async () => {
reset: true,
timeout: 60000 * 5,
target: 'target/bootstrap-test',
daemon: true,
detached: true,
user: 'current',
};
try {
Expand Down Expand Up @@ -318,9 +318,9 @@ General users should install this tool like any other node module.
* [`symbol-bootstrap compose`](docs/compose.md) - It generates the `docker-compose.yml` file from the configured network.
* [`symbol-bootstrap config`](docs/config.md) - Command used to set up the configuration files and the nemesis block for the current network
* [`symbol-bootstrap help`](docs/help.md) - display help for symbol-bootstrap
* [`symbol-bootstrap run`](docs/run.md) - It boots the network via docker using the generated `docker-compose.yml` file and configuration. The config and compose methods/commands need to be called before this method. This is just a wrapper for the `docker-compose up` bash call
* [`symbol-bootstrap run`](docs/run.md) - It boots the network via docker using the generated `docker-compose.yml` file and configuration. The config and compose methods/commands need to be called before this method. This is just a wrapper for the `docker-compose up` bash call.
* [`symbol-bootstrap start`](docs/start.md) - Single command that aggregates config, compose and run in one line!
* [`symbol-bootstrap stop`](docs/stop.md) - It stops the docker-compose network if running (symbol-bootstrap started with --daemon). This is just a wrapper for the `docker-compose down` bash call.
* [`symbol-bootstrap stop`](docs/stop.md) - It stops the docker-compose network if running (symbol-bootstrap started with --detached). This is just a wrapper for the `docker-compose down` bash call.

<!-- commandsstop -->
```
6 changes: 0 additions & 6 deletions cmds/clean-data.sh

This file was deleted.

2 changes: 1 addition & 1 deletion cmds/setup-network.sh
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
#!/bin/bash
./bin/run config -r
symbol-bootstrap config -r
2 changes: 1 addition & 1 deletion cmds/start-bootstrap-alpha.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
set -e

# docker rm -f $(docker ps -aq)
./bin/run start -p bootstrap -r -t target/bootstrap -a alpha $1
symbol-bootstrap start -p bootstrap -r -t target/bootstrap -a alpha $1
4 changes: 1 addition & 3 deletions cmds/start-bootstrap-optin.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
#!/bin/bash
set -e

sudo ./cmds/clean-all.sh
./bin/run start -u current -p bootstrap -c test/optin_preset.yml -t target/bootstrap-optin
symbol-bootstrap start -u current -p bootstrap -c test/optin_preset.yml -t target/bootstrap-optin $1
4 changes: 4 additions & 0 deletions cmds/start-bootstrap-sudo.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/bin/bash
set -e

symbol-bootstrap start -p bootstrap -r -u '' -t target/bootstrap $1
2 changes: 1 addition & 1 deletion cmds/start-bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
set -e

# docker rm -f $(docker ps -aq)
./bin/run start -p bootstrap -r -u current -t target/bootstrap $1
symbol-bootstrap start -p bootstrap -u current -t target/bootstrap $1
2 changes: 1 addition & 1 deletion cmds/start-light.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
set -e

./bin/run start -p light -r -u current -t target/light $1
symbol-bootstrap start -p light -r -u current -t target/light $1
4 changes: 2 additions & 2 deletions cmds/start-peers.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
sudo ./cmds/clean-all.sh
./bin/run config
./bin/run compose -u current
symbol-bootstrap config
symbol-bootstrap compose -u current
docker-compose -f target/docker/docker-compose.yml up --build peer-node-0 peer-node-1
2 changes: 1 addition & 1 deletion cmds/start-testnet.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#!/bin/bash
set -e
./bin/run start -r -p testnet -a dual -u current -t target/testnet-dual $1
symbol-bootstrap start -p testnet -a dual -u current -t target/testnet-dual $1
2 changes: 1 addition & 1 deletion cmds/stop-bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
set -e

# docker rm -f $(docker ps -aq)
./bin/run stop -t target/bootstrap
symbol-bootstrap stop -t target/bootstrap
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ echo "RUNNING runServerRecover.sh $name"
source /userconfig/prepare.sh $name

ulimit -c unlimited

cd /data
if [ -e "/data/broker.lock" ] || [ -e "/data/server.lock" ]; then
echo "!!!! Have lock file present, going to run recovery...."
exec /usr/catapult/bin/catapult.recovery /userconfig
exec env LD_LIBRARY_PATH=/usr/catapult/lib:/usr/catapult/deps /usr/catapult/bin/catapult.recovery /userconfig
echo "!!!! Finished running recovery, should be moving on to start server..."
else
echo "!!!! DO NOT HAVE ANY LOCk FILE.."
echo "!!!! DO NOT HAVE ANY LOCK FILE... There is no need to recover"
exit 0;
fi

Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ rm /data/startup/datadir-initialized

touch "/state/$name"

exec /usr/catapult/bin/catapult.broker /userconfig
exec env LD_LIBRARY_PATH=/usr/catapult/lib:/usr/catapult/deps /usr/catapult/bin/catapult.broker /userconfig
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ touch /data/startup/datadir-initialized

echo "!!!! Going to start server now...."

exec /usr/catapult/bin/catapult.server /userconfig
exec env LD_LIBRARY_PATH=/usr/catapult/lib:/usr/catapult/deps /usr/catapult/bin/catapult.server /userconfig
2 changes: 1 addition & 1 deletion config/docker/mongo/mongoDbPrepare.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

db.createCollection('finalizedBlocks');
db.finalizedBlocks.createIndex({ 'block.height': 1 }, { unique: true });
db.finalizedBlocks.createIndex({ 'block.finalizationPoint': 1 }, { unique: true });
db.finalizedBlocks.createIndex({ 'block.finalizationPoint': 1 });

db.createCollection('transactions');
addCommonTransactionIndexes(db.transactions);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ extension.syncsource = {{syncsource}}

# common extensions
extension.diagnostics = true
extension.finalization = true
extension.hashcache = true
extension.networkheight = false
extension.nodediscovery = true
Expand Down
14 changes: 14 additions & 0 deletions config/node/resources/config-finalization.properties.mustache
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[finalization]

enableVoting = {{voting}}

size = 10'000
threshold = 7'000
stepDuration = {{{stepDuration}}}

shortLivedCacheMessageDuration = 10m
messageSynchronizationMaxResponseSize = 20MB

maxHashesPerPoint = 256
prevoteBlocksMultiple = 4
votingKeyDilution = 128
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ colorMode = Ansi
sinkType = Async
level = {{#if brokerLogLevel}}{{brokerLogLevel}}{{else}}{{logLevel}}{{/if}}
directory = logs
filePattern = catapult_broker%4N.log
filePattern = logs/catapult_broker%4N.log
rotationSize = 25MB
maxTotalSize = 2500MB
minFreeSpace = 100MB
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ colorMode = Ansi
sinkType = Async
level = {{#if recoveryLogLevel}}{{recoveryLogLevel}}{{else}}{{logLevel}}{{/if}}
directory = logs
filePattern = catapult_recovery%4N.log
filePattern = logs/catapult_recovery%4N.log
rotationSize = 25MB
maxTotalSize = 2500MB
minFreeSpace = 100MB
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ colorMode = Ansi
sinkType = {{sinkType}}
level = {{#if serverLogLevel}}{{serverLogLevel}}{{else}}{{logLevel}}{{/if}}
directory = logs
filePattern = catapult_server%4N.log
filePattern = logs/catapult_server%4N.log
rotationSize = 25MB
maxTotalSize = 2500MB
minFreeSpace = 100MB
Expand Down
3 changes: 1 addition & 2 deletions config/node/resources/config-network.properties.mustache
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ harvestingMosaicId = {{{toHex harvestingMosaicId}}}

blockGenerationTargetTime = {{{blockGenerationTargetTime}}}
blockTimeSmoothingFactor = {{{blockTimeSmoothingFactor}}}
blockFinalizationInterval = {{{blockFinalizationInterval}}}

importanceGrouping = {{{importanceGrouping}}}
importanceActivityPercentage = {{{importanceActivityPercentage}}}
Expand All @@ -36,6 +35,7 @@ maxHarvesterBalance = {{{toAmount maxHarvesterBalance}}}
minVoterBalance = {{{toAmount minVoterBalance}}}

# assuming finalization ~20 minutes
votingSetGrouping = {{{votingSetGrouping}}}
maxVotingKeysPerAccount = {{{maxVotingKeysPerAccount}}}
minVotingKeyLifetime = {{{minVotingKeyLifetime}}}
maxVotingKeyLifetime = {{{maxVotingKeyLifetime}}}
Expand All @@ -44,7 +44,6 @@ harvestBeneficiaryPercentage = {{{harvestBeneficiaryPercentage}}}
harvestNetworkPercentage = {{{harvestNetworkPercentage}}}
harvestNetworkFeeSinkAddress = {{{harvestNetworkFeeSinkAddress}}}

blockPruneInterval = {{{blockPruneInterval}}}
maxTransactionsPerBlock = {{{toAmount maxTransactionsPerBlock}}}

[plugin:catapult.plugins.accountlink]
Expand Down
1 change: 1 addition & 0 deletions config/node/resources/config-node.properties.mustache
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ enableAutoSyncCleanup = {{{enableAutoSyncCleanup}}}
enableTransactionSpamThrottling = {{{enableTransactionSpamThrottling}}}
transactionSpamThrottlingMaxBoostFee = {{{toAmount transactionSpamThrottlingMaxBoostFee}}}

maxHashesPerSyncAttempt = {{{maxHashesPerSyncAttempt}}}
maxBlocksPerSyncAttempt = {{{maxBlocksPerSyncAttempt}}}
maxChainBytesPerSyncAttempt = {{{maxChainBytesPerSyncAttempt}}}

Expand Down
84 changes: 58 additions & 26 deletions config/node/resources/config-task.properties.mustache
Original file line number Diff line number Diff line change
@@ -1,39 +1,41 @@
[age peers task for service Api Writers]
startDelay = 1m
repeatDelay = 1m
### diagnostics ###

[age peers task for service Readers]
[logging task]
startDelay = 1m
repeatDelay = 10m

### finalization ###

[connect peers task for service Finalization]
startDelay = 2s
repeatDelay = 1m

[batch partial transaction task]
startDelay = 500ms
repeatDelay = 500ms
[finalization task]
startDelay = 2m
repeatDelay = 15s

[batch transaction task]
startDelay = 500ms
repeatDelay = 500ms
[pull finalization messages task]
startDelay = 3s
repeatDelay = 1s

[connect peers task for service Pt]
startDelay = 1s
repeatDelay = 1m
[pull finalization proof task]
startDelay = 10s
repeatDelay = 50s

[connect peers task for service Sync]
startDelay = 1s
repeatDelay = 1m
### harvesting ###

[harvesting task]
startDelay = 30s
repeatDelay = 1s

[logging task]
startDelay = 1m
repeatDelay = 10m
### networkheight ###

[network chain height detection]
startDelay = 1s
repeatDelay = 15s

### nodediscovery ###

[node discovery peers task]
startDelay = 1m
minDelay = 1m
Expand All @@ -45,28 +47,58 @@ numTransitionRounds = 20
startDelay = 2m
repeatDelay = 5m

### packetserver ###

[age peers task for service Readers]
startDelay = 1m
repeatDelay = 1m

### partialtransaction ###

[batch partial transaction task]
startDelay = 500ms
repeatDelay = 500ms

[connect peers task for service Pt]
startDelay = 3s
repeatDelay = 1m

[pull partial transactions task]
startDelay = 10s
repeatDelay = 3s

### sync ###

[batch transaction task]
startDelay = 500ms
repeatDelay = 500ms

[connect peers task for service Sync]
startDelay = 1s
repeatDelay = 1m

[pull unconfirmed transactions task]
startDelay = 4s
repeatDelay = 3s

[static node refresh task]
startDelay = 5ms
minDelay = 15s
maxDelay = 24h
numPhaseOneRounds = 20
numTransitionRounds = 20

[synchronizer task]
startDelay = 3s
repeatDelay = 3s

### timesync ###

[time synchronization task]
startDelay = 1m
minDelay = 3m
maxDelay = 180m
numPhaseOneRounds = 5
numTransitionRounds = 10

### local (built-in) ###

[static node refresh task]
startDelay = 5ms
minDelay = 15s
maxDelay = 24h
numPhaseOneRounds = 20
numTransitionRounds = 20
1 change: 1 addition & 0 deletions config/node/resources/config-user.properties.mustache
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ enableDelegatedHarvestersAutoDetection = true
certificateDirectory = /userconfig/resources/cert
dataDirectory = /data
pluginsDirectory = /usr/catapult/lib
votingKeysDirectory = /data/votingkeys
Loading

0 comments on commit e2bb4a1

Please sign in to comment.