-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ci: build and run tests, caching the build of itcoin-core keyed on it…
…s commit id - gcc12 on ubuntu 22.04 - itcoin-core continues to be checked out on the side of itcoin-fbft. We are not using the automatic checkout feature of our cmake scripts - the prolog tests and the C++ tests are folded together in "make test" target: it would be good to split those - when bringing up the infrstructure for the tests, an adaptive retry strategy with timeouts and exponential backoff is used, in order to minimize the waiting time, but also be prepared to the occasional slowdown - the retry script is taken from https://github.com/kadwanev/retry (license: Apache 2.0) - the change also adds a badge to the README The caching strategy is very coarse: as long as a successful build of itcoin-core was performed in the past, it is reused, otherwise a full build is done. Even with this simple stategy, when the itcoin-core build hits the cache, a CI workflow takes 4-5 minutes instead of 35-40 minutes. The list of itcoin-core libraries is already in thirdparty/CMakeLists.txt, so we reuse it via a CMake trick (the CMake language does not offer any straightforward way of doing it). The usual solution for this kind of task involves writing a separate somescript.cmake which does what we want (in this case simply print the content of ITCOIN_CORE_LIBRARIES) and invoke it via cmake -P somescript.cmake. We would then also need to extract the definition of ITCOIN_CORE_LIBRARIES in a dedicated file in order to import it from both. This would be very cumbersome. Let's instead abuse the target structure and just add a fake target that prints what we want. In the CI yaml we then have to grep out the diagnostic message "Built target print_itcoin_core_libraries", because CMake does not offer any hooks for selectively disabling just a single diagnostic message (it's all or nothing, and we want diagnostic messages in our builds). One final issue is that the Azure runner has a "/bin/sh" whose "echo" command does not understand the "-e" parameter, hence we have to force it to use bash instead.
- Loading branch information
Showing
4 changed files
with
330 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| name: Test itcoin-fbft | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| pull_request: | ||
| # Allows you to run this workflow manually from the Actions tab | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| tests: | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Install the build toolchain (gcc-12) | ||
| run: | | ||
| sudo apt install --no-install-recommends -y \ | ||
| autoconf \ | ||
| automake \ | ||
| bsdextrautils \ | ||
| ca-certificates \ | ||
| cmake \ | ||
| g++-12 \ | ||
| gcc-12 \ | ||
| git \ | ||
| jq \ | ||
| libtool \ | ||
| make \ | ||
| openssh-client \ | ||
| parallel \ | ||
| pkg-config \ | ||
| python3 \ | ||
| xxd \ | ||
| zlib1g-dev | ||
| - name: Install the build and runtime dependencies | ||
| run: | | ||
| sudo apt install --no-install-recommends -y \ | ||
| libargtable2-dev \ | ||
| libboost-filesystem1.74-dev \ | ||
| libboost-log1.74-dev \ | ||
| libboost-program-options1.74-dev \ | ||
| libboost-test1.74-dev \ | ||
| libboost-thread1.74-dev \ | ||
| libcurl4-openssl-dev \ | ||
| libdb5.3++-dev \ | ||
| libevent-dev \ | ||
| libsqlite3-dev \ | ||
| libssl-dev \ | ||
| libzmq3-dev \ | ||
| swi-prolog | ||
| - name: Checkout itcoin-fbft | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 1 | ||
| path: itcoin-fbft | ||
| - name: Checkout itcoin-core | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| repository: bancaditalia/itcoin-core | ||
| fetch-depth: 1 | ||
| path: itcoin-core | ||
| - name: save the revision id of itcoin-core in env variable ITCOIN_CORE_REVISION_ID | ||
| run: | | ||
| cd itcoin-core | ||
| printf "ITCOIN_CORE_REVISION_ID=%s\n" $(git rev-parse HEAD) | tee --append "${GITHUB_ENV}" | ||
| - name: Prepare the build | ||
| run: | | ||
| mkdir -p itcoin-fbft/build | ||
| cd itcoin-fbft/build | ||
| cmake \ | ||
| -DCMAKE_C_COMPILER=$(which gcc-12) \ | ||
| -DCMAKE_CXX_COMPILER=$(which g++-12) \ | ||
| -DITCOIN_CORE_SRC_DIR=../../itcoin-core \ | ||
| .. | ||
| - name: retrieve the names of the itcoin-core libraries that need to be cached | ||
| run: | | ||
| cd itcoin-fbft/build | ||
| # modified from: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings | ||
| EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64) | ||
| echo "ITCOIN_CORE_LIBRARIES<<${EOF}" | tee --append "${GITHUB_ENV}" | ||
| # Let's use a subshell so that "set -o pipefail" does not mess with | ||
| # the runner's environment. | ||
| # | ||
| # Also, GitHub runner's /bin/sh does not support "echo -e", which we | ||
| # need, so we have to explicitly force the use of "/bin/bash" | ||
| (set -o pipefail ; make SHELL=/bin/bash print_itcoin_core_libraries | grep -v "Built target" | tee --append "${GITHUB_ENV}") | ||
| echo "${EOF}" | tee --append "${GITHUB_ENV}" | ||
| - name: Cache itcoin-core artifacts for version ${{ env.ITCOIN_CORE_REVISION_ID }} | ||
| id: cache-itcoin-core | ||
| uses: actions/cache@v3 | ||
| with: | ||
| key: ${{ env.ITCOIN_CORE_REVISION_ID }} | ||
| path: | | ||
| ${{ env.ITCOIN_CORE_LIBRARIES }} | ||
| ${{ github.workspace }}/itcoin-core/src/bitcoin-cli | ||
| ${{ github.workspace }}/itcoin-core/src/bitcoind | ||
| ${{ github.workspace }}/itcoin-core/src/config/bitcoin-config.h | ||
| ${{ github.workspace }}/itcoin-core/src/secp256k1/libsecp256k1-config.h | ||
| ${{ github.workspace }}/itcoin-fbft/build/thirdparty/itcoin-core-repo-status | ||
| - name: build itcoin-fbft | ||
| run: | | ||
| cd itcoin-fbft/build | ||
| make -j $(nproc) | ||
| - name: start infrastructure in background, wait until started | ||
| run: | | ||
| cd itcoin-fbft/infra | ||
| ./reset-infra.sh 10 | ||
| ./start-infra.sh & | ||
| # Concurrently for each node N in (0..4), failing at the first error: | ||
| # - invoke bitcoin-cli N uptime | ||
| # - if the invocation fails, retry with exponential backoff at most 6 | ||
| # times | ||
| # - if a bitcoin-cli invocation becomes unresponsive, it is timed out | ||
| # after 30 seconds and is considered a failure (eventually retried) | ||
| parallel --halt-on-error now,fail=1 \ | ||
| ./retry --min=0.5 --max=15 --tries=6 -- \ | ||
| timeout --kill-after=40 30 \ | ||
| ./bitcoin-cli.sh {1} uptime \ | ||
| ::: {0..3} | ||
| - name: Run tests (failure will be detected in next step) | ||
| continue-on-error: true | ||
| id: tests | ||
| run: | | ||
| cd itcoin-fbft/build | ||
| make test | ||
| - name: Summarize test outcome. Fail and print log if tests failed. | ||
| run: | | ||
| # summary | ||
| RED='\033[0;31m' | ||
| GREEN='\033[0;32m' | ||
| NC='\033[0m' # No Color | ||
| echo -n "itcoin-fbft tests: " | ||
| if [[ ${{ steps.tests.outcome }} == "success" ]]; then | ||
| printf "${GREEN}SUCCESS${NC}\n" | ||
| exit 0 | ||
| fi | ||
| # if we arrive here, the tests failed | ||
| printf "${RED}FAIL${NC} (${{ steps.tests.outcome }})\n" | ||
| echo "Contents of build/Testing/Temporary/LastTest.log:" | ||
| cat /home/runner/work/itcoin-fbft/itcoin-fbft/itcoin-fbft/build/Testing/Temporary/LastTest.log | ||
| exit 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # source: https://github.com/kadwanev/retry/blob/20997c7712a4d70c7243c536ab7cc119c18060d9/retry | ||
| # License: Apache 2.0 | ||
|
|
||
| GETOPT_BIN=$IN_GETOPT_BIN | ||
| GETOPT_BIN=${GETOPT_BIN:-getopt} | ||
|
|
||
| __sleep_amount() { | ||
| if [ -n "$constant_sleep" ]; then | ||
| sleep_time=$constant_sleep | ||
| else | ||
| #TODO: check if user would rather use one of the other possible dependencies: python, ruby, bc, dc | ||
| sleep_time=`awk "BEGIN {t = $min_sleep * $(( (1<<($attempts -1)) )); print (t > $max_sleep ? $max_sleep : t)}"` | ||
| fi | ||
| } | ||
|
|
||
| __log_out() { | ||
| echo "$1" 1>&2 | ||
| } | ||
|
|
||
| # Parameters: max_tries min_sleep max_sleep constant_sleep fail_script EXECUTION_COMMAND | ||
| retry() | ||
| { | ||
| local max_tries="$1"; shift | ||
| local min_sleep="$1"; shift | ||
| local max_sleep="$1"; shift | ||
| local constant_sleep="$1"; shift | ||
| local fail_script="$1"; shift | ||
| if [ -n "$VERBOSE" ]; then | ||
| __log_out "Retry Parameters: max_tries=$max_tries min_sleep=$min_sleep max_sleep=$max_sleep constant_sleep=$constant_sleep" | ||
| if [ -n "$fail_script" ]; then __log_out "Fail script: $fail_script"; fi | ||
| __log_out "" | ||
| __log_out "Execution Command: $*" | ||
| __log_out "" | ||
| fi | ||
|
|
||
| local attempts=0 | ||
| local return_code=1 | ||
|
|
||
|
|
||
| while [[ $return_code -ne 0 && $attempts -le $max_tries ]]; do | ||
| if [ $attempts -gt 0 ]; then | ||
| __sleep_amount | ||
| __log_out "Before retry #$attempts: sleeping $sleep_time seconds" | ||
| sleep $sleep_time | ||
| fi | ||
|
|
||
| P="$1" | ||
| for param in "${@:2}"; do P="$P '$param'"; done | ||
| #TODO: replace single quotes in each arg with '"'"' ? | ||
| export RETRY_ATTEMPT=$attempts | ||
| bash -c "$P" | ||
| return_code=$? | ||
| #__log_out "Process returned $return_code on attempt $attempts" | ||
| if [ $return_code -eq 127 ]; then | ||
| # command not found | ||
| exit $return_code | ||
| elif [ $return_code -ne 0 ]; then | ||
| attempts=$[$attempts +1] | ||
| fi | ||
| done | ||
|
|
||
| if [ $attempts -gt $max_tries ]; then | ||
| if [ -n "$fail_script" ]; then | ||
| __log_out "Retries exhausted, running fail script" | ||
| eval $fail_script | ||
| else | ||
| __log_out "Retries exhausted" | ||
| fi | ||
| fi | ||
|
|
||
| exit $return_code | ||
| } | ||
|
|
||
| # If we're being sourced, don't worry about such things | ||
| if [ "$BASH_SOURCE" == "$0" ]; then | ||
| # Prints the help text | ||
| help() | ||
| { | ||
| local retry=$(basename $0) | ||
| cat <<EOF | ||
| Usage: $retry [options] -- execute command | ||
| -h, -?, --help | ||
| -v, --verbose Verbose output | ||
| -t, --tries=# Set max retries: Default 10 | ||
| -s, --sleep=secs Constant sleep amount (seconds) | ||
| -m, --min=secs Exponential Backoff: minimum sleep amount (seconds): Default 0.3 | ||
| -x, --max=secs Exponential Backoff: maximum sleep amount (seconds): Default 60 | ||
| -f, --fail="script +cmds" Fail Script: run in case of final failure | ||
| EOF | ||
| } | ||
|
|
||
| # show help for no arguments if stdin is a terminal | ||
| if { [ -z "$1" ] && [ -t 0 ] ; } || [ "$1" == '-h' ] || [ "$1" == '-?' ] || [ "$1" == '--help' ] | ||
| then | ||
| help | ||
| exit 0 | ||
| fi | ||
|
|
||
| $GETOPT_BIN --test > /dev/null | ||
| if [[ $? -ne 4 ]]; then | ||
| echo "I’m sorry, 'getopt --test' failed in this environment. Please load GNU getopt." | ||
| exit 1 | ||
| fi | ||
| if ! command -v awk &> /dev/null; then | ||
| __log_out "Error: awk not found, please install awk." | ||
| exit 1 | ||
| fi | ||
|
|
||
| OPTIONS=vt:s:m:x:f: | ||
| LONGOPTIONS=verbose,tries:,sleep:,min:,max:,fail: | ||
|
|
||
| PARSED=$($GETOPT_BIN --options="$OPTIONS" --longoptions="$LONGOPTIONS" --name "$0" -- "$@") | ||
| if [[ $? -ne 0 ]]; then | ||
| # e.g. $? == 1 | ||
| # then getopt has complained about wrong arguments to stdout | ||
| exit 2 | ||
| fi | ||
| # read getopt’s output this way to handle the quoting right: | ||
| eval set -- "$PARSED" | ||
|
|
||
| max_tries=10 | ||
| min_sleep=0.3 | ||
| max_sleep=60.0 | ||
| constant_sleep= | ||
| fail_script= | ||
|
|
||
| # now enjoy the options in order and nicely split until we see -- | ||
| while true; do | ||
| case "$1" in | ||
| -v|--verbose) | ||
| VERBOSE=true | ||
| shift | ||
| ;; | ||
| -t|--tries) | ||
| max_tries="$2" | ||
| shift 2 | ||
| ;; | ||
| -s|--sleep) | ||
| constant_sleep="$2" | ||
| shift 2 | ||
| ;; | ||
| -m|--min) | ||
| min_sleep="$2" | ||
| shift 2 | ||
| ;; | ||
| -x|--max) | ||
| max_sleep="$2" | ||
| shift 2 | ||
| ;; | ||
| -f|--fail) | ||
| fail_script="$2" | ||
| shift 2 | ||
| ;; | ||
| --) | ||
| shift | ||
| break | ||
| ;; | ||
| *) | ||
| echo "Programming error" | ||
| exit 3 | ||
| ;; | ||
| esac | ||
| done | ||
|
|
||
| retry "$max_tries" "$min_sleep" "$max_sleep" "$constant_sleep" "$fail_script" "$@" | ||
|
|
||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters