Update CI #552

b-yap · 2024-09-17T06:42:48Z

To avoid disk space issues or similar to this:

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

The hosted runner encountered an error while running your job. (Error Type: Disconnect).

Received request to deprovision: The request was cancelled by the remote provider.

I've split one huge job into multiple jobs (as suggested in actions/runner-images#10401)

New CI

The jobs have shared actions:

└── prerequisite-nightly
      ├── prerequisite
      │   ├── free-up-disk-space
      │   └── installations
      │        └── cache, protoc
      └── install-rust-nightly
          └── setup nightly

Jobs using stable rust version uses prerequisite action.
Jobs using nightly rust version uses prerequisite-nightly action.

Flow of the jobs:

└── Check Build (stable)
      ├── Run Clippy and Format Checks (stable)
      ├── Run Clippy and Format Checks (nightly)
      └-├─ Run Tests for Pallets (nightly)
      └-├─ Run Tests for Clients (nightly)
        └── Run Tests for Vault (nightly)

Cleanup Dependencies

Ran #![deny(unused_crate_dependencies)] on the pallets (and other directories) to identify the dependencies to:

remove; or
move to [dev-dependencies].

Most of these are serde(still questionable), sp-io, sp-runtime and sp-core, pallet-balances and pallet-timestamp to name a few.

All other touched files are caused by clippy.

b-yap · 2024-09-18T16:56:29Z

pallets/clients-info/Cargo.toml


 # Substrate dependencies
-sp-runtime = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }


Note: sp-runtime is a dependency found in frame-support. We don't need to add sp-runtime as a dependency.

b-yap · 2024-09-18T16:57:04Z

pallets/clients-info/Cargo.toml

-sp-runtime = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
-sp-arithmetic = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
-sp-std = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
-sp-io = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }


sp-core and sp-io are only used for testing. I've moved them to [dev-dependencies]

b-yap · 2024-09-18T16:58:01Z

pallets/currency/Cargo.toml

 frame-support = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
 frame-system = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
-pallet-balances = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }


The pallet-balances is used only in the tests; moving it to [dev-dependencies].

b-yap · 2024-09-18T17:00:36Z

scripts/cmd-all

@@ -45,6 +45,7 @@ cargo $sub_cmd -p oracle --all-features $options
 cargo $sub_cmd -p pooled-rewards --all-features $options
 cargo $sub_cmd -p redeem --all-features $options
 cargo $sub_cmd -p replace --all-features $options
+cargo $sub_cmd -p reward --all-features $options


we forgot to check reward

b-yap · 2024-09-20T08:01:12Z

pallets/currency/src/amount.rs

@@ -174,8 +177,7 @@ mod math {
 				.ok_or(ArithmeticError::Underflow)?
 				.checked_div(accuracy)
 				.ok_or(ArithmeticError::Underflow)?
-				.try_into()


This was a clippy suggestion

b-yap · 2024-09-20T08:02:41Z

pallets/nomination/Cargo.toml

@@ -5,82 +5,64 @@ authors = ["Pendulum"]
 edition = "2021"

 [dependencies]
-serde = { version = "1.0.130", default-features = false, features = ["derive"], optional = true }


serde was flagged as unused dependency by
#![deny(unused_crate_dependencies)]

b-yap · 2024-09-20T08:05:10Z

pallets/nomination/src/lib.rs

@@ -245,8 +246,8 @@ impl<T: Config> Pallet<T> {
 		}

 		ext::vault_registry::pool_manager::withdraw_collateral::<T>(
-			&vault_id,


vault_id and nominator_idare both reference variables,&` is not needed anymore.

(From Clippy)

b-yap · 2024-09-20T08:05:31Z

pallets/oracle/Cargo.toml


 frame-support = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
 frame-system = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }
 frame-benchmarking = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false, optional = true }
-pallet-balances = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }


Only used during testing

b-yap · 2024-09-20T08:06:20Z

pallets/oracle/src/dia.rs

@@ -142,9 +142,7 @@ where
 		let Ok(coin_info) = Dia::get_coin_info(blockchain, symbol) else { return None };

 		let value = ConvertPrice::convert(coin_info.price)?;
-		let Some(timestamp) = ConvertMoment::convert(coin_info.last_update_timestamp) else {


Clippy suggestion. ? returns None for Options; Err for Result.

b-yap · 2024-09-20T08:06:44Z

pallets/oracle/src/lib.rs

@@ -247,7 +247,7 @@ impl<T: Config> Pallet<T> {
 		}

 		let current_status_is_online = Self::is_oracle_online();
-		let new_status_is_online = oracle_keys.len() > 0 &&


Clippy suggestion

b-yap · 2024-09-20T08:08:55Z

pallets/reward-distribution/src/benchmarking.rs

@@ -39,15 +39,11 @@ pub mod benchmarks {
 			collateral_currency,
 		);

-		let _stake = T::VaultStaking::deposit_stake(&vault_id, &nominator, nominated_amount.into())


The trait Staking in pallet-staking returns Result<(), DispatchError>; so no need for let <variable_name>.

b-yap · 2024-09-20T08:11:55Z

pallets/security/Cargo.toml

 sp-std = {git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false}



 [dev-dependencies]
 mocktopus = "0.8.0"
-sp-io = {git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false}


we need the std feature, to avoid externalities issues, hence default-features should be true:

error: cannot find macro `thread_local` in this scope --> /home/runner/.cargo/git/checkouts/polkadot-sdk-67b6d29d759a1af6/e51a91f/substrate/primitives/externalities/src/scope_limited.rs:22:1 | 22 | environmental::environmental!(ext: trait Externalities); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = note: `thread_local` is in scope, but it is an attribute: `#[thread_local]` = note: this error originates in the macro `$crate::thread_local_impl` which comes from the expansion of the macro `environmental::environmental` (in Nightly builds, run with -Z macro-backtrace for more info)

See failed CI

b-yap · 2024-09-20T08:13:41Z

pallets/stellar-relay/src/validation.rs

@@ -186,7 +186,7 @@ pub fn validate_envelopes<'a, T: Config>(
 		let (value, n_h) = get_externalized_info(envelope)?;

 		// Check if the tx_set_hash matches the one included in the envelope
-		let tx_set_hash = Pallet::<T>::get_tx_set_hash(&value)?;


value here is already a reference; & is not needed.

(Clippy error)

b-yap · 2024-09-20T08:15:09Z

scripts/cmd-all

@@ -23,69 +23,11 @@ echo "------------------------------------------------------------------------"

 cargo $sub_cmd -p spacewalk-primitives --all-features $options

-echo "------------------------------------------------------------------------"


Broken down this script into multiple ones, to avoid the issue aforementioned in the description.

b-yap · 2024-09-20T08:15:26Z

testchain/runtime/testnet/src/lib.rs

@@ -479,7 +479,7 @@ impl XCMCurrencyConversion for SpacewalkNativeCurrencyKey {
 		if blockchain.len() != 1 && blockchain[0] != 0 || symbol.len() != 1 {
 			return None;
 		}
-		return Some(symbol[0]);


Clippy suggestion

ebma

Really nice job, it's amazing @b-yap 🚀 sorry for the delayed review!
Only left two comments but we can merge after

ebma · 2024-10-02T15:28:30Z

scripts/cmd-all

+./scripts/cmd-pallets "$reason" "$sub_cmd" "$options"

-if [[ $sub_cmd == *"nightly"* ]]; then
-  cargo $sub_cmd -p currency $options
-else
-  echo "------------------------------------------------------------------------"
-  echo "     Excluding testing-utils feature of pallet currency;"
-  echo "     that is for nightly version."
-  echo "------------------------------------------------------------------------"
-  cargo $sub_cmd -p currency --features std,testing-constants,try-runtime,runtime-benchmarks $options
-fi
+# run the clients
+./scripts/cmd-clients "$reason" "$sub_cmd" "$options"

-cargo $sub_cmd -p fee --all-features $options
-cargo $sub_cmd -p issue --all-features $options
-cargo $sub_cmd -p nomination --all-features $options
-cargo $sub_cmd -p oracle --all-features $options
-cargo $sub_cmd -p pooled-rewards --all-features $options
-cargo $sub_cmd -p redeem --all-features $options
-cargo $sub_cmd -p replace --all-features $options
-cargo $sub_cmd -p reward-distribution --all-features $options
-cargo $sub_cmd -p security --all-features $options
-cargo $sub_cmd -p staking --all-features $options
-cargo $sub_cmd -p stellar-relay --all-features $options
-cargo $sub_cmd -p vault-registry --all-features $options
-
-echo "------------------------------------------------------------------------"
-echo "Checks" $reason "for pallets with rpc"
-echo "------------------------------------------------------------------------"
-cargo $sub_cmd -p module-issue-rpc --all-features $options
-cargo $sub_cmd -p module-oracle-rpc --all-features $options
-cargo $sub_cmd -p module-redeem-rpc --all-features $options
-cargo $sub_cmd -p module-replace-rpc --all-features $options
-cargo $sub_cmd -p module-vault-registry-rpc --all-features $options
-cargo $sub_cmd -p module-issue-rpc-runtime-api --all-features $options
-cargo $sub_cmd -p module-oracle-rpc-runtime-api --all-features $options
-cargo $sub_cmd -p module-redeem-rpc-runtime-api --all-features $options
-cargo $sub_cmd -p module-replace-rpc-runtime-api --all-features $options
-cargo $sub_cmd -p module-vault-registry-rpc-runtime-api --all-features $options
+# run the others
+./scripts/cmd-others $reason $sub_cmd $options


Any reason why we sometimes wrap it with "" and sometimes we don't? Like, "$reason" vs $reason

Ah, there is no underlying reason. I removed the middle ""

ebma · 2024-10-02T15:30:44Z

scripts/cmd-pallets

+##### Executes the command to ALL packages of this project.
+##### An exception is the package `subxt-client`, as it needs the package `runtime` for it to work.
+##### The currently supported (and tested) commands are: check and clippy
+##### arguments:
+#####   * reason = what the command is for
+#####   * sub_cmd = the actual sub command. For commands separated by spaces, enclose with ""
+#####     e.g. "build --lib"
+#####          "+nightly check"
+#####   * options = additional options of the sub command. If separated by spaces, enclose with ""
+#####     e.g. "-- -W clippy::all"


The comment here is out of place and should be only in cmd-all right?

yes. removed

…784112, #552 (comment)

…797/job/30240327947?pr=552

https://github.com/pendulum-chain/spacewalk/actions/runs/10943093849/job/30384114712

…784112, #552 (comment)

b-yap force-pushed the fix-ci branch from 53bccfc to 79074dd Compare September 17, 2024 06:44

b-yap commented Sep 18, 2024

View reviewed changes

b-yap commented Sep 20, 2024

View reviewed changes

b-yap marked this pull request as ready for review September 24, 2024 05:37

b-yap requested a review from a team September 24, 2024 05:37

ebma approved these changes Oct 2, 2024

View reviewed changes

b-yap added a commit that referenced this pull request Oct 3, 2024

https://github.com/pendulum-chain/spacewalk/pull/552#discussion_r1784…

a86a94b

…784112, #552 (comment)

b-yap added 10 commits October 3, 2024 15:25

chop steps into different jobs

ec3559c

fix https://github.com/pendulum-chain/spacewalk/actions/runs/10897911816

6bee24c

fix https://github.com/pendulum-chain/spacewalk/actions/runs/10897974…

3710d75

…797/job/30240327947?pr=552

add other installations

3fe86c0

add other installations

4fc1a06

require secrets and specify shell

78d1597

add secrets

ad174d0

remove secret from the top

88cbe8b

passing parameters to actions

e22e36f

add shell

d2a6e9d

b-yap added 27 commits October 3, 2024 15:25

fix clippy

7c6103f

fix error:

a266198

https://github.com/pendulum-chain/spacewalk/actions/runs/10943093849/job/30384114712

re-enable build check

7aa001f

testing clients need nightly version of rust

cbcf544

add the environments

7b83c22

clean up the scripts

77bb9b5

run vault test in a different job

66d47a9

update version to node 20

51c684b

only integration-test

ff5ae53

only integration-test

4ad3f4a

clean up vault and test

eb51646

install protoc

ae70649

use ubuntu 20.04

632168a

https://github.com/tauri-apps/tauri/issues/9005

d47bbe5

install necessary libs

9fc1e7e

install necessary libs

5891399

"uses"

7b263ab

remove codec error

aec0395

install libpq-dev

386f479

switch to ubuntu-latest

19d6dc8

last attempt

f4bc738

run test

83270d4

run whole thing

f63fc4e

run 4 tests in parallel

198909f

shorten CI

594ef06

https://github.com/pendulum-chain/spacewalk/pull/552#discussion_r1784…

f814163

…784112, #552 (comment)

fix cargo fmt issues

176f396

b-yap force-pushed the fix-ci branch from a86a94b to 176f396 Compare October 3, 2024 07:46

b-yap merged commit 7c79898 into main Oct 3, 2024
2 of 4 checks passed

b-yap deleted the fix-ci branch October 3, 2024 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update CI #552

Update CI #552

b-yap commented Sep 17, 2024 •

edited

Loading

b-yap Sep 18, 2024

b-yap Sep 18, 2024

b-yap Sep 18, 2024

b-yap Sep 18, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024 •

edited

Loading

b-yap Sep 20, 2024

b-yap Sep 20, 2024

b-yap Sep 20, 2024

ebma left a comment

ebma Oct 2, 2024

b-yap Oct 3, 2024

ebma Oct 2, 2024

b-yap Oct 3, 2024


		# Substrate dependencies
		sp-runtime = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-polkadot-v1.1.0", default-features = false }

		@@ -23,69 +23,11 @@ echo "------------------------------------------------------------------------"

		cargo $sub_cmd -p spacewalk-primitives --all-features $options

		echo "------------------------------------------------------------------------"

Update CI #552

Update CI #552

Conversation

b-yap commented Sep 17, 2024 • edited Loading

New CI

Cleanup Dependencies

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-yap Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-yap commented Sep 17, 2024 •

edited

Loading

b-yap Sep 20, 2024 •

edited

Loading