Centrifuge Testnet Not syncing fast - Relay chain Peers 0

[sgrsaga]

Description

Created the Centrifuge Testnet node with high spec 16 CPU and 128GB RAM. Applied the docker compose file mentioned below with given boot nodes.

Referred docs: https://docs.centrifuge.io/developer/guides/running-a-centrifuge-node/

[Description of the issue or feature]

Docker file

version: '3'
services:
centrifuge:
container_name: centrifuge-chain
image: "centrifugeio/centrifuge-chain:v0.14.0"
platform: "linux/amd64"
restart: on-failure
ports:
- "30333:30333"
- "9933:9933"
volumes:
- /data/centrifuge-data/data:/data
command:
- "--port=30333"
- "--rpc-port=9933"
- "--rpc-external"
- "--rpc-cors=all"
- "--rpc-max-request-size=40"
- "--rpc-max-response-size=40"
- "--rpc-max-connections=512"
- "--in-peers=100"
- "--out-peers=100"
- "--db-cache=16384"
- "--chain=/resources/demo-spec-raw.json"
- "--parachain-id=2031"
- "--base-path=/data"
- "--log=main,info,xcm=trace,xcm-executor=trace"
- "--database=rocksdb"
- "--execution=wasm"
- "--wasm-execution=compiled"
- "--bootnodes=/ip4/35.246.168.210/tcp/30333/p2p/12D3KooWCtdW3HWLuxDLD2fuTZfTspCJDHWxnonKCEgT5JfGsoYQ"
- "--bootnodes=/ip4/34.89.182.4/tcp/30333/p2p/12D3KooWETyS1VZTS4fS7dBZpXbPKMP129dy4KpFSWoErBWJ5i5d"
- "--bootnodes=/ip4/35.198.144.90/tcp/30333/p2p/12D3KooWMJPzvEp5Jhea8eKsUDufBbAzGrn265GcaCmcnp3koPk4"
- "--name=chainlink_centrifuge_testnet_archive"
- "--pruning=archive"
- "--"
- "--execution=wasm"
- "--wasm-execution=compiled"
- "--chain=/resources/westend-alphanet-raw-specs.json"

Node is syncing very slowly. Shows Relay chain peers as 0.

Logs:
'
{"log":"2024-10-17 18:00:03.558 INFO main sc_sysinfo: [Relaychain] 🏁 CPU score: 1018.55 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.568335252Z"}
{"log":"2024-10-17 18:00:03.558 INFO main sc_sysinfo: [Relaychain] 🏁 Memory score: 18.82 GiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.568340763Z"}
{"log":"2024-10-17 18:00:03.558 INFO main sc_sysinfo: [Relaychain] 🏁 Disk score (seq. writes): 316.86 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.568346073Z"}
{"log":"2024-10-17 18:00:03.558 INFO main sc_sysinfo: [Relaychain] 🏁 Disk score (rand. writes): 141.91 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.568352214Z"}
{"log":"2024-10-17 18:00:03.571 WARN main sc_service::config: [🌀Parachain] Using default protocol ID "sup" because none is configured in the chain specs \n","stream":"stderr","time":"2024-10-17T18:00:03.618329133Z"}
{"log":"2024-10-17 18:00:03.582 INFO main sub-libp2p: [🌀Parachain] 🏷 Local node identity is: 12D3KooWAqQWi5UaoNt4Biqavgp4WQSdczq9vacZtdfDSzgUtnmR \n","stream":"stderr","time":"2024-10-17T18:00:03.61836438Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Operating system: linux \n","stream":"stderr","time":"2024-10-17T18:00:03.61836966Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 CPU architecture: x86_64 \n","stream":"stderr","time":"2024-10-17T18:00:03.618373718Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Target environment: gnu \n","stream":"stderr","time":"2024-10-17T18:00:03.618377325Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 CPU: AMD EPYC-Milan Processor \n","stream":"stderr","time":"2024-10-17T18:00:03.618381142Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 CPU cores: 16 \n","stream":"stderr","time":"2024-10-17T18:00:03.618385029Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Memory: 127802MB \n","stream":"stderr","time":"2024-10-17T18:00:03.618388586Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Kernel: 5.15.0-122-generic \n","stream":"stderr","time":"2024-10-17T18:00:03.618392223Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Linux distribution: Ubuntu 22.04.4 LTS \n","stream":"stderr","time":"2024-10-17T18:00:03.618396071Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_sysinfo: [🌀Parachain] 💻 Virtual machine: yes \n","stream":"stderr","time":"2024-10-17T18:00:03.618399988Z"}
{"log":"2024-10-17 18:00:03.594 INFO main sc_service::builder: [🌀Parachain] 📦 Highest known block at #3605265 \n","stream":"stderr","time":"2024-10-17T18:00:03.618403916Z"}
{"log":"2024-10-17 18:00:03.594 INFO tokio-runtime-worker substrate_prometheus_endpoint: [🌀Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615 \n","stream":"stderr","time":"2024-10-17T18:00:03.618407582Z"}
{"log":"2024-10-17 18:00:03.600 INFO main sc_rpc_server: [🌀Parachain] Running JSON-RPC server: addr=0.0.0.0:9933, allowed origins=["*"] \n","stream":"stderr","time":"2024-10-17T18:00:03.618424805Z"}
{"log":"2024-10-17 18:00:03.602 INFO main sc_sysinfo: [🌀Parachain] 🏁 CPU score: 1018.55 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.618428742Z"}
{"log":"2024-10-17 18:00:03.602 INFO main sc_sysinfo: [🌀Parachain] 🏁 Memory score: 18.82 GiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.618432549Z"}
{"log":"2024-10-17 18:00:03.602 INFO main sc_sysinfo: [🌀Parachain] 🏁 Disk score (seq. writes): 316.86 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.618437498Z"}
{"log":"2024-10-17 18:00:03.602 INFO main sc_sysinfo: [🌀Parachain] 🏁 Disk score (rand. writes): 141.91 MiBs \n","stream":"stderr","time":"2024-10-17T18:00:03.618441276Z"}
{"log":"2024-10-17 18:00:03.606 INFO tokio-runtime-worker libp2p_mdns::behaviour: [🌀Parachain] discovered: 12D3KooWGt6HjC1wDi1zaca7PdPv3H3LWK7L9WzJrE8D7Ftqhgsc /ip4/172.18.0.2/tcp/30334/ws \n","stream":"stderr","time":"2024-10-17T18:00:03.618444983Z"}
{"log":"2024-10-17 18:00:03.606 INFO tokio-runtime-worker libp2p_mdns::behaviour: [Relaychain] discovered: 12D3KooWAqQWi5UaoNt4Biqavgp4WQSdczq9vacZtdfDSzgUtnmR /ip4/172.18.0.2/tcp/30333/ws \n","stream":"stderr","time":"2024-10-17T18:00:03.618449412Z"}
{"log":"2024-10-17 18:00:07.373 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605326 (0xf4c4…890e) \n","stream":"stderr","time":"2024-10-17T18:00:07.375141066Z"}
{"log":"2024-10-17 18:00:08.558 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:08.577706228Z"}
{"log":"2024-10-17 18:00:08.601 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (2 peers), best: #3605326 (0xf4c4…890e), finalized #0 (0xe0e4…91f0), ⬇ 108.4kiB/s ⬆ 1.2kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:08.627832013Z"}
{"log":"2024-10-17 18:00:13.559 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:13.586917162Z"}
{"log":"2024-10-17 18:00:13.602 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (2 peers), best: #3605326 (0xf4c4…890e), finalized #0 (0xe0e4…91f0), ⬇ 0.6kiB/s ⬆ 94 B/s \n","stream":"stderr","time":"2024-10-17T18:00:13.63701708Z"}
{"log":"2024-10-17 18:00:15.704 INFO tokio-runtime-worker sub-libp2p: [🌀Parachain] 🔍 Discovered new external address for our node: /ip4/86.109.11.10/tcp/30333/ws/p2p/12D3KooWAqQWi5UaoNt4Biqavgp4WQSdczq9vacZtdfDSzgUtnmR \n","stream":"stderr","time":"2024-10-17T18:00:15.74080493Z"}
{"log":"2024-10-17 18:00:18.560 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:18.595900439Z"}
{"log":"2024-10-17 18:00:18.602 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (3 peers), best: #3605326 (0xf4c4…890e), finalized #0 (0xe0e4…91f0), ⬇ 0.9kiB/s ⬆ 0.4kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:18.646038777Z"}
{"log":"2024-10-17 18:00:23.560 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:23.605020639Z"}
{"log":"2024-10-17 18:00:23.603 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605326 (0xf4c4…890e), finalized #0 (0xe0e4…91f0), ⬇ 1.2kiB/s ⬆ 0.4kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:23.605061116Z"}
{"log":"2024-10-17 18:00:25.556 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605327 (0xd757…9717) \n","stream":"stderr","time":"2024-10-17T18:00:25.559085571Z"}
{"log":"2024-10-17 18:00:25.876 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605327 (0xabdc…fb7b) \n","stream":"stderr","time":"2024-10-17T18:00:25.909847288Z"}
{"log":"2024-10-17 18:00:28.561 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:28.565233817Z"}
{"log":"2024-10-17 18:00:28.603 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605327 (0xd757…9717), finalized #0 (0xe0e4…91f0), ⬇ 14.9kiB/s ⬆ 0.2kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:28.615317858Z"}
{"log":"2024-10-17 18:00:33.561 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:33.575349984Z"}
{"log":"2024-10-17 18:00:33.604 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605327 (0xd757…9717), finalized #0 (0xe0e4…91f0), ⬇ 39 B/s ⬆ 86 B/s \n","stream":"stderr","time":"2024-10-17T18:00:33.625448987Z"}
{"log":"2024-10-17 18:00:37.381 INFO tokio-runtime-worker sc_informant: [🌀Parachain] ♻️ Reorg on #3605327,0xd757…9717 to #3605328,0xcebf…1af9, common ancestor #3605326,0xf4c4…890e \n","stream":"stderr","time":"2024-10-17T18:00:37.382083298Z"}
{"log":"2024-10-17 18:00:37.381 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605328 (0xcebf…1af9) \n","stream":"stderr","time":"2024-10-17T18:00:37.382111582Z"}
{"log":"2024-10-17 18:00:37.700 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605328 (0x2806…b22a) \n","stream":"stderr","time":"2024-10-17T18:00:37.732780095Z"}
{"log":"2024-10-17 18:00:38.561 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:38.584569885Z"}
{"log":"2024-10-17 18:00:38.604 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605328 (0xcebf…1af9), finalized #0 (0xe0e4…91f0), ⬇ 16.1kiB/s ⬆ 0.4kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:38.634657884Z"}
{"log":"2024-10-17 18:00:43.562 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:43.594719797Z"}
{"log":"2024-10-17 18:00:43.605 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605328 (0xcebf…1af9), finalized #0 (0xe0e4…91f0), ⬇ 26 B/s ⬆ 88 B/s \n","stream":"stderr","time":"2024-10-17T18:00:43.644832285Z"}
{"log":"2024-10-17 18:00:48.562 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:48.604083821Z"}
{"log":"2024-10-17 18:00:48.605 INFO tokio-runtime-worker substrate: [🌀Parachain] 💤 Idle (4 peers), best: #3605328 (0xcebf…1af9), finalized #0 (0xe0e4…91f0), ⬇ 39 B/s ⬆ 0.1kiB/s \n","stream":"stderr","time":"2024-10-17T18:00:48.654217402Z"}
{"log":"2024-10-17 18:00:49.212 INFO tokio-runtime-worker sc_informant: [🌀Parachain] ♻️ Reorg on #3605328,0xcebf…1af9 to #3605329,0xdef7…3deb, common ancestor #3605327,0xabdc…fb7b \n","stream":"stderr","time":"2024-10-17T18:00:49.255339598Z"}
{"log":"2024-10-17 18:00:49.212 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605329 (0xdef7…3deb) \n","stream":"stderr","time":"2024-10-17T18:00:49.255364315Z"}
{"log":"2024-10-17 18:00:49.533 INFO tokio-runtime-worker substrate: [🌀Parachain] ✨ Imported #3605329 (0xe077…3530) \n","stream":"stderr","time":"2024-10-17T18:00:49.555935975Z"}
{"log":"2024-10-17 18:00:53.563 INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (0 peers), best: #0 (0xe1ea…9443), finalized #0 (0xe1ea…9443), ⬇ 0 ⬆ 0 \n","stream":"stderr","time":"2024-10-17T18:00:53.613253617Z"}
'

This is related to previous issue reported in
#1697

[sgrsaga]

Chain sync fast till the blocks around Block#3628526. Then no reasonable progress.
(Timestamp: 2023-08-17 08:54:54 (+UTC) | Block Time: 430 days 18 hrs ago)

[wischli]

Even though running testnet nodes is permissionless, it is beyond the scope of this repository to debug synching issues. IMO, there is no reason for externals to run testnet nodes, especially since we will soon deprecate this testnet and move to Paseo Relay network.