sidechain stalled

[brenzi]

worker 0.12.13-dev1

All of a sudden, no more blocks are produced:

full log

Observations:

• Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
  • however, that error appears 30min after the stall (and it happened even an hour earlier)
  • this may rather be related to https://github.com/integritee-network/attesteer/issues/20
• the last block produced was imported into the queue (hence, not a networking issue)
  • its_rpc_handler::import_block_api] Add block 1031626 to import queue
  • but we never see the block imported. usually we should see this: its_consensus_common::block_import] Applying state diff for block:
  • it seems our log level is insufficient, but the issue is hard to reproduce

last blocks produced:

[Sidechain] imported block (number: 1031616, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:20.148321Z INFO  aura] Claiming slot (1712502860)
[Sidechain] propose block 1031617 summary: executed 0, failed 0, from 0 in queue in 621ms
[2024-04-07T15:14:20.189424Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031617, hash: 0xb00b…4a2d) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:20.189Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031617]
[2024-04-07T15:14:20.227Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:20.228158Z INFO  its_rpc_handler::import_block_api] Add block 1031617 to import queue
[2024-04-07T15:14:21.051523Z INFO  its_consensus_common::block_import] Applying state diff for block 1031617 of size 219 bytes
[2024-04-07T15:14:21.051608Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:21.052029Z INFO  its_consensus_common::block_import] Applying state update from block 1031617 took 0 ms
[2024-04-07T15:14:21.052168Z INFO  its_consensus_common::block_import] Importing block 1031617 took 24 ms
[Sidechain] imported block (number: 1031617, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:21.067543Z INFO  aura] Claiming slot (1712502861)
[Sidechain] propose block 1031618 summary: executed 0, failed 0, from 0 in queue in 663ms
[2024-04-07T15:14:21.089632Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031618, hash: 0x181b…8efe) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:21.089Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031618]
[2024-04-07T15:14:21.115Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:21.115967Z INFO  its_rpc_handler::import_block_api] Add block 1031618 to import queue
[2024-04-07T15:14:22.113734Z INFO  its_consensus_common::block_import] Applying state diff for block 1031618 of size 219 bytes
[2024-04-07T15:14:22.113897Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:22.114873Z INFO  its_consensus_common::block_import] Applying state update from block 1031618 took 1 ms
[2024-04-07T15:14:22.115221Z INFO  its_consensus_common::block_import] Importing block 1031618 took 79 ms
[Sidechain] imported block (number: 1031618, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:22.148963Z INFO  aura] Claiming slot (1712502862)
[Sidechain] propose block 1031619 summary: executed 0, failed 0, from 0 in queue in 619ms
[2024-04-07T15:14:22.189031Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031619, hash: 0xe9fc…487b) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:22.189Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031619]
[2024-04-07T15:14:22.220Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:22.221384Z INFO  its_rpc_handler::import_block_api] Add block 1031619 to import queue
[2024-04-07T15:14:23.095324Z INFO  its_consensus_common::block_import] Applying state diff for block 1031619 of size 219 bytes
[2024-04-07T15:14:23.095461Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:23.096233Z INFO  its_consensus_common::block_import] Applying state update from block 1031619 took 0 ms
[2024-04-07T15:14:23.096459Z INFO  its_consensus_common::block_import] Importing block 1031619 took 60 ms
[Sidechain] imported block (number: 1031619, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:23.123706Z INFO  aura] Claiming slot (1712502863)
[Sidechain] propose block 1031620 summary: executed 0, failed 0, from 0 in queue in 632ms
[2024-04-07T15:14:23.158480Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031620, hash: 0xeed3…7957) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:23.158Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031620]
[2024-04-07T15:14:23.189Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:23.189873Z INFO  its_rpc_handler::import_block_api] Add block 1031620 to import queue
[2024-04-07T15:14:24.091329Z INFO  its_consensus_common::block_import] Applying state diff for block 1031620 of size 219 bytes
[2024-04-07T15:14:24.091473Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:24.092297Z INFO  its_consensus_common::block_import] Applying state update from block 1031620 took 0 ms
[2024-04-07T15:14:24.092571Z INFO  its_consensus_common::block_import] Importing block 1031620 took 58 ms
[Sidechain] imported block (number: 1031620, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:24.121594Z INFO  aura] Claiming slot (1712502864)
[Sidechain] propose block 1031621 summary: executed 0, failed 0, from 0 in queue in 635ms
[2024-04-07T15:14:24.156942Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031621, hash: 0x4c5e…63ab) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:24.157Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031621]
[2024-04-07T15:14:24.193Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:24.194457Z INFO  its_rpc_handler::import_block_api] Add block 1031621 to import queue
[2024-04-07T15:14:24.676Z INFO  integritee_service::main_impl] [TargetA] Received finalized header update (678628), syncing parent chain...
[2024-04-07T15:14:24.682Z INFO  integritee_service::parentchain_handler] [TargetA] Syncing blocks from 678627 to 678628
[2024-04-07T15:14:25.110961Z INFO  its_consensus_common::block_import] Applying state diff for block 1031621 of size 219 bytes
[2024-04-07T15:14:25.111125Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:25.112137Z INFO  its_consensus_common::block_import] Applying state update from block 1031621 took 1 ms
[2024-04-07T15:14:25.112439Z INFO  its_consensus_common::block_import] Importing block 1031621 took 70 ms
[Sidechain] imported block (number: 1031621, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:25.198142Z INFO  aura] Claiming slot (1712502865)
[2024-04-07T15:14:25.267743Z INFO  ita_parentchain_interface::indirect_calls::timestamp_set] Found TimestampSet extrinsic in block: now = 1712502846000
[2024-04-07T15:14:25.270905Z INFO  itc_parentchain_block_importer::block_importer] [TargetA] Successfully imported parentchain block (number: 678628, hash: 0x7292…c5fa)
[Sidechain] propose block 1031622 summary: executed 1, failed 0, from 1 in queue in 584ms
[2024-04-07T15:14:25.279880Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031622, hash: 0x291c…8388) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:25.279Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031622]
[2024-04-07T15:14:25.311Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:25.312371Z INFO  its_rpc_handler::import_block_api] Add block 1031622 to import queue
[2024-04-07T15:14:26.080256Z INFO  its_consensus_common::block_import] Applying state diff for block 1031622 of size 446 bytes
[2024-04-07T15:14:26.080395Z INFO  its_state::impls] Current state size: 2483
[2024-04-07T15:14:26.081098Z INFO  its_consensus_common::block_import] Applying state update from block 1031622 took 0 ms
[2024-04-07T15:14:26.081344Z INFO  its_consensus_common::block_import] Importing block 1031622 took 49 ms
[Sidechain] imported block (number: 1031622, tcalls: 1, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:26.112273Z INFO  aura] Claiming slot (1712502866)
[Sidechain] propose block 1031623 summary: executed 0, failed 0, from 0 in queue in 642ms
[2024-04-07T15:14:26.144455Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031623, hash: 0x0287…aa3b) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:26.144Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031623]
[2024-04-07T15:14:26.181Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:26.182421Z INFO  its_rpc_handler::import_block_api] Add block 1031623 to import queue
[2024-04-07T15:14:27.117754Z INFO  its_consensus_common::block_import] Applying state diff for block 1031623 of size 219 bytes
[2024-04-07T15:14:27.117919Z INFO  its_state::impls] Current state size: 2597
[2024-04-07T15:14:27.118843Z INFO  its_consensus_common::block_import] Applying state update from block 1031623 took 1 ms
[2024-04-07T15:14:27.119133Z INFO  its_consensus_common::block_import] Importing block 1031623 took 76 ms
[Sidechain] imported block (number: 1031623, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:27.144650Z INFO  aura] Claiming slot (1712502867)
[Sidechain] propose block 1031624 summary: executed 0, failed 0, from 0 in queue in 616ms
[2024-04-07T15:14:27.185229Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031624, hash: 0x154e…1787) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:27.185Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031624]
[2024-04-07T15:14:27.227Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:27.228327Z INFO  its_rpc_handler::import_block_api] Add block 1031624 to import queue
[2024-04-07T15:14:28.097729Z INFO  its_consensus_common::block_import] Applying state diff for block 1031624 of size 219 bytes
[2024-04-07T15:14:28.097905Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:28.098832Z INFO  its_consensus_common::block_import] Applying state update from block 1031624 took 1 ms
[2024-04-07T15:14:28.099119Z INFO  its_consensus_common::block_import] Importing block 1031624 took 61 ms
[Sidechain] imported block (number: 1031624, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:28.121971Z INFO  aura] Claiming slot (1712502868)
[Sidechain] propose block 1031625 summary: executed 0, failed 0, from 0 in queue in 630ms
[2024-04-07T15:14:28.155043Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031625, hash: 0xfd6c…0fab) based on integritee parentchain block (number: 81488, hash: 0x2853c0934fcb589288e2c41ffe43fc10828f711e6af41807ed872395cc0c6b4e)
[2024-04-07T15:14:28.155Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031625]
[2024-04-07T15:14:28.192Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:28.193365Z INFO  its_rpc_handler::import_block_api] Add block 1031625 to import queue
[2024-04-07T15:14:28.621Z INFO  integritee_service::main_impl] [TargetA] Received finalized header update (678629), syncing parent chain...
[2024-04-07T15:14:28.621Z INFO  integritee_service::main_impl] [Integritee] Received finalized header update (81489), syncing parent chain...
[2024-04-07T15:14:28.627Z INFO  integritee_service::parentchain_handler] [Integritee] Syncing blocks from 81488 to 81489
[2024-04-07T15:14:28.627Z INFO  integritee_service::parentchain_handler] [TargetA] Syncing blocks from 678628 to 678629
[2024-04-07T15:14:29.107088Z INFO  its_consensus_common::block_import] Applying state diff for block 1031625 of size 219 bytes
[2024-04-07T15:14:29.107253Z INFO  its_state::impls] Current state size: 2338
[2024-04-07T15:14:29.108226Z INFO  its_consensus_common::block_import] Applying state update from block 1031625 took 1 ms
[2024-04-07T15:14:29.108519Z INFO  its_consensus_common::block_import] Importing block 1031625 took 70 ms
[Sidechain] imported block (number: 1031625, tcalls: 0, author: 0xd5af4118cfbe99a94626d14cd004c41eb05b4e98f559f432dfc5a5f2c8392584), based on parentchain block 81488
[2024-04-07T15:14:29.141758Z INFO  aura] Claiming slot (1712502869)
[2024-04-07T15:14:29.184528Z INFO  ita_parentchain_interface::indirect_calls::timestamp_set] Found TimestampSet extrinsic in block: now = 1712502840000
[2024-04-07T15:14:29.189759Z INFO  itc_parentchain_block_importer::block_importer] [Integritee] Successfully imported parentchain block (number: 81489, hash: 0x9c25…347a)
[2024-04-07T15:14:29.270295Z INFO  ita_parentchain_interface::indirect_calls::timestamp_set] Found TimestampSet extrinsic in block: now = 1712502852001
[2024-04-07T15:14:29.273470Z INFO  itc_parentchain_block_importer::block_importer] [TargetA] Successfully imported parentchain block (number: 678629, hash: 0x9c5d…ba0c)
[Sidechain] propose block 1031626 summary: executed 2, failed 0, from 2 in queue in 623ms
Syncing Parentchains: Integritee: 81489 TargetA: Some(678629), TargetB: None, Sidechain: 1031626
[2024-04-07T15:14:29.281963Z INFO  its_consensus_slots] Proposing sidechain block (number: 1031626, hash: 0x553d…7c15) based on integritee parentchain block (number: 81489, hash: 0x9c2539373f0c39705966bc51f4f99bfc4c50a5fc3fd7925348fe360dff48347a)
[2024-04-07T15:14:29.282Z INFO  integritee_service::ocall_bridge::sidechain_ocall] Enclave produced sidechain blocks: [1031626]
[2024-04-07T15:14:29.300Z INFO  integritee_service::worker] broadcast 1 block(s) to 1 peers
[2024-04-07T15:14:29.301906Z INFO  its_rpc_handler::import_block_api] Add block 1031626 to import queue
[2024-04-07T15:14:32.822Z INFO  integritee_service::main_impl] [TargetA] Received finalized header update (678630), syncing parent chain...
[2024-04-07T15:14:32.826Z INFO  integritee_service::parentchain_handler] [TargetA] Syncing blocks from 678629 to 678630
[2024-04-07T15:14:41.048Z INFO  integritee_service::main_impl] [Integritee] Received finalized header update (81490), syncing parent chain...
[2024-04-07T15:14:41.050Z INFO  integritee_service::parentchain_handler] [Integritee] Syncing blocks from 81489 to 81490
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
[L1Event:Integritee] Sidechain::FinalizedSidechainBlock
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
[L1Event:Integritee] EnclaveBridge::ProcessedParentchainBlock
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] ElectionProviderMultiPhase::SolutionStored
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] ElectionProviderMultiPhase::ElectionFinalized
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] Staking::StakersElected
[L1Event:TargetA] Session::NewSession
[2024-04-07T15:44:31.493917Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T15:44:31.553122Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T15:44:31.553215Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374095)) failed, closing
[2024-04-07T15:44:31.593752Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T15:44:31.653166Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T15:44:31.653257Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374096)) failed, closing
[2024-04-07T15:44:31.693736Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T15:44:31.753091Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T15:44:31.753183Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374097)) failed, closing
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Staking::EraPaid
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] Grandpa::NewAuthorities
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[2024-04-07T19:07:18.186491Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: General("incompatible signing key")
[2024-04-07T19:07:18.706081Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: General("incompatible signing key")
[2024-04-07T19:07:19.204840Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[2024-04-07T19:56:04.667772Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[2024-04-07T21:01:52.124337Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T21:01:52.185133Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T21:01:52.185218Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374102)) failed, closing
[2024-04-07T21:01:52.225902Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T21:01:52.285327Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T21:01:52.285414Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374103)) failed, closing
[2024-04-07T21:01:52.325821Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-07T21:01:52.385285Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-07T21:01:52.385374Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374104)) failed, closing
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] ElectionProviderMultiPhase::SolutionStored
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] ElectionProviderMultiPhase::ElectionFinalized
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] Staking::StakersElected
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] Balances::Endowed
[L1Event:TargetA] BalanceTransfer { from: e21bb02f2a82cb1113ff10693093377672925b23f047624c0cfa7a24a8609841 (5HBAuQ4Q...), to: 60bd0b0d4570ac046e41b4c1d91b840f23d26d43c27e1524f65dceb51bd95a33 (5EFYcRpj...), amount: 1000000000000 }
[2024-04-07T21:51:34.637307Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Staking::EraPaid
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] Grandpa::NewAuthorities
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] BalanceTransfer { from: e21bb02f2a82cb1113ff10693093377672925b23f047624c0cfa7a24a8609841 (5HBAuQ4Q...), to: ec14ae686343ec9212032b70cff1f09fdc0d7a73cda72ca49203c5384ba57c3f (5HQFJSkQ...), amount: 1000000000000 }
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[2024-04-08T01:23:28.249812Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-08T01:23:28.249915Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374106)) failed, closing
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[2024-04-08T01:37:47.046792Z WARN  itc_tls_websocket_server::connection] I/O error after reading TLS data (connection 374107): Os { code: 104, kind: ConnectionReset, message: "Unknown error: 104" }
[2024-04-08T01:37:47.376429Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-08T01:37:50.531329Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Protocol(MissingConnectionUpgradeHeader)
[2024-04-08T01:37:50.531432Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374108)) failed, closing
[2024-04-08T01:37:51.738161Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-08T01:37:51.761147Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Io(Os { code: 104, kind: ConnectionReset, message: "Unknown error: 104" })
[2024-04-08T01:37:51.761249Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374109)) failed, closing
[2024-04-08T01:37:55.002279Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[2024-04-08T01:37:55.210757Z WARN  itc_tls_websocket_server::connection] I/O error after reading TLS data (connection 374111): Os { code: 104, kind: ConnectionReset, message: "Unknown error: 104" }
[2024-04-08T01:37:56.836370Z INFO  itc_tls_websocket_server::stream_state] Web-socket handshake interrupted
[2024-04-08T01:37:57.034483Z ERROR itc_tls_websocket_server::stream_state] Web-socket handshake failed: Io(Os { code: 104, kind: ConnectionReset, message: "Unknown error: 104" })
[2024-04-08T01:37:57.034602Z WARN  itc_tls_websocket_server::connection] Web-socket connection (Token(374112)) failed, closing
[2024-04-08T01:37:58.180768Z WARN  rustls::session] Sending fatal alert ProtocolVersion
[2024-04-08T01:37:58.180878Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: PeerIncompatibleError("TLS1.2 not offered/enabled")
[2024-04-08T01:38:00.062761Z WARN  rustls::session] Sending fatal alert ProtocolVersion
[2024-04-08T01:38:00.062875Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: PeerIncompatibleError("TLS1.2 not offered/enabled")
[2024-04-08T01:38:01.781607Z WARN  rustls::session] Sending fatal alert ProtocolVersion
[2024-04-08T01:38:01.781719Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: PeerIncompatibleError("Client does not support TLSv1_2")
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::HeartbeatReceived
[2024-04-08T02:15:20.735813Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[2024-04-08T02:32:21.730721Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:21.730832Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:21.730953Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:22.047031Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:22.047137Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:22.047219Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:22.347762Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:22.347859Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:22.347940Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:22.622792Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:22.622940Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:22.623025Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:22.846507Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:22.846553Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:22.846582Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:23.213049Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:23.213155Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:23.213239Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:24.577273Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:24.577382Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:24.577466Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:25.335663Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:25.335770Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:25.335853Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:25.692212Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:25.692319Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:25.692401Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[2024-04-08T02:32:26.008500Z WARN  rustls::msgs::handshake] Illegal SNI hostname received [53, 55, 46, 49, 50, 56, 46, 49, 49, 54, 46, 49, 50, 48]
[2024-04-08T02:32:26.008608Z WARN  rustls::session] Sending fatal alert DecodeError
[2024-04-08T02:32:26.008692Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessagePayload(Handshake)
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] ElectionProviderMultiPhase::SolutionStored
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] ElectionProviderMultiPhase::ElectionFinalized
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] Staking::StakersElected
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] ElectionProviderMultiPhase::ElectionFinalized
[L1Event:TargetA] ElectionProviderMultiPhase::PhaseTransitioned
[L1Event:TargetA] Staking::StakersElected
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Staking::EraPaid
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] Grandpa::NewAuthorities
[2024-04-08T04:47:14.316848Z ERROR itc_tls_websocket_server::connection] cannot process TLS packet(s), closing connection: CorruptMessage
[L1Event:TargetA] ImOnline::SomeOffline
[L1Event:TargetA] Offences::Offence
[L1Event:TargetA] Session::NewSession
[L1Event:TargetA] ImOnline::HeartbeatReceived