thread 'inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired' panicked at fog/overseer/server/tests/inactive_outstanding_key_idle_has_key.rs:43:12:
Timed out waiting for active node to process data; recovery_db has 3 blocks, target is 12
stack backtrace:
2024-06-18 01:43:30.754167827 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8616/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:30.754231365 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8616, mc.peers.addr: 0.0.0.0:8616, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:30.980056717 UTC DEBG Ingesting block #3: 4/4 txs ingested, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:587
2024-06-18 01:43:30.980172161 UTC INFO add_block_data, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:596
2024-06-18 01:43:30.988531961 UTC DEBG Peer backup insecure-igp://0.0.0.0:8616/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
   0: rust_begin_unwind
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/panicking.rs:72:14
   2: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_callback
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:308:13
   3: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync_up_to
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:279:9
   4: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:272:9
   5: inactive_outstanding_key_idle_has_key::__wrapped_inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired
             at ./tests/inactive_outstanding_key_idle_has_key.rs:43:5
   6: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired::{{closure}}
             at ./tests/inactive_outstanding_key_idle_has_key.rs:18:1
   7: slog_scope::scope
             at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-scope-4.4.0/lib.rs:248:5
   8: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired
             at ./tests/inactive_outstanding_key_idle_has_key.rs:18:1
   9: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired::{{closure}}
             at ./tests/inactive_outstanding_key_idle_has_key.rs:18:1
  10: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-06-18 01:43:31.049452105 UTC INFO Disabling overseer worker, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_overseer_server::service, mc.src: fog/overseer/server/src/service.rs:114
2024-06-18 01:43:31.989204106 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:31.989291849 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8611/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:31.989366137 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8611, mc.peers.addr: 0.0.0.0:8611, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.020653063 UTC DEBG Peer backup insecure-igp://0.0.0.0:8611/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:32.021076038 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8616/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:32.021152841 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8616, mc.peers.addr: 0.0.0.0:8616, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.049328327 UTC DEBG Peer backup insecure-igp://0.0.0.0:8616/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:33.701691070 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:34.956138915 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:36.026168097 UTC INFO Overseer worker thread stopping., mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_overseer_server::worker, mc.src: fog/overseer/server/src/worker.rs:165
2024-06-18 01:43:37.050259824 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:37.050355272 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8611/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:37.050412298 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8611, mc.peers.addr: 0.0.0.0:8611, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:37.079614111 UTC DEBG Peer backup insecure-igp://0.0.0.0:8611/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:37.080040012 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8616/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:37.080127464 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8616, mc.peers.addr: 0.0.0.0:8616, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:37.108544580 UTC DEBG Peer backup insecure-igp://0.0.0.0:8616/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:38.702237581 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:39.142227990 UTC INFO Succeeded writing block 3 to the database, invocation id Some(IngestInvocationId(1)) ingress key CompressedRistrettoPublic(8ae1f2a7dedd3d05086473622166bd6f075f79bf83d5d2ffe128f4edf464281f), mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:636
2024-06-18 01:43:39.142302358 UTC DEBG Incremented next block index: 4, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller_state, mc.src: fog/ingest/server/src/controller_state.rs:155
2024-06-18 01:43:39.142345398 UTC DEBG Controller state: { mode: Active, next_block_index: 4, pubkey_expiry_window: 10, ingest_invocation_id: 1, peers: [insecure-igp://0.0.0.0:8606/, insecure-igp://0.0.0.0:8611/, insecure-igp://0.0.0.0:8616/] }, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:647
2024-06-18 01:43:39.142386424 UTC INFO Finished ingesting block #3, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:659
2024-06-18 01:43:39.159270857 UTC DEBG Writing state file to StateFile { file_path: "/tmp/.tmpcnMaux/mc-fog-ingest-state-0" }, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1314
2024-06-18 01:43:39.956738761 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:41.787848338 UTC INFO Stop Requested: Polling loop stopped at block number 4, is_idle false, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:42.109407022 UTC DEBG Peer checkup thread stop requested., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:42.298183524 UTC DEBG Report cache thread stop requested., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:42.990299863 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:43.702692050 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:43.957082955 UTC DEBG Peer checkup thread stop requested., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:44.138322097 UTC DEBG Report cache thread stop requested., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:44.989152081 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:45.702896727 UTC DEBG Peer checkup thread stop requested., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:46.507489649 UTC DEBG Report cache thread stop requested., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_has_key::inactive_oustanding_key_idle_node_has_original_key_node_is_activated_and_key_is_not_reported_lost_or_retired, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289

thread 'active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated' panicked at fog/overseer/server/tests/retired_key_not_outstanding_idles_have_different_keys.rs:47:12:
Timed out waiting for active node to process data; recovery_db has 4 blocks, target is 12
stack backtrace:
2024-06-18 01:43:22.688645401 UTC DEBG Writing state file to StateFile { file_path: "/tmp/.tmpCIyw9m/mc-fog-ingest-state-0" }, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1314
   0: rust_begin_unwind
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/panicking.rs:72:14
   2: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_callback
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:308:13
   3: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync_up_to
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:279:9
   4: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:272:9
   5: retired_key_not_outstanding_idles_have_different_keys::__wrapped_active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated
             at ./tests/retired_key_not_outstanding_idles_have_different_keys.rs:47:5
   6: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated::{{closure}}
             at ./tests/retired_key_not_outstanding_idles_have_different_keys.rs:17:1
   7: slog_scope::scope
             at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-scope-4.4.0/lib.rs:248:5
   8: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated
             at ./tests/retired_key_not_outstanding_idles_have_different_keys.rs:17:1
   9: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated::{{closure}}
             at ./tests/retired_key_not_outstanding_idles_have_different_keys.rs:17:1
  10: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-06-18 01:43:22.943831597 UTC INFO Disabling overseer worker, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_overseer_server::service, mc.src: fog/overseer/server/src/service.rs:114
2024-06-18 01:43:23.796413134 UTC INFO Overseer worker thread stopping., mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_overseer_server::worker, mc.src: fog/overseer/server/src/worker.rs:165
2024-06-18 01:43:26.023914195 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:26.442915834 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:26.900191254 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:31.024436723 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:31.443435134 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:36.025001685 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:36.443933197 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:39.957947141 UTC INFO Stop Requested: Polling loop stopped at block number 4, is_idle false, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:39.958062055 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8761/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:39.958127336 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8761, mc.peers.addr: 0.0.0.0:8761, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:39.987282716 UTC DEBG Peer backup insecure-igp://0.0.0.0:8761/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:39.987788247 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8766/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:39.987880067 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8766, mc.peers.addr: 0.0.0.0:8766, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:40.016348459 UTC DEBG Peer backup insecure-igp://0.0.0.0:8766/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:41.016916688 UTC DEBG Peer checkup thread stop requested., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:41.025509257 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:41.293262892 UTC DEBG Report cache thread stop requested., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:41.444455981 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:42.061015530 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:43.025708973 UTC DEBG Peer checkup thread stop requested., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:43.052740935 UTC DEBG Report cache thread stop requested., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:43.980815379 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:44.444781242 UTC DEBG Peer checkup thread stop requested., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:45.286328069 UTC DEBG Report cache thread stop requested., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_different_keys::active_key_is_retired_not_outstanding_idle_nodes_have_different_keys_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289

thread 'active_key_is_retired_not_outstanding_new_key_is_set_node_activated' panicked at fog/overseer/server/tests/retired_key_not_outstanding_idles_have_same_key.rs:47:12:
Timed out waiting for active node to process data; recovery_db has 4 blocks, target is 12
stack backtrace:
2024-06-18 01:43:26.896430354 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
   0: rust_begin_unwind
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/panicking.rs:72:14
   2: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_callback
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:308:13
   3: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync_up_to
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:279:9
   4: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:272:9
   5: retired_key_not_outstanding_idles_have_same_key::__wrapped_active_key_is_retired_not_outstanding_new_key_is_set_node_activated
             at ./tests/retired_key_not_outstanding_idles_have_same_key.rs:47:5
   6: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated::{{closure}}
             at ./tests/retired_key_not_outstanding_idles_have_same_key.rs:18:1
   7: slog_scope::scope
             at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-scope-4.4.0/lib.rs:248:5
   8: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated
             at ./tests/retired_key_not_outstanding_idles_have_same_key.rs:18:1
   9: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated::{{closure}}
             at ./tests/retired_key_not_outstanding_idles_have_same_key.rs:18:1
  10: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-06-18 01:43:26.937257729 UTC INFO Disabling overseer worker, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_overseer_server::service, mc.src: fog/overseer/server/src/service.rs:114
2024-06-18 01:43:28.701875839 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:29.956176907 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:32.353124322 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8811/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:32.353254573 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8811, mc.peers.addr: 0.0.0.0:8811, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.371044543 UTC DEBG Now ingesting block #4 (id = BlockID(35d8b4f0e307bf9497a1566aa5b27d741ea8161116e2d3eb12ad8051da216a83)), mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:428
2024-06-18 01:43:32.375051104 UTC INFO publishing report to DB, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1250
2024-06-18 01:43:32.375756905 UTC INFO update was a no-op, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_sql_recovery_db, mc.src: fog/sql_recovery_db/src/lib.rs:1194
2024-06-18 01:43:32.375991911 UTC INFO check ingress key passed, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_sql_recovery_db, mc.src: fog/sql_recovery_db/src/lib.rs:1197
2024-06-18 01:43:32.376054638 UTC INFO Got status for key: IngressPublicKeyStatus { start_block: 1, pubkey_expiry: 11, retired: true, lost: false }, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_sql_recovery_db, mc.src: fog/sql_recovery_db/src/lib.rs:1214
2024-06-18 01:43:32.376122233 UTC INFO Cannot publish key because it is retired, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_sql_recovery_db, mc.src: fog/sql_recovery_db/src/lib.rs:1216
2024-06-18 01:43:32.578352723 UTC DEBG Ingesting block #4: 4/4 txs ingested, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:587
2024-06-18 01:43:32.578425618 UTC INFO add_block_data, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:596
2024-06-18 01:43:32.587042762 UTC DEBG Peer backup insecure-igp://0.0.0.0:8811/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:32.587649639 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8816/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:32.587697638 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8816, mc.peers.addr: 0.0.0.0:8816, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.616378454 UTC DEBG Peer backup insecure-igp://0.0.0.0:8816/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:33.616920777 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:33.616999323 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8811/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:33.617058072 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8811, mc.peers.addr: 0.0.0.0:8811, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:33.645923901 UTC DEBG Peer backup insecure-igp://0.0.0.0:8811/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:33.646315038 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8816/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:33.646394435 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8816, mc.peers.addr: 0.0.0.0:8816, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:33.674538374 UTC DEBG Peer backup insecure-igp://0.0.0.0:8816/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:33.702403575 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:34.956706220 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:37.391365102 UTC INFO Overseer worker thread stopping., mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_overseer_server::worker, mc.src: fog/overseer/server/src/worker.rs:165
2024-06-18 01:43:38.675425287 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:38.675531374 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8811/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:38.675592087 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8811, mc.peers.addr: 0.0.0.0:8811, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:38.702969731 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:38.704859465 UTC DEBG Peer backup insecure-igp://0.0.0.0:8811/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:38.705255992 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8816/, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:38.705334509 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8816, mc.peers.addr: 0.0.0.0:8816, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:38.733485400 UTC DEBG Peer backup insecure-igp://0.0.0.0:8816/ was confirmed, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:39.142245480 UTC INFO Succeeded writing block 4 to the database, invocation id Some(IngestInvocationId(1)) ingress key CompressedRistrettoPublic(da1cb6fa4c43bf45885cdf3fa5a2291839da53a7fb8cbced624029ec9fbbfe36), mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:636
2024-06-18 01:43:39.142338243 UTC DEBG Incremented next block index: 5, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller_state, mc.src: fog/ingest/server/src/controller_state.rs:155
2024-06-18 01:43:39.142406229 UTC DEBG Controller state: { mode: Active, next_block_index: 5, pubkey_expiry_window: 10, ingest_invocation_id: 1, peers: [insecure-igp://0.0.0.0:8806/, insecure-igp://0.0.0.0:8811/, insecure-igp://0.0.0.0:8816/] }, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:647
2024-06-18 01:43:39.142474636 UTC INFO Finished ingesting block #4, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:659
2024-06-18 01:43:39.159434767 UTC DEBG Writing state file to StateFile { file_path: "/tmp/.tmpJY0RPj/mc-fog-ingest-state-0" }, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1314
2024-06-18 01:43:39.957254248 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:41.787870486 UTC INFO Stop Requested: Polling loop stopped at block number 5, is_idle false, mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:42.734262564 UTC DEBG Peer checkup thread stop requested., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:43.295038454 UTC DEBG Report cache thread stop requested., mc.node_id: 0, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:43.703464623 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:44.251215560 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:44.957681628 UTC DEBG Peer checkup thread stop requested., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:45.100597914 UTC DEBG Report cache thread stop requested., mc.node_id: 1, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:45.988066522 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:46.703772811 UTC DEBG Peer checkup thread stop requested., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:47.517164057 UTC DEBG Report cache thread stop requested., mc.node_id: 2, mc.test_name: retired_key_not_outstanding_idles_have_same_key::active_key_is_retired_not_outstanding_new_key_is_set_node_activated, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289

thread 'inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost' panicked at fog/overseer/server/tests/inactive_outstanding_key_idle_does_not_have_key.rs:40:12:
Timed out waiting for active node to process data; recovery_db has 3 blocks, target is 12
stack backtrace:
2024-06-18 01:43:30.754182558 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8566/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:30.754241267 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8566, mc.peers.addr: 0.0.0.0:8566, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:30.965714178 UTC DEBG Ingesting block #3: 4/4 txs ingested, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:587
2024-06-18 01:43:30.965820615 UTC INFO add_block_data, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:596
2024-06-18 01:43:30.974314914 UTC DEBG Peer backup insecure-igp://0.0.0.0:8566/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
   0: rust_begin_unwind
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/panicking.rs:72:14
   2: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_callback
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:308:13
   3: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync_up_to
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:279:9
   4: mc_fog_ingest_server_test_utils::IngestServerTestHelper::wait_till_recovery_db_in_sync
             at /__w/mobilecoin/mobilecoin/fog/ingest/server/test-utils/src/lib.rs:272:9
   5: inactive_outstanding_key_idle_does_not_have_key::__wrapped_inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost
             at ./tests/inactive_outstanding_key_idle_does_not_have_key.rs:40:5
   6: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost::{{closure}}
             at ./tests/inactive_outstanding_key_idle_does_not_have_key.rs:18:1
   7: slog_scope::scope
             at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/slog-scope-4.4.0/lib.rs:248:5
   8: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost
             at ./tests/inactive_outstanding_key_idle_does_not_have_key.rs:18:1
   9: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost::{{closure}}
             at ./tests/inactive_outstanding_key_idle_does_not_have_key.rs:18:1
  10: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/ca62d2c445628587660ae48013f460b08b1f5552/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-06-18 01:43:31.025317657 UTC INFO Disabling overseer worker, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_overseer_server::service, mc.src: fog/overseer/server/src/service.rs:114
2024-06-18 01:43:31.975010421 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:31.975098314 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8561/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:31.975170228 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8561, mc.peers.addr: 0.0.0.0:8561, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.006336238 UTC DEBG Peer backup insecure-igp://0.0.0.0:8561/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:32.006864590 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8566/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:32.006956490 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8566, mc.peers.addr: 0.0.0.0:8566, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:32.035568769 UTC DEBG Peer backup insecure-igp://0.0.0.0:8566/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:33.702413268 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:34.956005855 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:36.012554278 UTC INFO Overseer worker thread stopping., mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_overseer_server::worker, mc.src: fog/overseer/server/src/worker.rs:165
2024-06-18 01:43:37.036451455 UTC DEBG Checking up on peers..., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:37.036558664 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8561/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:37.036619367 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8561, mc.peers.addr: 0.0.0.0:8561, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:37.066198391 UTC DEBG Peer backup insecure-igp://0.0.0.0:8561/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:37.066614173 UTC DEBG Checking on peer: insecure-igp://0.0.0.0:8566/, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1363
2024-06-18 01:43:37.066689984 UTC WARN Creating insecure gRPC connection to 0.0.0.0:8566, mc.peers.addr: 0.0.0.0:8566, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_util_grpc::grpcio_extensions, mc.src: util/grpc/src/grpcio_extensions.rs:48
2024-06-18 01:43:37.095082253 UTC DEBG Peer backup insecure-igp://0.0.0.0:8566/ was confirmed, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1378
2024-06-18 01:43:38.703002899 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:39.142239466 UTC INFO Succeeded writing block 3 to the database, invocation id Some(IngestInvocationId(1)) ingress key CompressedRistrettoPublic(def661e7dde6fcb2b5ef7666935ebce1081ffcbc39009b30a5ce0d27fd6b630f), mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:636
2024-06-18 01:43:39.142324925 UTC DEBG Incremented next block index: 4, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller_state, mc.src: fog/ingest/server/src/controller_state.rs:155
2024-06-18 01:43:39.142380879 UTC DEBG Controller state: { mode: Active, next_block_index: 4, pubkey_expiry_window: 10, ingest_invocation_id: 1, peers: [insecure-igp://0.0.0.0:8556/, insecure-igp://0.0.0.0:8561/, insecure-igp://0.0.0.0:8566/] }, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:647
2024-06-18 01:43:39.142433476 UTC INFO Finished ingesting block #3, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:659
2024-06-18 01:43:39.159689758 UTC DEBG Writing state file to StateFile { file_path: "/tmp/.tmpvcVLOC/mc-fog-ingest-state-0" }, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::controller, mc.src: fog/ingest/server/src/controller.rs:1314
2024-06-18 01:43:39.956555183 UTC DEBG Checking up on peers..., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:41.787862685 UTC INFO Stop Requested: Polling loop stopped at block number 4, is_idle false, mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:42.095978476 UTC DEBG Peer checkup thread stop requested., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:42.294231351 UTC DEBG Report cache thread stop requested., mc.node_id: 0, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:43.017549100 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:43.703495363 UTC DEBG Checking up on peers..., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:235
2024-06-18 01:43:43.956901447 UTC DEBG Peer checkup thread stop requested., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:44.089629875 UTC DEBG Report cache thread stop requested., mc.node_id: 1, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289
2024-06-18 01:43:44.990090824 UTC INFO Stop Requested: Polling loop stopped at block number 0, is_idle true, mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:75
2024-06-18 01:43:45.703704197 UTC DEBG Peer checkup thread stop requested., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:229
2024-06-18 01:43:46.556334185 UTC DEBG Report cache thread stop requested., mc.node_id: 2, mc.test_name: inactive_outstanding_key_idle_does_not_have_key::inactive_oustanding_key_idle_node_does_not_have_key_idle_node_is_activated_and_original_key_is_reported_lost, mc.module: mc_fog_ingest_server::worker, mc.src: fog/ingest/server/src/worker.rs:289