Flaxy test: heavy_tasks_doesnt_block_graphql

[rafal-ch]

heavy_tasks_doesnt_block_graphql seems to occasionally fail in the CI.

Failed build:
https://github.com/FuelLabs/fuel-core/actions/runs/11819543221/job/32929762981?pr=2389

However, the hyper-related error messages seem to be unrelated, because the test intentionally puts a high load on the GraphQL endpoint. What is important is Health check timed out: Elapsed(()).

The test itself seem to be constructed correctly, allowing 5 sec. timeout before concluding that node is unresponsive.

Log:

failures:

---- dos::heavy_tasks_doesnt_block_graphql stdout ----
thread 'dos::heavy_tasks_doesnt_block_graphql' panicked at tests/tests/dos.rs:720:25:
Health check timed out: Elapsed(())
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }
thread 'tokio-runtime-worker' panicked at /home/runner/actions-runner/_work/fuel-core/fuel-core/tests/test-helpers/src/lib.rs:31:61:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(36715), path: "/v1/graphql", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }


failures:
    dos::heavy_tasks_doesnt_block_graphql

[rafal-ch]

The test itself seem to be constructed correctly, allowing 5 sec. timeout before concluding that node is unresponsive.

This may indicate that:

1. 5 sec. is still a bit too low for the CI system (on my local machine the problems starts to randomly appear with timeout set to ~500ms).
2. GraphQL is indeed becoming unresponsive due to other reason

It could be that the node running on CI just cannot reliably fulfill the health check request in time. Instead of bumping the timeout, I'd suggest retrying the health check request 2 or 3 times before bailing.

If the test continues to be flaky we might want to investigate the GraphQL API deeper or just loosen our response time requirements (I'm not sure if 5 secs. is just an arbitrary number or it has some more meaning).