Tokio console opt-in via env variable (#2878)

# Description
The `tokio-console` comes with some pretty hefty memory overhead that
caused many pods to crashloop due to OOM issues which I didn't expect.
Therefore we should not have it enabled by default. However, I still
think it's valuable to have support for the feature baked in by default
to reduce the friction of actually using the feature.

# Changes
Now you have to set `TOKIO_CONSOLE=true` to enable it.
This flag does not show up in the `--help` text and only in the repo
`README.md` because otherwise it would have to be added a bit awkwardly
and passed around a lot which I didn't really like in the previous
iteration to begin with.
I think this is okay because it's just a debugging thing and not knowing
about the flag doesn't prevent you from using the binaries correctly but
LMK if you disagree and I should parse the arg via `clap` instead.

## How to test
Ran a local test with and without `TOKIO_CONSOLE=true`

README.md

@@ -119,11 +119,11 @@ ANVIL_IP_ADDR=0.0.0.0 anvil \
 
 ### Profiling
 
-The most important binaries support [tokio-console](https://github.com/tokio-rs/console) to allow you a could look inside the tokio runtime.
+All binaries are compiled with support for [tokio-console](https://github.com/tokio-rs/console) by default to allow you to look inside the tokio runtime.
+However, this feature is not enabled at runtime by default because it comes with a pretty significant memory overhead. To enable it you just have to set the environment variable `TOKIO_CONSOLE=true` and run the binary you want to instrument.
 
-Simply enable the feature by passing `--enable-tokio-console true` when running a binary and then in another shell, run
-
-```
+You can install and run `tokio-console` with:
+```bash
 cargo install --locked tokio-console
 tokio-console
 ```

crates/observe/src/tracing.rs

@@ -70,14 +70,20 @@ fn set_tracing_subscriber(env_filter: &str, stderr_threshold: LevelFilter) {
         }};
     }
 
-    if cfg!(tokio_unstable) {
+    let enable_tokio_console: bool = std::env::var("TOKIO_CONSOLE")
+        .unwrap_or("false".to_string())
+        .parse()
+        .unwrap();
+
+    if cfg!(tokio_unstable) && enable_tokio_console {
         let (env_filter, reload_handle) =
             tracing_subscriber::reload::Layer::new(EnvFilter::new(&initial_filter));
 
         tracing_subscriber::registry()
             .with(console_subscriber::spawn())
             .with(fmt_layer!(env_filter, stderr_threshold))
             .init();
+        tracing::info!("started programm with support for tokio-console");
 
         if cfg!(unix) {
             spawn_reload_handler(initial_filter, reload_handle);
@@ -92,6 +98,7 @@ fn set_tracing_subscriber(env_filter: &str, stderr_threshold: LevelFilter) {
             .with(tracing::level_filters::LevelFilter::TRACE)
             .with(fmt_layer!(env_filter, stderr_threshold))
             .init();
+        tracing::info!("started programm without support for tokio-console");
 
         if cfg!(unix) {
             spawn_reload_handler(initial_filter, reload_handle);