Better ETA estimation for long running benchmarks

[martinus]

I'm currently using hyperfine to benchmark something where a single benchmark run takes about 2 hours. It would be nice if the ETA would updated more often (say once every second) with a new estimate

[sharkdp]

It would be nice if the ETA would updated more often (say once every second) with a new estimate

This should actually be the case? Or would you expect any update on the initial timing run?

[sharkdp]

Actually, something seems wrong with the ETA estimates at the moment. If I run e.g.

hyperfine --runs=2 "sleep 5"

The ETA shows 5 seconds after the initial timing run (which is expected). But then the ETA time starts increasing... instead of decreasing.

I'm pretty sure this was working correctly in the past. This seems like a regression.

(note that ETA estimates are done by the "indicatif" dependency, not by hyperfine directly. but maybe we are using it incorrectly.)

[martinus]

I did a git bisect, it seems in 2cd2773 the strange runaway time was introduced, most likely when indicatif was upgraded from 0.17.4 to 0.17.5

Before that, it was working correctly but doesn't update as often as I'd like to.

hyperfine "sleep 5"

only shows a new ETA after a run is complete, so after the first run it jumps to 5 seconds, and stays at 5 seconds until it has finished. That's ok for 5 seconds, but when the benchmark takes 2 hours that means the ETA will be stuck at 2 hours until it's actually finished

[sharkdp]

only shows a new ETA after a run is complete, so after the first run it jumps to 5 seconds, and stays at 5 seconds until it has finished. That's ok for 5 seconds, but when the benchmark takes 2 hours that means the ETA will be stuck at 2 hours until it's actually finished

Yeah, this is not great. I agree. I would expect indicatif to handle this. We are updating the ETA estimate every few milliseconds.

I did a git bisect, it seems in 2cd2773 the strange runaway time was introduced, most likely when indicatif was upgraded from 0.17.4 to 0.17.5

Thanks! I can actually reproduce the bug using a very simple program (indicatif 0.17.6):

use std::thread;
use std::time::Duration;

use indicatif::{ProgressBar, ProgressStyle};

fn main() {
    let pb = ProgressBar::new(2);
    pb.enable_steady_tick(Duration::from_millis(100));
    pb.set_style(
        ProgressStyle::default_spinner()
            .template("{wide_bar} ETA {eta_precise}")
            .unwrap(),
    );
    for _ in 0..2 {
        thread::sleep(Duration::from_millis(5000));
        pb.inc(1);
    }
    pb.finish_and_clear();
}

This shows the same problem (ETA jumps to 5 sec — which is fine, then increases). I will report this upstream, unless you want to do that?

[martinus]

I can't actually program in rust, I'd prefer if you can do that :)

[sharkdp]

console-rs/indicatif#580