Only send flushes when Downstairs is idle; send Barrier otherwise

[mkeeter]

This PR removes automatic flushes, per RFD 518. Instead, the new Barrier operation is sent. If the system is idle for a particular amount of time, we send a final Flush to put everything into a known state.

When the Upstairs retires jobs after a barrier operation, the system as a whole becomes ineligible for replay. This state determines whether the new Downstairs reconnects through Offline (which does replay) or Faulted (which does live-repair instead).

Removing automatic flushes is a noticeable performance improvement:

(tested on the London mini-cluster, with Upstairs and 3x Downstairs on different sleds)

region info:
  block size:      4096 bytes
  blocks / extent: 16384
  extent size:     64 MiB
  extent count:    2048
  total blocks:    33554432
  total size:      128 GiB
  encryption:      yes

[mkeeter]

Rebased to stage on top of #1507, because we want to only send Barrier operations if there's enough bytes / jobs buffered (which requires keeping track!)

[leftwo]

Do we have any way to check that replay is no longer available to this downstairs? I'm not sure if the test has access at this layer to it.

[mkeeter]

The end of this unit test is checking that live-repair works, so I think we're implicitly testing that we don't do replay.

[leftwo]

Ah, I was thinking more that the sending of a barrier operation has also flipped the can_replay in the Downstairs struct. I'm not sure if we can do that though here (or in a test elsewhere), as can_replay might not be exposed like that.

[mkeeter]

Yeah, it's not easy to check from the outside (when we only have the Guest handle). If you really want it, we could probably add it to the downstairs_state helper function.

[leftwo]

I don't think it's worth adding it to the downstairs_state helper function.
I don't see anywhere else that seems like a good place either. Nothing in the upstairs tests cover this case.

[leftwo]

With this, we can probably close #1358 as fixed

[faithanalog]

We should probably tighten up IO_CACHED_MAX_BYTES soon. Right now we can theoretically buffer up to 1GiB of jobs for replay, and that feels like quite a lot to me. We usually won't- we will only get to that point if the guest is doing large amounts of IO, continuously so we never go idle, and the guest never sends a flush. That's unlikely/rare during normal filesystem operations (but will be easier to hit when writing to the raw block device like iodriver does). Because of that, we shouldn't expect more than a few VMs to hit this under normal operation. But in interest in not overcommitting resources, I think that bound should be lower.