net: introduce input_buffer_factory concept.

[rzarzynski]

https://docs.google.com/document/d/18T0LdTX24GZS2iHvqUm8779ka1WTS78MiThymaim6ys/edit?usp=sharing

Signed-off-by: Radoslaw Zarzynski [email protected]

[tchaikov]

aside from the nit. lgtm after squashing the related commits.

[cyx1231st]

I'm not 100% sure that 1 syscall per message is efficient enough for small reads... maybe consider to support prefetch across the message boundary (i.e. less syscall but some more memory copy introduced).

[cyx1231st]

I understand this is not the final implementation, just for the record:

For input_buffer_factory of v2 protocol, return seastar::make_temporary_buffer<char>(allocator, get_current_msg_size() + FRAME_PLAIN_EPILOGUE_SIZE + FRAME_PREAMBLE_SIZE + 41) would be too simple, because it is the 3rd segment (DATA) which needs to be page-aligned.

[rzarzynski]

Yes, you're entirely correct. We would need to allocate a bigger buffer and ::trim_front it accordingly.

[cyx1231st]

Curious about how to make use of the unused tail of the rx buffer, feed back to the data_souce later?

[rzarzynski]

Yup, that's the idea. For huge chunks even the POSIX stack tends to fragment the output. This is likely because of the readable(fd) triggering just when something becomes available to read:

future<size_t>
reactor::read_some(pollable_fd_state& fd, void* buffer, size_t len) {
    return readable(fd).then([this, &fd, buffer, len] () mutable {
        auto r = fd.fd.read(buffer, len);
        // ...
}

[rzarzynski]

I'm not 100% sure that 1 syscall per message is efficient enough for small reads... maybe consider to support prefetch across the message boundary (i.e. less syscall but some more memory copy introduced).

Yeah, the current write testing shows the threshold is somewhere around 4-8 KB. Still, it depends on many things including kernel version/hardware/* (security-driven workarounds blurring the difference between context- and mode-switch on one-side; io_uring on the contrary one).

I believe that ultimately we'll be doing some prefetching even for the sake of reads (very small messages).