look into using Rayon for the IO thread pool. Can we associate an IoUring instance with each thread? Can we publish tasks to other threads?

[JackKelly]

I'll use this "issue" to document my research into using Rayon for the IO threadpool.

Why use Rayon's threadpool for the IO threadpool?

In #61, I'm talking myself into building my own work-stealing threadpool (using crossbeam::deque). But this hand-built threadpool is starting to sound a lot like Rayon's threadpool! So maybe I can just use Rayon's threadpool? (And hence save myself a bunch of work. And, more importantly, end up with code which is more robust & easier to maintain!)

Related:

• Multiple threads (each with a uring) & crossbeam::deque & a pipeline of I/O steps #61
• Drop ObjectStore & async/await. Use Channels instead. Focus entirely (for now) on io_uring for local file storage. #93

[JackKelly]

Can we associate exactly one IoUring instance with each Rayon thread?

This StackOverflow answer suggests that, yes, we can "use the scoped ThreadLocal from the thread_local crate" to "ensure that you only use as many objects as there are threads".

struct ThreadLocal<T: Send> and IoUring implements Send.

Or, actually, maybe for_each_init is better. This almost works! Except I can't see how to drop in_tx2 (and, if we can't drop in_tx2 then the in_rx.into_iter() will never end!)

[JackKelly]

OK, I've figured out how to spawn tasks into the current threadpool. Here's some code which I think provides the full framework for making this work.

Some things that would need to be added to that code:

• We'd have a custom struct, probably exactly like the Worker struct we currently have.
• Within our Rayon closure, we'd use thread_local_storage.get_or(|| Worker::new()) to create exactly one Worker per thread.
• As in the current code, our Rayon::pool.spawn(|| {in_rx.into_iter().par_bridge().for_each(|op| { closure would start by looping through that uring's completion queue. Then submit any strictly local (non-stealable) ops to its uring. Then, if the uring has space, submit the new op (but don't block here. If the uring doesn't have space for this op then either add this op to the uring's strictly local queue; or spawn it to the threadpool). Then check the completion queue again.
• Finally, after the for_each statement, we'd pool.broadcast to all threads, to loop through its completion queue.

[JackKelly]

I was worrying about what would happen if we get to the end of the tasks in the queue, but we don't hang up the sender, so the last few CQEs will not be reaped.

I think the solution is that the user always submits a vector of ops. Then, when we get to the end of the vec, we broadcast to all threads to finish their CQ. This is the only time when threads should wait on the CQ.