Rhizome parallelization #68
Conversation
The purpose is to execute rhizome actions on a thread different from that handling routing and MDP stuff, so that a long rhizome action does not jam the communications. This first step adds a "struct fdqueue" type with all the fields specific to an fdqueue. Two instances are created: main_fdqueue and rhizome_fdqueue. All public fdqueue functions now take an fdqueue argument. They are thread-safe. fd_poll() uses synchronization privimives for waiting until the next event, so that the wait can be interrupted for executing a new action earlier.
When poll() is waiting for events on watched file descriptors, it should not block any thread wanting to (un)schedule an alarm. But the mutex cannot be released while executing poll(), because the watched file descriptors must not be modified (or closed!) by another thread. Therefore, poll() now waits for one more file descriptor, a pipe, to become ready. Before acquiring a mutex, fdqueue functions write 1 byte to the pipe. Once the mutex is acquired, they read 1 byte. Thus, poll() is guaranteed to be non-blocking when a thread waits for the mutex. If the pipe is ready for I/O, then fd_poll() release the mutex for 1ms.
When running tests, if no trace is logged before fdqueues_init(), then fd 0 will not be token for logfile. This commit adds a dummy trace.
Implement rhizome thread function (which consumes its own fdqueue). Add a util function schedules a new (allocated) alarm from only 3 parameters: - the function to execute; - its parameter (void *); - the fdqueue to use. Of course, the parameter of the function must not be stored on the call stack (else it will not exist anymore when the other thread will use it).
As performance_timing.c was designed to be monothreaded, its global variables have been moved to the fdqueue structure, and the fdqueue parameter is added to all its functions.
Multithreading is needed only in overlay mode. Add a flag enabling use of the Rhizome fdqueue. If disabled, always use the main fdqueue.
Enable multithreading and start a Rhizome thread in overlay mode.
Define overlay_mdp_dispatch_alarm, a wrapper for overlay_mdp_dispatch to be scheduled.
Define overlay_payload_enqueue_alarm, a wrapper for overlay_payload_enqueue to be scheduled.
Define overlay_rhizome_saw_advertisements_alarm, a wrapper for overlay_rhizome_saw_advertisements to be scheduled.
Schedule rhizome_saw_advertisements_alarm in process_incoming_frame.
Schedule overlay_mdp_dispatch_alarm in rhizome_fetch_mdp_requestblocks.
Schedule overlay_mdp_dispatch_alarm in overlay_rhizome_saw_advertisements.
Schedule overlay_payload_enqueue_alarm in overlay_rhizome_advertise.
Schedule overlay_mdp_dispatch_alarm in rhizome_sync_request.
Schedule overlay_mdp_dispatch_alarm in rhizome_sync_send_requests.
Schedule overlay_mdp_dispatch in sync_send_response.
Define rhizome_received_content_alarm, a wrapper for rhizome_received_content to be scheduled.
Schedule rhizome_receive_content_alarm in overlay_mdp_service_rhizome_response.
Define rhizome_mdp_send_block_alarm, a wrapper for rhizome_mdp_send_block to be scheduled.
Schedule rhizome_mdp_send_block in overlay_mdp_service_rhizomerequest.
Schedule overlay_mdp_dispatch_alarm in rhizome_mdp_send_block.
Define rhizome_advertise_manifest_alarm, a wrapper for rhizome_advertise_manifest to be scheduled.
Code refactor paving the way to schedule retrieve+advertise manifest.
Define rhizome_retrieve_and_advertise_manifest_alarm, a wrapper for rhizome_retrieve_and_advertise_manifest to be scheduled.
Schedule rhizome_retrieve_and_advertise_manifest_alarm in overlay_mdp_service_manifest_response.
Define rhizome_sync_send_requests_alarm, a wrapper for rhizome_sync_send_requests to be scheduled.
Schedule rhizome_sync_send_requests_alarm in overlay_mdp_service_rhizome_sync.
I only added the glue to make the current code multithreaded, without changing the logic.
Paul also said:
I read somewhere that you wanted to avoid threads, and use only monothreaded processes. The reasons are unclear to me. Is avoiding residual inter-thread locks the main reason? Could you explain? |
|
On Tue, Aug 13, 2013 at 7:27 PM, ®om [email protected] wrote:
I've split overlay_mdp_dispatch such that internal services that only sent Instead of allocating a new alarm per frame, we can probably build some On that point, should we rename the "rhizome" thread to the "background" I agree, the missing part of the work is to rewrite some algorithms, when
|
Ah? Where are these variables?
Which services do send packets to "local" mdp clients? Which are these "local" mdp clients?
Maybe. But, that way, it could only apply for passing one frame from one thread to another. My idea was to pass "runnables" (a generic function+argument to post whatever action you want). In practice, the alarms I scheduled do not always post frames (see parallel.h
I don't know if the overrhead of these The way I've implemented it uses the same mechanism both for main thread and rhizome thread. As a consequence, if rhizome blocks waiting for main thread to be idle, then main thread will also block waiting for rhizome thread to be idle. The situation where the main thread needs to post a runnable on the rhizome thread occurs (1 2 3), but maybe it can be avoided…
I've considered this background thread to be rhizome-specific: another service would have its own thread too… Although, even a single service could have several threads.
I think it is a good idea. Ideally, I think Rhizome could simply work as any other service on top of MDP: it would open an MDP socket on a predefined port and exchange with other peers, without any lower-level knowledge and, above all, without being referenced by any lower-level code. This would remove the need of "internal services" hack: each service would use its own port dynamically (like with TCP or UDP). In that case, Rhizome would create its own thread to handle its stuff separately, without impacting overlay* code. What do you think? |
quixotique
force-pushed
the
development
branch
from
62ce14f
to
463e276
Compare
I recently found that mdp packets and Rhizome stuff were executed on the same thread. Consequently, audio communications were broken during Rhizome synchronization.
I propose some code making all the Rhizome stuff executed in a separate POSIX thread (only in overlay mode, not necessary elsewhere).
With this code, I successfully exchange Rhizome bundles without impacting audio communications.
Implementation
Here are the principles of my implementation (for more information, read the messages of the firsts commits).
The main idea is to use two fdqueue instances: one "main" and one "rhizome". In overlay mode, a new thread is started, which calls
fd_poll()on the rhizome fdqueue.An alarm (
sched_ent) has now a fieldfdqueue, specifying on which fdqueue the alarm must be scheduled/watched. Thus, the main thread can schedule an alarm to be executed on the rhizome thread (and vice-versa).Of course, the alarm data (its context) cannot be stack-stored (there is one stack per thread) and these alarms must be allocated dynamically, which implies quite a lot of changes. Commits with message beggining by "Alarm" create a wrapper to be scheduled on an fdqueue. Commits with message beggining by "Schedule" allocate parameters (if needed) and schedule an alarm to be executed on the other thread.
Tests
All tests pass, except
FileTransferUnreliableBigMDP.This test sometimes fails even without my changes. If I change the filesize from 1MB to 500kB (in
setup_bugfile_common()intests/rhizomeprotocol), it pass.