Dynamic dataflows can not be stopped on Linux and MacOS for 0.3.6

[Hennzau]

Describe the bug

Issue #575 has resurfaced, even though it was previously resolved by #583.

When I launch a dataflow where a node is dynamic, I can't stop it; neither the coordinator nor the daemon crashes (whether I use CTRL+C if attached or dora stop if detached). The only way to stop it is by running dora destroy followed by dora up.

This issue also occurs when running the full dataflow. Even after starting it and running all the nodes, the dataflow doesn't terminate as expected.

Steps to Reproduce

1. I was working on some courses, so I created a directory, cd into this directory.
2. Start the Dora daemon: dora up.
3. Start a new dataflow: dora start dataflow.yaml.
   • You can also run the two dynamic nodes if you wish. Everything functions correctly, but the dataflow doesn't stop at the end.
4. Attempt to stop it using CTRL+C.

At this step, it freezes until I do the following:
5. Destroy the dataflow: dora destroy.

Environment (please complete the following information):

• System info: MacOS and Linux
• Dora version: tag v0.3.6

[haixuanTao]

I have seen this get back as well!

Super annoying.

I'm going to catch up on certain things for lerobot, but I agree that this should be fixed.

FYI, philip is on vacation until the 21st I think.

[haixuanTao]

The key issue is that we have to back propagate drop event token when some node finishes to use an input so that the shared memory can be freed.

However sometimes it seems that the drop event token is blocked with GIL and the garbage collector, making the whole dataflow unable to sstop gracefully.

[haixuanTao]

If you add enough time in grace duration you should see the timeout.

[Hennzau]

all right, I'll take a look after #650

[Hennzau]

probably linked to #625