Python Dataflow not gracefully stopping

[haixuanTao]

After trying to add a new feature in Python, I have notived that python dataflow does not gracefully end anymore.

Screencast.from.2024-08-13.07-05-38.webm

To reproduce

cd examples/python-dataflow
dora build dataflow.yml
dora start dataflow.yml
# Ctrl - C

[phil-opp]

I just tried it a few times with the main branch but I couldn't reproduce this issue.

But I noticed that the example is now combining pip-installed nodes and with the dora-cli in git since the node hub changes. This will lead to compatibiliy issues whenever we change the message format or do some other breaking change. The version check doesn't catch this because we only bump the version number on release. So there will be some deserialization issues or weird behavior in the future and we will probably experience CI failures because of this.

[haixuanTao]

pip always reinstall the latest git version of the nodes used as we're passing the source code in the graph.

Also, as we're preinstalling dora-rs within run.rs, it should also always follow the latest dora-rs pip version.

I'll try to make another example maybe easier to reproduce.

[haixuanTao]

FYI, I have changed the CI to use the cli and is now blocked by this: https://github.com/dora-rs/dora/actions/runs/11084418707/job/30799710937#step:9:1

it happens within the CLI but not when using the example runner

[haixuanTao]

Actually now that I think about it, the daemon stop gracefully because it doesn't react to a stop command.