pipeline might not stop

[robkooper]

If we start more jobs than nodes available (for example 5 icy-resin and 5 golden-muscat) we can get into a situation where some jobs are in pending. Once all work is finished and nodes become available the pending jobs get started. These newly started jobs will run, but not finish.

Possible problem rabbitmq worker never sets time correctly and thus never detects that no new work was started in the past x minutes.

check code at https://github.com/DARPA-CRITICALMAAS/uiuc-pipeline/blob/main/pipeline.py#L494