Status Update: 2022-08-03

[haixuanTao]

Status Update ! 😄

[haixuanTao]

Implement Python Node API #54

Description of the API

The Python API will consist of two principal methods: next and send_output. Those methods will be connected to a tokio runtime through channels.

Problem

The Python Node API requires an async threadpool as the communication Layer also known as the Middleware Layer returns an async Future stream.

Solution

I solve this issue by adding a tokio runtime on a separate thread that is connected with two channels. One for sending data and one for receiving data.

Those channels are then exposed synchronously to Python. This should not be cause for concern as channels are really fast.

from dora import Node

node = Node()

(id, value) = node.next()

# or

for id, value in node:
    ...

node.send_output("input_id", data)

[haixuanTao]

Application Reliability ?

• We should think about token for success of messages.

[haixuanTao]

Implement runtime-provided periodic timer input for source operators #51

The periodic timer inputs allows operators to periodically produce values. To use the new input types, operators specify an input that maps to dora/timer/secs/X where X is the number of seconds between ticks. In addition to secs, the runtime also support millis for millisecond intervals.

The new dora timer nodes are visualized in the following way:

flowchart TB
  node_1[\node_1/]
  logger[/logger\]
subgraph python-operator [node]
  python-operator/op[op]
end
subgraph ___dora___ [dora]
  subgraph ___timer_timer___ [timer]
    dora/timer/secs/5[\secs/5/]
    dora/timer/millis/100[\millis/100/]
  end
end 
  python-operator/op -- counter as test --> logger
  dora/timer/secs/5 -- tick --> logger
  dora/timer/millis/100 -- dora_time --> python-operator/op
  node_1 -- data --> python-operator/op

Loading

[haixuanTao]

Timer can later be used as a heartbeat as well.

[haixuanTao]

dora-drives status update

• Rust code that was bridging dora and dora-drives has been removed thanks to the new python node API and the single runtime - single operator API fix
• Discussion with Sebastien Crozet, lead us to discuss about the physics engine of rapier which is parry which implement many of the physical/graphical functionality we need! However, parry is not very ready to be to dora has many proper API is still lacking. It can be interesting to follow its development and come back to it later on.

[haixuanTao]

Other changes

• Allow operators to stop themselves
• Moving folders in correspondance with #41
• Add support for single operator nodes in YAML dataflow specification