Status Update: 2022-08-17

[phil-opp]

In this discussion, we describe our recent progress and the main topics that we would like to discuss. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[phil-opp]

New Documentation Book

We started a documentation book on dora, published at https://dora-rs.github.io/dora/. The source code of the book is available in the docs subfolder.

[phil-opp]

New Examples Dataflows

We added small example dataflows to showcase the different languages that dora supports:

• Rust-dataflow
• C-dataflow
• Python-dataflow

Dora does not provide native C++ support yet, but it is still possible to create C++ operators and nodes based on either C API or the Rust API (in combination the cxx crate). The c++-dataflow example shows both approaches.

[phil-opp]

Performance improvement of dora-drives

• Documentation
• Push docker image to help with installation.
• Compression of image before sending it
  • Using jpeg format we can reduce by up to 20x the size of an image. Making the messages also more standard to image format.
• Replace efficient detection with yolov5 for better performance.

[phil-opp]

Next Steps

• More docs
  • Add more documentation within the code: docstring, etc...
  • Add more information about how to describe a yaml file
  • requirement information
• Create a basic command line interface for controlling a running dataflow
  • stop command to stop a running dataflow
  • communicate with control plane via gRPC
• Release v0.1 on crates.io
  • As soon as we have a MVP implementation with proper docs
  • What things are missing?
• Use MPSC channels for communication within a runtime node (i.e., operator to operator)
  • to improve performance
• Use CaptnProto format for all messages
  • To allow use of extension libraries (e.g. tracing)
• Typed inputs and outputs
  • Specify the types of inputs and outputs in the dataflow.yml
    • primitive types (e.g., int or string) are directly supported
    • other types can be defined by referencing a CaptnProto schema
    • type annotation is optional -> defaults to byte array
  • When loading a dataflow, the coordinator verifies that the types match for all input/output mappings
  • The input API for operators and nodes should pass the type information
    • For example in form of an enum for the Rust API
    • Operators could implement optional typed on_input functions (e.g., on_string_input)
  • Verify that operator/node outputs match defined types
    • ensures that the nodes/operators behave as defined
  • Is it possible to verify input/output types already at load time somehow?

[heyong4725]

Regarding to Dora distributed architecture discussion, we talked about Dora daemon process, which may facilitate with coordinator remotely. Please take a look of this Tauri’s process model, this Dora daemon may be similar to Core process in Tauri’s case. https://tauri.app/v1/guides/architecture/process-model, I also I can imagine that this Dora daemon may facilitate share memory among processes, similar to another project: https://github.com/eclipse-iceoryx

[heyong4725]

@phil-opp @haixuanTao https://iceoryx.io/v2.0.2/examples/iceperf/ iceoryx did have a very impressive latency with various payload size. It is worth studying

[phil-opp]

Future Ideas

Longer-term ideas that we have been brainstorming a bit:

• Associate each message with an HLC timestamp?
  • we could use the uhlc crate
  • would allow logical ordering of all events in the system, preserving the "happens before" relation
  • drawback: a bit of additional overhead
• Heartbeat protocol to detect crashed nodes?
  • The coordinator could restart nodes that appear to be crashed.
  • Challenges
    • Custom nodes define their own main function, so they would probably need to send the heartbeat messages manually.
      • Additional "tick" timer input that nodes need to reply to
    • The "crashed" node might come back again. How do we prevent collisions with the newly started node?
      • assign a random UUID to each node at startup -> control plane sends new UUID when node restarts