Status Update: 2022-05-11

[haixuanTao]

In this discussion, we describe our recent progress and the main topics that we would like to discuss in today's meeting. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[haixuanTao]

Meeting 2022-05-11

Messages Design

Metadata field in messages help us keep track of informations throughout the distributed system, such as:

• Watermark
• Deadline
• Opentelemetry (spans and log)

Exemple with Capnproto

struct Metadata {
  metadataVersion @0 :Text;
  watermark @1 :UInt64; 
  deadline @2 :UInt64;
  openTelemetryContext @3 :Text;
  depth @4 :UInt32; # Depth within the graph of the message 
}


struct Message {
  id @0 :UInt32;
  data @1 :Data; # ByteArray
  metadata @2 :Metadata;
}

[haixuanTao]

Remark:
Metadataversion -> Text -> UInt32
Metadata @2 -> Metadata @1

[haixuanTao]

Future design ideas:

• Make Data Format configurable.

[haixuanTao]

Older Serialization benchmark: https://blog.logrocket.com/rust-serialization-whats-ready-for-production-today/
New kid on the block: protocol: https://github.com/otake84/dlhn

[haixuanTao]

OpenTelemetry with Jaeger

• Cross language telemetry.
• Distributed telemetry for dora-rs, enabling tracing for the whole graph.
• Records spans and logs across language.
• Queryable metrics enabling academic research but also large scale production system.
• Can be integrated to Influxdb, Prometheus, Datadog.

[haixuanTao]

https://github.com/tokio-rs/tracing

[haixuanTao]

Implementation progress

• Improvement on shared libraries C-API.
• Progress on nodes and operators implementation.
• Progress on the data plane and control plane part of the middleware.

[heyong4725]

Zenoh team may update their API further to make it better, here is the discussion: eclipse-zenoh/roadmap#23
Also there are good discussion on Rust async/sync: eclipse-zenoh/roadmap#17
Hope this is useful for our Dora work

[heyong4725]

Also Zenoh Flow is refactoring new deadline implementation: eclipse-zenoh/roadmap#22
@haixuanTao @phil-opp

[heyong4725]

@phil-opp you may want to take a look of this repo: https://github.com/tower-rs/tower/tree/master/tower-service
Tower-service is trying to abstract the request/response comm pattern called service, which is common, and similar to ROS2 service abstraction. It seems that tower use http as transport, but you could potentially use other transport.
It may be part of our middleware abstraction as well, and potentially for essa project as well.