MLA - Middleware Layer Abstraction

[heyong4725]

Middleware (communication) layer abstraction (MLA)

We need to implement MLA as a separate crate to provides a middleware abstraction layer that enables scalable, high performance communications for inter async tasks, intra-process (OS threads), interprocess communication on a single computer node or between different nodes in a computer network. MLA needs to support different communication patterns:

• publish-subscribe push / push pattern - the published message is pushed to subscribers
• publish-subscribe push / pull pattern - the published message is write to storage and later pulled by subscribers
• Request / reply pattern
• Point-to-point pattern
• Client / Server pattern

The MLA needs to abstract following details:

• inter-async tasks (e.g., tokio channels), intraprocess (OS threads, e.g., shared memory), interprocess and inter-host / inter-network communication
• different transport layer implementations (shared memory, UDP, TCP)
• builtin support for multiple serialization / deserialization protocols, e.g, capnproto, protobuf, flatbuffers etc
• different language bindings to Rust, Python, C, C++ etc
• telemetry tools for logs, metrics, distributed tracing, live data monitoring (e.g., tap a live data), recording and replay

Rust eco-system has abundant crates to provide underlaying communications, e.g.,:

• tokio / crossbeam provides different types of channels serving different purpose: mpsc, oneshot, broadcast, watch etc
• Tonic provides gRPC services
• Tower provides request/reply service
• Zenoh middleware provides many different pub/sub capabilities

MLA also needs to provide high level APIs:

• publish(topic, value, optional fields):- optional fields may contain senders' identify to help MLA logics to satify above requirements
• subscriber(topic, optional fields)-> future streams
• put(key, value, optional fields)
• get(key, optional fields) -> value
• send(key, msg, optional fields)
• recv(key, optional fields)->value

[phil-opp]

A few unordered thoughts on this:

• For communication across different machines and networks, we need some kind of routing solution. This could be a centralized broker or some kind of decentralized solution as for example used in zenoh. The challenge is that all patterns should use the same routing approach, otherwise some nodes might be reachable using the publish/subscribe API, but not through point-to-point.

  • This requirement makes it much more difficult to support the wide range of communication patterns mentioned above since we cannot easily "pick-and-mix" . So we can either only support the communication systems that implement all the required patterns (not even zenoh does), or we need to manually synchronize the routing state across different frameworks somehow.

• The main advantage of (mpsc) channels as a form of communication within a process is that it enables a higher-level API:

  • Both halves of the channel are within the same process, so each half can trust the other half (thanks to the compiler).
  • This trust makes it possible to send raw data without serialization or verification steps.
  • Since both ends are compiled by the same compiler, the sent data does not even need a stable and specified layout in memory, so almost all types can be used.
  • The compiler can verify that both ends use the same type, so no runtime errors can occur because of mismatched types.
  • No data corruption can happen and no data can be lost since it is never sent over the network.

  By migrating from this API to a more general e.g. publish/subscribe one, we now need to deal with a lot more error cases. The necessary serialization/verification also leads to a considerable performance overhead and drastly reduces the types that can be used with this channel. E.g. sending thread join handles would not be possible anymore, since they might cross a process or even machine boundary.

  So I don't think that we can realisticly replace in-process mpsc channels with a more general middleware.

[heyong4725]

@phil-opp, for your comment#1, definitely we are not looking for "pick-and-mix". My hope is that for each communication patterns, we choose the best middleware provider. For example, for pub/sub pattern, Zenoh will be default since it is by far the best Rust middleware implementation.

[haixuanTao]

When we talk about point-to-point do we strictly need a direct connection. Or, can we just use Zenoh, and create a specific topic and make two points communicate separately on this specific topic ?

Knowing that any client-server communication will require that we initiate a server in addition to the Zenoh session on all nodes.

This server will need additional constraints on IP and Port in order to be seen by all. This can be complex in a k8 environment, for example.

I feel that the best MLA at the moment is just zenoh. If we switch out of zenoh, we can use the tokio ecosystem with a point-to-point approach first.

[heyong4725]

@haixuanTao for point-to-point, I hope it is a direct link, since Zenoh may not be the best efficient way of doing that. We have experienced that performance issue in other project.
For each communication pattern, MLA should transparently select the best underlying technology to implement that, and provide APIs for that.

[heyong4725]

Since Dora needs to support log/recording, replay and monitoring for data steams (e.g., similar to ROSBag) such that developers can offline debug issues (it is critical since online debug for physical robotic physical machine is hard), I would suggest these capabilities be built into MLA. @phil-opp @haixuanTao, let me know what you think.

Similarly message encoding, such as CapnProto (or protobuf, flatbuffer...) etc should also be part of MLA.

[haixuanTao]

So just to make a synthetic view of the MLA, I have made a mental tree of the planned MLA:

MLA/
├─ Encoding Layer/
│  ├─ Capnproto/
│  ├─ (Future) Protobuf/
├─ PubSub Communication Layer/
│  ├─ Zenoh
│  ├─ Iceoryx
├─ Request/Reply Communication Layer/
│  ├─ ...
├─ Extension/
│  ├─ Replay/
│  ├─ Opentelemetry?/

Let me know if I misunderstood something. I think that this should be accessible. I feel like if we include Replay in the MLA, most extensions are going to be tied to the MLA and not dora itself.

Putting the extension on the MLA level make it usable by more packages, so it makes sense.

[heyong4725]

I think extension should be separate from MLA, since we would like open telemetry also be able to apply to other part of dora code as well. Same is true for replay. We may need to revisit our folder structure discussion

[heyong4725]

Since MLA will need to support Service (request/reply) as in ROS2 API, this Rust open source project may be a reference: https://github.com/cloudwego/motore
This crate may be a Service provider for Dora

Also CloudWeGo https://www.cloudwego.io/ also provides other MLA abstractions as well, such as Rust RPC framework

[haixuanTao]

I see that motore use Generic associated types (GAT). I wonder in which extent we should be planning on doing the same.

Zenoh has recently published a new version 0.6 and some lifetime parameters have changed for publishers and subscribers. I'm not sure if we can keep the current CommunicationLayer trait intact.

I feel like GAT could help us in making sure our MLA is future-proof, but not sure if the added complexity is worth it.

[heyong4725]

What I am thinking is we can re-use cloudwego crates as our middleware providers for other comm patterns, such as RPC, Service etc. These are common patterns. If we can package /wrapper them into Dora SDK, that would be perfect. We don't need to re-invite the wheels, just as for pub/sub, we are using Zenoh, Iceoryx for pub/sub MLA provider

[phil-opp]

The motore crate currently only work on nightly Rust since it uses GAT and TAIT (type alias impl trait), which are both still unstable. GAT will become stable in 1.65, which is scheduled for 2022-11-03. TAIT is also close to be stabilized (see rust-lang/rust#63063 (comment)), but it will take at least a few months until it reaches the stable channel.

So if we decide to use motore, that part of our code will also be nightly-only until then.

[heyong4725]

Currently, we are not using those comm patterns, we can wait until GAT and TAIT are stable.