Replies: 0 comments 5 replies
-
|
Is there a browser-based capnproto implementation that could be used to dynamically read schemas and deserialize messages? |
Beta Was this translation helpful? Give feedback.
-
|
The issue title mentions |
Beta Was this translation helpful? Give feedback.
-
I'm not sure if this currently exists, but it doesn't seem far fetched to load native code which achieves this as a WASM module. See for example which does dynamic schema parsing into plot juggler: commaai/PlotJuggler@0050165#diff-24b0c52ead32bdc92c4fcf38f0b5cd875e90e8456fd9dbffced93f9992780541 Converting into a foxglove-friendly schema seems to be a separate issue: https://foxglove.dev/docs/studio/messages/introduction So it seems that supporting this has two parts:
Perhaps as a less portable implementation, the plugin can directly convert raw binary data into a msg type of choice in the foxglove schema. |
Beta Was this translation helpful? Give feedback.
-
To be more to the point, yes: https://www.npmjs.com/package/capnp-ts |
Beta Was this translation helpful? Give feedback.
-
|
The project has a prototype but has been abandoned as of now. See: #860 (comment) |
Beta Was this translation helpful? Give feedback.
-
https://capnproto.org/
https://capnproto.org/images/infinity-times-faster.png
Introduction
Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap’n Proto is INFINITY TIMES faster than Protocol Buffers.
This benchmark is, of course, unfair. It is only measuring the time to encode and decode a message in memory. Cap’n Proto gets a perfect score because there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!
But doesn’t that mean the encoding is platform-specific?
NO! The encoding is defined byte-for-byte independent of any platform. However, it is designed to be efficiently manipulated on common modern CPUs. Data is arranged like a compiler would arrange a struct – with fixed widths, fixed offsets, and proper alignment. Variable-sized elements are embedded as pointers. Pointers are offset-based rather than absolute so that messages are position-independent. Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.
Doesn’t that make backwards-compatibility hard?
Not at all! New fields are always added to the end of a struct (or replace padding space), so existing field positions are unchanged. The recipient simply needs to do a bounds check when reading each field. Fields are numbered in the order in which they were added, so Cap’n Proto always knows how to arrange them for backwards-compatibility.
Won’t fixed-width integers, unset optional fields, and padding waste space on the wire?
Yes. However, since all these extra bytes are zeros, when bandwidth matters, we can apply an extremely fast Cap’n-Proto-specific compression scheme to remove them. Cap’n Proto calls this “packing” the message; it achieves similar (better, even) message sizes to protobuf encoding, and it’s still faster.
When bandwidth really matters, you should apply general-purpose compression, like zlib or LZ4, regardless of your encoding format.
Isn’t this all horribly insecure?
No no no! To be clear, we’re NOT just casting a buffer pointer to a struct pointer and calling it a day.
Cap’n Proto generates classes with accessor methods that you use to traverse the message. These accessors validate pointers before following them. If a pointer is invalid (e.g. out-of-bounds), the library can throw an exception or simply replace the value with a default / empty object (your choice).
Thus, Cap’n Proto checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content.
Cap’n Proto was built to be used in Sandstorm.io, and is now heavily used in Cloudflare Workers, two environments where security is a major concern. Cap’n Proto has undergone fuzzing and expert security review. Our response to security issues was once described by security guru Ben Laurie as “the most awesome response I’ve ever had.” (Please report all security issues to [email protected].)
Beta Was this translation helpful? Give feedback.
All reactions