Comparison with deku

[RReverser]

Couldn't help but notice a lot of similarities, at least in terms of goals if not exact design, with a project I saw & used in the past - deku by @sharksforarms.

The more the merrier, but I was wondering if there's a comparison between the two somewhere and description of what binrw is trying to do differently?

[v1gnesh]

I approve of this question @RReverser :)
Had the same question and wish to see a nice write-up of how they compare & to showcase the unique capabilities of each.

[RReverser]

One I can think of off the top of my head is that deku supports (focuses on?) bit-level read/writes while binrw seems to operate at byte level only, but beyond that they feel pretty similar, so would love to learn more about differences from someone more knowledgeable.

[csnover]

It is difficult for me to provide an accurate comparison because I (1) am an obviously biased source, and (2) have not really used deku since I found its fundamental design choice to use slices instead of streams to be a fatal error. Hopefully others will be able to provide corrections and additions so this can be a more fair comparison.

Feature	binrw	deku
Operates on	Any std::io (or binrw::io in no_std mode) stream	Slices only
Primitive level	Byte-level (can use bitflags or modular-bitfield crates for byte-aligned bit values)	Bit-level
Attribute syntax	Normal Rust code	Code in strings
Error handling	Transparent forwarding of any error	Only Deku errors
Runtime error reporting	Full backtrace with stream positions	A string
Compile-time error output	Points to source of error[0]	Sometimes points to the #[derive]
Data access	Random-access (if supported by underlying stream)	Sequential-only
Parser debugging	#[br(dbg)] attribute	Nothing?
Result value	The requested object	A tuple containing the object plus the slice of the rest of the data that must be retained for future parsing
Named arguments	Yes	No
Fallible conversions	Yes	No
Whole object mapping	Yes	No
Accidental machine dependency	Impossible (requires explicit opt-in)	Yes, both endianness and pointer size
Dependency size	~345–770KiB	~1.4–2.1MiB
Magic	Variable-length	Fixed-length
Attributes count	41	29
Collection reading	Any FromIterator	Only Vec, HashMap, HashSet
Maximum tuple size	32	11
Helper count	15: file indirection, per-item collection arguments, collect until eof, collect until terminator, punctuated, string	0
Test coverage	96%	73%
GH popularity contest	322 (binrw) + 1438 (binread)	247

There are probably many other subtle differences, and some places where Deku probably does things better (it probably has better performance since it doesn’t seem to do much state tracking for error reporting, and it operates directly on memory slices instead of having an indirection of the stream), but a substantial amount of work has gone into trying to make binrw a best-in-class library and so I think the comparison reflects that.

[0] If binrw ever uselessly points to its derive when an error occurs, it is a bug that should be reported :-).

[RReverser]

Interesting, thanks for the details! Most of those make sense to me and binrw choices do look compelling.

Few clarifications (from my limited knowledge of binrw):

• "Compile-time error output Points to source of error[0]" - not sure what this means, do you mean points to Rust field or ..? Would love to see an example.
• "Data access" - not sure if it's fair to say that deku is sequential-only since, as you mentioned earlier, it operates on slice inputs. It allows you to e.g. parse part of the input as a subslice (zero-copy), and then come back to it and do further processing, so I'd say both allow random data access here, it's just that the API details of how to do that differ.
• One thing I'm instantly missing from my first glance in binrw and isn't reflected in the comparison table above is usage of custom contexts.
  With deku I'm able to define completely custom read/write routines for all the built-in types when used with my custom contexts, which helps a lot for binary formats that e.g. encode [iu]{8, 16, 32, ...} as LEB128, whereas binrw allows to customise only Endianness as an option, so my only other option would be to wrap each and every of built-in types into newtypes, which is... uncomfortable.

[csnover]

• "Compile-time error output Points to source of error[0]" - not sure what this means, do you mean points to Rust field or ..? Would love to see an example.

Sure. So, in this deku test output, the error is pointing at the derive. If you look at the corresponding source the error is actually in a deku attribute somewhere else. The same kind of undefined variable error in binrw points at the actual source of the error.

• "Data access" - not sure if it's fair to say that deku is sequential-only since, as you mentioned earlier, it operates on slice inputs. It allows you to e.g. parse part of the input as a subslice (zero-copy), and then come back to it and do further processing, so I'd say both allow random data access here, it's just that the API details of how to do that differ.

Sure, this is a fair criticism. What I meant is that deku returns the unparsed part of the input slice in the result to know where to start parsing the next thing, and so files that have offset tables after the data (e.g. zip central directory, mp4 moov atom) cannot seek backwards to those offsets from the input. In contrast, since binrw uses seekable streams, there is no design limitation on backward seeking. If you have a better way to describe this I am happy to update my silly table :-)

• One thing I'm instantly missing from my first glance in binrw and isn't reflected in the comparison table above is usage of custom contexts.

With deku I'm able to define completely custom read/write routines for all the built-in types when used with my custom contexts, which helps a lot for binary formats that e.g. encode [iu]{8, 16, 32, ...} as LEB128, whereas binrw allows to customise only Endianness as an option, so my only other option would be to wrap each and every of built-in types into newtypes, which is... uncomfortable.

You can do whatever using parse_with or map/try_map so you do not need newtypes to do an arbitrary parse. There are also helper attributes to make it easy to write custom functions like this (#[binrw::parser] and #[binrw::writer]). There are ideas about how to improve the UX of arbitrary parsers more, see #98 and #172.

As far as I could tell from the documentation, the use of a context in deku seems equivalent to import/args in binrw, but I would be happy to learn more about that if it is substantially different, as stealing good ideas is always nice :-).

[RReverser]

Sure. So, in this deku test output, the error is pointing at the derive. If you look at the corresponding source the error is actually in a deku attribute somewhere else. The same kind of undefined variable error in binrw points at the actual source of the error.

Oh, looks great!

What I meant is that deku returns the unparsed part of the input slice in the result to know where to start parsing the next thing, and so files that have offset tables after the data (e.g. zip central directory, mp4 moov atom) cannot seek backwards to those offsets from the input. In contrast, since binrw uses seekable streams, there is no design limitation on backward seeking. If you have a better way to describe this I am happy to update my silly table :-)

Yeah not sure if there's a better way to describe it; as I said, I think this one seems to be possible in both although in quite different ways, so personally I'd probably just not list it as a difference.

You can do whatever using parse_with or map/try_map so you do not need newtypes to do an arbitrary parse. There are also helper attributes to make it easy to write custom functions like this (#[binrw::parser] and #[binrw::writer]). There are ideas about how to improve the UX of arbitrary parsers more, see #98 and #172.

As far as I could tell from the documentation, the use of a context in deku seems equivalent to import/args in binrw, but I would be happy to learn more about that if it is substantially different, as stealing good ideas is always nice :-).

I'll definitely need to dig into binrw more as I haven't even seen those attributes yet, but from a first glance, yeah, they seem similar. One - but important in context of usecases like above - difference here is that the arguments are a generic param, whereas in binrw they're an associated type.

This means - as far as I understand - that in binrw I can't define custom implementation of those built-in types like f64 / u32 / bool / etc. as they're all already implemented with Args = ().

If Args was a generic param like in deku, then I could have complete separate set of implementations that could be shared across dozens of structs & enums with ease.

In general, this issue feels similar to endianness, which binrw gets right - not having any default - in that most binary formats have a pretty strong opinion on how integers and other core types are represented, but it's not limited just to endianness, but also to things like fixed-length vs variable-length encodings, what length is usize, how to encode Vecs of elements, and so on.

The ideal end state here seems to be some crossover between binrw strengths (not having defaults) and deku strengths (arbitrary parametrisation) that perhaps looks like with attribute from #98, but where current built-ins are also moved to a module/trait/something and become just yet another possible opt-in rather than assumed whenever user forgets to put an attribute on their type.

That would make much harder to accidentally use a type representation that is not compatible with the given binary format.

[v1gnesh]

Probably unrelated, but would like to know if this will be helpful to either of the crates - rust-lang/rust#89517

trait std::os::unix::fs::FileExt {
    // Existing methods: `read_at`, `read_exact_at`, `write_at`, `write_all_at`

    fn read_vectored_at(&self, bufs: &mut [io::IoSliceMut<'_>], offset: u64) -> io::Result<usize>;
    fn write_vectored_at(&self, bufs: &[io::IoSlice<'_>], offset: u64) -> io::Result<usize>;
}

[sempervictus]

It is difficult for me to provide an accurate comparison because I (1) am an obviously biased source, and (2) have not really used deku since I found its fundamental design choice to use slices instead of streams to be a fatal error. Hopefully others will be able to provide corrections and additions so this can be a more fair comparison.

Feature binrw deku
Operates on Any std::io (or binrw::io in no_std mode) stream Slices only
Primitive level Byte-level (can use bitflags or modular-bitfield crates for byte-aligned bit values) Bit-level
Attribute syntax Normal Rust code Code in strings
Error handling Transparent forwarding of any error Only Deku errors
Runtime error reporting Full backtrace with stream positions A string
Compile-time error output Points to source of error[0] Sometimes points to the #[derive]
Data access Random-access (if supported by underlying stream) Sequential-only
Parser debugging #[br(dbg)] attribute Nothing?
Result value The requested object A tuple containing the object plus the slice of the rest of the data that must be retained for future parsing
Named arguments Yes No
Fallible conversions Yes No
Whole object mapping Yes No
Accidental machine dependency Impossible (requires explicit opt-in) Yes, both endianness and pointer size
Dependency size ~345–770KiB ~1.4–2.1MiB
Magic Variable-length Fixed-length
Attributes count 41 29
Collection reading Any FromIterator Only Vec, HashMap, HashSet
Maximum tuple size 32 11
Helper count 15: file indirection, per-item collection arguments, collect until eof, collect until terminator, punctuated, string 0
Test coverage 96% 73%
GH popularity contest 322 (binrw) + 1438 (binread) 247
There are probably many other subtle differences, and some places where Deku probably does things better (it probably has better performance since it doesn’t seem to do much state tracking for error reporting, and it operates directly on memory slices instead of having an indirection of the stream), but a substantial amount of work has gone into trying to make binrw a best-in-class library and so I think the comparison reflects that.

[0] If binrw ever uselessly points to its derive when an error occurs, it is a bug that should be reported :-).

I think the major operational difference between the two is in that Deku runs async while binrw does not - writing std_async and tokio code is fairly common, especially in efforts which parse streams on-data. That said, this lib's sugar is definitely easier to use, even if you do have to wrap it up in blocking executors.

[kitlith]

I think I get what they're talking about, though it's a weird way to phrase it.

The real difference here is that deku reads from a memory buffer, letting the user handle IO, while binrw handles IO itself. Therefore, deku doesn't need to do anything special to support applications that want to use async IO, but binrw straight up doesn't support it yet.

That said, if you want to use binrw with an async reader right now, you can read the file into a memory buffer in advance using your async reader, and then pass a Cursor as the reader. Inverse for writing.

we should probably support async io directly at some point though.

[sharksforarms]

Ah, I see. The library addition/support to read/write from/to tokio::io::Async[Read|Write] (or other async executor's io trait)

Thanks for the clarification @kitlith !

[joelreymont]

There's one other significant difference in that Deku supports bit fields.

binrw requires me to declare a temp field to grab bitfields from and then the bit fields themselves, plus a bunch of attributes with bit shifts and filters for reading and writing.

Are there any plans to support bitfields in binrw?

[kitlith]

Depends on what kind of support you are looking for. Do you want binrw to natively support reading arrays of structs that are, say, 19 bits wide? (that's probably not going to happen.) Or do you want to work with byte aligned types that happen to contain bitfields?

The latter you can kinda already do, by defining a bitfield using another crate dedicated to bitfields, then generating/writing BinRead/BinWrite for that bitfield struct (or using the map attribute, etc.)

A completely bit-oriented version of binrw would probably use different traits. (i.e. bit oriented io::Read/Write)

[joelreymont]

Just byte-aligned types that contain bitfields.

Without using additional crates and jumping through extra hoops.