Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make bitser endian safe #36

Open
gvx opened this issue Dec 31, 2024 · 5 comments
Open

Make bitser endian safe #36

gvx opened this issue Dec 31, 2024 · 5 comments
Labels

Comments

@gvx
Copy link
Owner

gvx commented Dec 31, 2024

Currently, bitser implicitly expects data to be deserialized on a computer with the same endianness as the computer that data was serialized on. If that is not the case, bitser is likely to either silently give you the wrong data or give confusing error messages.

Here's the potential solutions I can see to that:

  • Status quo. Simple, but a footgun waiting to happen.
  • Mark endianness, error if non-matching. Only slightly less simple and has the advantage of never producing the wrong data and at least allowing the error message to make more sense.
  • Mark endianness, byteswap when deserializing. Would result in additional complexity of the bitser internals (which I keep intentionally simple), but would completely solve the problem.
  • Make bitser serialize to little-endian by mandate, big-endian machines have to byteswap (or vice versa). A lot like the previous solution. Difference is that internals get to be slightly less complex (since byteswapping no longer depends on per-buffer state), the 2 or 3 bytes that the endianness marker takes up does not have to be added to the output, and for some platforms, both serializing and deserializing needs byteswapping in all cases.
  • Configurable endianness. This is going to be a whole can of worms on its own, so I'm not going to consider this for the moment.
@gvx gvx added the bug label Dec 31, 2024
@gvx
Copy link
Owner Author

gvx commented Dec 31, 2024

Currently, the only place where this is directly relevant is when we're serializing numbers (except integers in the range [-27, 100], those are fine), but many other types involve serializing numbers, so this also breaks for:

  • Strings longer than 100 bytes;
  • Tables with more than 100 entries in their array part and/or more than 100 entries in their hash part;
  • References to values whose first occurrence is later than the 100th reference-able value;
  • Resources with a name longer than 100 characters (but please don't give your resources names that long);
  • Extensions with a name longer than 100 characters (please don't give the extensions you use names that long either).

@masakk1
Copy link

masakk1 commented Jan 12, 2025

This would be nice, so that the library ffi wasn't necessary. Specially because I want to run it with vanilla lua instead of love/luajit.

@gvx
Copy link
Owner Author

gvx commented Jan 24, 2025

Unfortunately, @masakk1, bitser is built from the ground up to rely on the ffi library, and that will not change if/when bitser is endian safe.

@masakk1
Copy link

masakk1 commented Feb 2, 2025

I see. But if ffi is from luaJIT, shouldn't I be able to run it with that?

@masakk1
Copy link

masakk1 commented Feb 2, 2025

Actually, forget I said anything, not only did I mix up both this issue, but I actually use love.timer.getTime() so I need to use love2d anyways. Nevermind!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants