lzfse support

[blacktop]

Would it be possible to add https://github.com/lzfse/lzfse support?

[klauspost]

@blacktop I am not really sure what its purpose is. It seems like there is a 2GB size limit, so it is for blocks only.

It seems to compress worse than zstandard at default settings (level 3) and a whole lot slower - at least the library above.

A decoder could be feasible, but honestly someone should just wrap the library in cgo.

[blacktop]

Ya that's what I did https://github.com/blacktop/lzfse-cgo, but it feels gross. Thought if you "liked" the algo you might pure-Go/optimize the heck out of it 😁

I hate having to introduce CGO as it makes the build process so gross and calling C is dangerous.

Lzfse is Apple's compressor of choice for most things.

[klauspost]

Yeah, it may be great for Apple, but it doesn't really seem great for much else.

For now, I will for now focus on "best in class" compressors, or commonly used ones. It just seems like the format isn't that great.

[blacktop]

Understood

[blacktop]

Hi @klauspost, Apple has introduced a NEW private compression algorithm called LZBITMAP which you can see mentioned in their docs here

There is no source anywhere just a dylib in the dyld_shared_cache called /usr/lib/libcompression.dylib

It is supposed to be pretty efficient/powerful due to leveraging the vector instructions added in modern processors.

Would this be something you'd be interesting in taking a look at?

So far Apple uses it in APFS so things like apfs-fuse and my own APFS non-darwin parsers are broken until Apple open sources the algo or someone does the RE.

Thanks!

[blacktop]

Oh also I added it here because I think it's a part of the Apple LZ compression algos, but it could be it's own thing and not tied to lzfse.

[klauspost]

Without any format specification and/or source code there isn't really much I can do wrt LZBITMAP.