Corrupted block detected error reported in -hb mode

[michaelmaniscalco]

The following file produces the following error:

./programs/fse -bh ./badfile FSE : Finite State Entropy, 64-bits demo by Yann Collet (Nov 9 2018) !! Error decompressing block 986 of cSize 12110 !! => (Corrupted block detected)

File is way too large to attach to ticket. It can be downloaded off my personal webiste:
www.michael-maniscalco.com/download/badfile.zip

[Cyan4973]

Thanks for the report @michaelmaniscalco, I'll look into it.

[Cyan4973]

Sorry, I completely forgot this old ticket.
Indeed, I believe this is the same story as #102 : the huffman codec, as used by the benchmark unit, is limited to 128 KB input.
I did not realized that it could be the problem here, as the error message mentions a block of compressed size 12110 bytes. But when I downloaded badfile.zip, I noticed it contains many big files, all > 128 KB. So it feels consistent.

The lesson here is that the error message is really unspecific, it doesn't help understanding and fixing the situation.
So this would be one thing to improve.

[Uzume]

I think the real bug here is that the benchmark module does not check and handle the 128kB limit. It should handle the issue by either giving an error (passing the buck) or perform the splitting necessary to eliminate the issue itself. Without providing either the splitting at the I/O layer or an appropriate error message, can result in a corrupted block at some later and perhaps confusing point in time.

Adding an error for the 128kB limit (or alternatively implementing the appropriate splitting and removing the limitation) should eliminate the corrupted block error unless it is the result of some other condition. The moral here is that the limitations of an algorithm should always be checked even if you do not want to handle it beyond providing an error.