Support history buffer to improve compression ratio

[skyskyhu]

Rectently I integrated the zstd plugin with intel QAT hardware acceleration. The hardware acceleration speed is siginificantly faster than that of software compression. However, my primary objective is to reducue total storage size and cost using zstd compression. For all testing files, I observed that the compression result of hardware acceleration is lower than software compression as shown below. I suspect this is caused by the limitation of zstd hardware acceleration doucmented at the link and link. History buffers are not currently supported, resulting in a lower hardware compression ratio compared to software compression. I am wondering if there is any plan to support history buffers in zstd hardware acceleration in the near future. Thanks!

hardware compression with level 12

./benchmark -m1 -l10 -c64K -t5 -E2 -L12 1.txt
Thread 1: Compression: 446020002 -> 130257784, Throughput: Comp: 508 MB/s, Decomp: 1273 MB/s, Compression Ratio: 29.20%, PASS

software compression with level 12

./benchmark -m0 -l10 -c64K -t5 -E2 -L12 1.txt
Thread 1: Compression: 446020002 -> 120037796, Throughput: Comp: 39 MB/s, Decomp: 1350 MB/s, Compression Ratio: 26.91%, PASS

software compression with level 9

./benchmark -m0 -l10 -c64K -t5 -E2 -L9 1.txt
Thread 1: Compression: 446020002 -> 120318010, Throughput: Comp: 99 MB/s, Decomp: 1341 MB/s, Compression Ratio: 26.98%, PASS

[embg]

There is some discussion here: facebook/zstd#3970