add xxHash checksum and fix a comparison bug #167

trollkarlen · 2025-01-08T17:22:13Z

add the very fast but not cryptographic checksum library xxHash perfect for this application.

try to fix some issue in the checksum test that made the test not run checksum because of too small test files.

trollkarlen · 2025-01-08T18:20:07Z

Issues with non updated workflows, give me some time and i will fix

trollkarlen · 2025-01-10T09:56:43Z

The failing workflows is now fixed!
But I am unsure if the devel or main branch is the target for pull requests ?

pauldreik · 2025-01-10T13:15:15Z

Thanks for the PR! I like the gist of this and will look at it. I can already now say that I am hesitant to change the default.

trollkarlen · 2025-01-10T17:31:30Z

The default checksum is still sha1, in the first revision of the patches I had changed it by accident. Is the devel branch the right for pull requests?

…

On Fri, Jan 10, 2025, 14:15 Paul Dreik ***@***.***> wrote: Thanks for the PR! I like the gist of this and will look at it. I can already now say that I am hesitant to change the default. — Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWIHEZGBYU5NN6SH4QESL2J7BXVAVCNFSM6AAAAABU2NCHWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBSGY4TCMZZGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

trollkarlen · 2025-01-10T17:34:10Z

testcases/checksum_options.sh

@@ -1,19 +1,19 @@
 #!/bin/sh
 # Test that selection of checksum works as expected.

-
+set -x


Remove this it's for debugging only

pauldreik · 2025-01-12T20:27:49Z

I tested this on a fast system (fast CPU, enough RAM and a fast SSD, but inside qemu).

Large files

Creating two files with zeros head -c 1000000000 /dev/zero >1G; cp 1G 1G_copy and invoking it like this: time ./rdfind -checksum xxh128 1G* I get the following:

sha1 takes 1.2 seconds
xxh128 takes 0.3 seconds

With two 7G files, I get

sha1 8.7 s
xxh128 2.4 s

so this is clearly an improvement for large files at least.

Small files

If i create very many small files with head -c100000000 /dev/zero |split --bytes 1000 I get:

sha1 1.5s
xxh128 1.5s

which is not worse than before (good).

xxh builtin benchmark

This is on the debian provided xxh package. The builtin benchmark on the same system gives:

pauldreik@xxx:~/code/eget/rdfind$ xxhsum  -b
xxhsum 0.8.2 by Yann Collet 
compiled as 64-bit x86_64 autoVec little endian with GCC 14.2.0 
Sample of 100 KB...        
 1#XXH32                         :     102400 ->    81133 it/s ( 7923.1 MB/s)   
 3#XXH64                         :     102400 ->   118104 it/s (11533.6 MB/s)   
 5#XXH3_64b                      :     102400 ->   684392 it/s (66835.1 MB/s)   
11#XXH128                        :     102400 ->   682966 it/s (66695.9 MB/s)

Taking the file system overhead into account, xxh128sum 7G 7G_copy (to simulate the double reading rdfind does) takes 1.1 s.

Modifying rdfind buffer size

Increasing the buffer size inside rdfind from 4096 to 16K gets the time down from 2.4 to to 1.7 seconds. Still a bit far from 1.1 seconds, but better.

trollkarlen · 2025-01-13T14:20:54Z

Maybe we could change the buffer size dependent on the checksum method used to make it as fast as possible.
And also a cli switch to change this of you have other form of media that preform better at other buffer sizes ?

trollkarlen · 2025-01-13T14:33:32Z

I also did a test of this and got a speedup between 60%-500% faster performance on fedora, ssd and Intel(R) Xeon(R) CPU X5670. The faster the disk the bigger the speedup, until we hit the limit for xxhash.

This is the test script:

cat <<'EOF' > test.sh
#!/bin/bash

TEST_DIR=testdir
mkdir -p "$TEST_DIR"

if [[ ! -f "$TEST_DIR/a" ]]; then
    echo "creating test files in $TEST_DIR"
    head -c $((1024*1024*500)) /dev/random >"$TEST_DIR/a"
    cp "$TEST_DIR/a" "$TEST_DIR/b"
    cp "$TEST_DIR/a" "$TEST_DIR/c"
    cp "$TEST_DIR/a" "$TEST_DIR/d"
    cp "$TEST_DIR/a" "$TEST_DIR/e"
fi

echo drop caches
echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
echo "test coldcache sha1"
time ./rdfind -checksum sha1 -dryrun true -deleteduplicates true "$TEST_DIR"

echo drop caches
echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
echo "test coldcache xxhash"
time ./rdfind -checksum xxh128 -dryrun true -deleteduplicates true "$TEST_DIR"

echo "test hotcache sha1"
time ./rdfind -checksum sha1 -dryrun true -deleteduplicates true "$TEST_DIR"
/usr/bin/time -f '%M kB' ./rdfind -checksum sha1 -dryrun true -deleteduplicates true "$TEST_DIR"

echo "test hotcache xxhash"
time ./rdfind -checksum xxh128 -dryrun true -deleteduplicates true "$TEST_DIR"
/usr/bin/time -f '%M kB' ./rdfind -checksum xxh128 -dryrun true -deleteduplicates true "$TEST_DIR"
EOF

$ sudo bash test.sh                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               drop caches                                                                                                                                                                                                                                
test coldcache sha1                                                                                                                                                                                                                        
(DRYRUN MODE) Now scanning "testdir", found 5 files.                                                                                                                                                                                       
(DRYRUN MODE) Now have 5 files in total.                                                                                                                                                                                                   
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.                                                                                                                                                                           
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB                                                                                                                                                                                      
Removed 0 files due to unique sizes from list. 5 files left.                                                                                                                                                                               (DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.                                                                                                                                    
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.                                                                                                                                     
(DRYRUN MODE) Now eliminating candidates based on sha1 checksum: removed 0 files from list. 5 files left.                                                                                                                                  
(DRYRUN MODE) It seems like you have 5 files that are not unique                                                                                                                                                                           
(DRYRUN MODE) Totally, 2 GiB can be reduced.                                                                                                                                                                                               
(DRYRUN MODE) Now making results file results.txt                                                                                                                                                                                          
(DRYRUN MODE) Now deleting duplicates:                                                                                                                                                                                                     
(DRYRUN MODE) delete testdir/b                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/c                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/d                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/e                                                                                                                                                                                                             
(DRYRUN MODE) Deleted 4 files.                                                                                                                                                                                                             
                                                                                                                                                                                                                                           
real    0m9.722s                                                                                                                                                                                                                           
user    0m7.409s                                                                                                                                                                                                                           
sys     0m1.559s                                                                                                                                                                                                                           
drop caches                                                                                                                                                                                                                                
test coldcache xxhash                                                                                                                                                                                                                      
(DRYRUN MODE) Now scanning "testdir", found 5 files.                                                                                                                                                                                       
(DRYRUN MODE) Now have 5 files in total.                                                                                                                                                                                                   
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.                                                                                                                                                                           
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB                                                                                                                                                                                      
Removed 0 files due to unique sizes from list. 5 files left.                                                                                                                                                                               
(DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.                                                                                                                                    
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.                                                                                                                                     
(DRYRUN MODE) Now eliminating candidates based on xxh128 checksum: removed 0 files from list. 5 files left.                                                                                                                                
(DRYRUN MODE) It seems like you have 5 files that are not unique                                                                                                                                                                           
(DRYRUN MODE) Totally, 2 GiB can be reduced.                                                                                                                                                                                               
(DRYRUN MODE) Now making results file results.txt                                                                                                                                                                                          
(DRYRUN MODE) Now deleting duplicates:                                                                                                                                                                                                     
(DRYRUN MODE) delete testdir/b                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/c                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/d                                                                                                                                                                                                             
(DRYRUN MODE) delete testdir/e                                                                                                                                                                                                             
(DRYRUN MODE) Deleted 4 files.                                                                                                                                                                                                             
                                                                                                                                                                                                                                           
real    0m6.148s                                                                                                                                                                                                                           
user    0m1.375s                                                                                                                                                                                                                           
sys     0m2.175s
test hotcache sha1                                                                                                                                                                                                                         
(DRYRUN MODE) Now scanning "testdir", found 5 files.                                                                                                                                                                                       
(DRYRUN MODE) Now have 5 files in total.                                                                                                                                                                                                   
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.                                                                                                                                                                           
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB
Removed 0 files due to unique sizes from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on sha1 checksum: removed 0 files from list. 5 files left.
(DRYRUN MODE) It seems like you have 5 files that are not unique
(DRYRUN MODE) Totally, 2 GiB can be reduced.
(DRYRUN MODE) Now making results file results.txt
(DRYRUN MODE) Now deleting duplicates:
(DRYRUN MODE) delete testdir/b
(DRYRUN MODE) delete testdir/c
(DRYRUN MODE) delete testdir/d
(DRYRUN MODE) delete testdir/e
(DRYRUN MODE) Deleted 4 files.

real    0m7.499s
user    0m6.868s
sys     0m0.588s
(DRYRUN MODE) Now scanning "testdir", found 5 files.
(DRYRUN MODE) Now have 5 files in total.
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB
Removed 0 files due to unique sizes from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on sha1 checksum: removed 0 files from list. 5 files left.
(DRYRUN MODE) It seems like you have 5 files that are not unique
(DRYRUN MODE) Totally, 2 GiB can be reduced.
(DRYRUN MODE) Now making results file results.txt
(DRYRUN MODE) Now deleting duplicates:
(DRYRUN MODE) delete testdir/b
(DRYRUN MODE) delete testdir/c
(DRYRUN MODE) delete testdir/d
(DRYRUN MODE) delete testdir/e
(DRYRUN MODE) Deleted 4 files.
4104 kB
test hotcache xxhash
(DRYRUN MODE) Now scanning "testdir", found 5 files.
(DRYRUN MODE) Now have 5 files in total.
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB
Removed 0 files due to unique sizes from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on xxh128 checksum: removed 0 files from list. 5 files left.
(DRYRUN MODE) It seems like you have 5 files that are not unique
(DRYRUN MODE) Totally, 2 GiB can be reduced.
(DRYRUN MODE) Now making results file results.txt
(DRYRUN MODE) Now deleting duplicates:
(DRYRUN MODE) delete testdir/b
(DRYRUN MODE) delete testdir/c
(DRYRUN MODE) delete testdir/d
(DRYRUN MODE) delete testdir/e
(DRYRUN MODE) Deleted 4 files.

real    0m1.476s
user    0m0.812s
sys     0m0.655s
(DRYRUN MODE) Now scanning "testdir", found 5 files.
(DRYRUN MODE) Now have 5 files in total.
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
(DRYRUN MODE) Total size is 2621440000 bytes or 2 GiB
Removed 0 files due to unique sizes from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on first bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on last bytes: removed 0 files from list. 5 files left.
(DRYRUN MODE) Now eliminating candidates based on xxh128 checksum: removed 0 files from list. 5 files left.
(DRYRUN MODE) It seems like you have 5 files that are not unique
(DRYRUN MODE) Totally, 2 GiB can be reduced.
(DRYRUN MODE) Now making results file results.txt
(DRYRUN MODE) Now deleting duplicates:
(DRYRUN MODE) delete testdir/b
(DRYRUN MODE) delete testdir/c
(DRYRUN MODE) delete testdir/d
(DRYRUN MODE) delete testdir/e
(DRYRUN MODE) Deleted 4 files.
4044 kB

trollkarlen · 2025-01-13T15:28:03Z

I did a patch for setting the buffersize from cli also should i push that also ?

looks like the 32KB is the optimal for xxhash and my disks/memory.

Testing buffersize 4096                                                                                                                                                                                                                    
test sha1                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                           
real    0m7.370s                                                                                                                                                                                                                           
user    0m6.730s                                                                                                                                                                                                                           
sys     0m0.596s                                                                                                                                                                                                                           
test xxhash                                                                                                                                                                                                                                
                                                                                                                                                                                                                                           
real    0m1.474s                                                                                                                                                                                                                           
user    0m0.842s                                                                                                                                                                                                                           
sys     0m0.623s                                                                                                                                                                                                                           
test xxhash memusage                                                                                                                                                                                                                       
4052 kB                                                                                                                                                                                                                                    
Testing buffersize 8192                                                                                                                                                                                                                    
test sha1                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                           
real    0m7.176s                                                                                                                                                                                                                           
user    0m6.592s                                                                                                                                                                                                                           
sys     0m0.541s                                                                                                                                                                                                                           
test xxhash                                                                                                                                                                                                                                
                                                                                                                                                                                                                                           
real    0m1.105s                                                                                                                                                                                                                           
user    0m0.483s                                                                                                                                                                                                                           
sys     0m0.615s                                                                                                                                                                                                                           
test xxhash memusage                                                                                                                                                                                                                       
4048 kB                                                                                                                                                                                                                                    
Testing buffersize 16384                                                                                                                                                                                                                   
test sha1                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                           
real    0m6.981s                                                                                                                                                                                                                           
user    0m6.398s
sys     0m0.543s
test xxhash

real    0m1.010s
user    0m0.463s
sys     0m0.541s
test xxhash memusage
4116 kB
Testing buffersize 32768
test sha1

real    0m7.147s
user    0m6.585s
sys     0m0.524s
test xxhash

real    0m0.954s
user    0m0.466s
sys     0m0.482s
test xxhash memusage
4192 kB
Testing buffersize 65536
test sha1

real    0m7.160s
user    0m6.595s
sys     0m0.527s
test xxhash

real    0m0.944s
user    0m0.451s
sys     0m0.486s
test xxhash memusage
4144 kB
Testing buffersize 131072
test sha1

real    0m7.067s
user    0m6.525s
sys     0m0.504s
test xxhash

real    0m0.999s
user    0m0.469s
sys     0m0.523s
test xxhash memusage
4240 kB

trollkarlen · 2025-01-13T15:45:25Z

I updated the pathset because i had forgotten to update the help string

- std::cerr << "expected md5/sha1/sha256/sha512, not \""
+ std::cerr << "expected md5/sha1/sha256/sha512/xxh128, not \""

trollkarlen · 2025-01-13T16:09:55Z

Made new pull request for buffersize #177

pauldreik · 2025-01-14T06:51:33Z

Thanks for this work, I intend to merge this feature.
I fixed the CI issues and the codesize bug on devel separately.

Please make the PR against current devel
xxhash should be optional
the default hash should be unchanged, and the code comment should follow that
let's control the buffer size outside of this PR
don't touch the CI jobs except for adding a with/without job so we know the default works (in presence and absence of the library)
please make separate commits - one for autoconf , one for the code changes, one for the man page, one for the documentation etc.
inline the hash size macro into the cc file if possible

I made a branch here: https://github.com/pauldreik/rdfind/tree/trollkarlens_xxhash to test out the changes I want (work in progress, but you can at least see the autoconf change which makes it optional). I am totally fine with fixing this myself if you don't want to!

trollkarlen · 2025-01-22T15:41:27Z

updated the patch set to reflect the suggestions in #167 (comment), please review.

The hard part was the ifdef and tried to keep that to a minimum.
Thats why I left the help text and so on there independent on with-xxhash.
Also did not move "hash size macro" to the .cc file to minimise the need for ifdefs.

pauldreik · 2025-01-22T19:42:06Z

.github/workflows/ubuntu-default-with-xxhash.yml

+      run: WITH_XXHASH=1 make check
+    - name: WITH_XXHASH=1 make distcheck


why are the WITH_XXHASH=1 here?

UPDATE: aha, it was for testing. ok, see the comment there.

Checksum.cc

Checksum.hh

configure.ac

rdfind.cc

pauldreik · 2025-01-22T19:57:57Z

testcases/checksum_options.sh

-
-allchecksumtypes="md5 sha1 sha256 sha512"
-
+if [ "$WITH_XXHASH" = "1" ]; then


there must be some better way to detect this than using the environment variable.

not sure if autoconf can help here.

Did not find any good way to get/set that from autoconf, but if you know one please advice.

The best would be if this was set as an env in the Makefile directly when HAVE_LIBXXHASH is set, but could not find a way to do that.

pauldreik · 2025-01-22T19:58:32Z

testcases/checksum_speedtest.sh

@@ -21,7 +27,7 @@ if [ ! -e speedtest/largefile1 ] ; then
 fi


-for checksumtype in md5 sha1 sha256; do
+for checksumtype in $allchecksumtypes; do


wow, sha512 was missing before! nice to get that fixed.

maybe move allchecksumtypes to the common_funcs.sh so its the same everywhere ?

Signed-off-by: Robert Marklund <[email protected]>

add a very very fast non cryptographic checksumming library https://xxhash.com Signed-off-by: Robert Marklund <[email protected]>

Signed-off-by: Robert Marklund <[email protected]>

trollkarlen force-pushed the devel branch from 825cd14 to caf0f61 Compare January 8, 2025 18:09

trollkarlen force-pushed the devel branch from caf0f61 to 30e7185 Compare January 10, 2025 09:25

trollkarlen commented Jan 10, 2025

View reviewed changes

pauldreik mentioned this pull request Jan 12, 2025

fix minor bug in digest length check #175

Merged

trollkarlen force-pushed the devel branch from 30e7185 to 0738996 Compare January 13, 2025 15:41

trollkarlen force-pushed the devel branch from 0738996 to 20b4063 Compare January 22, 2025 15:35

trollkarlen force-pushed the devel branch from 20b4063 to a5fc247 Compare January 22, 2025 15:46

pauldreik requested changes Jan 22, 2025

View reviewed changes

detect xxhash in configure.ac

5e5c186

Signed-off-by: Robert Marklund <[email protected]>

trollkarlen force-pushed the devel branch from a5fc247 to b441a2d Compare January 22, 2025 22:31

Paul Dreik and others added 3 commits January 22, 2025 23:37

add xxHash checksum

5594c67

add a very very fast non cryptographic checksumming library https://xxhash.com Signed-off-by: Robert Marklund <[email protected]>

update manpages for xxhash

5c3c7ff

Signed-off-by: Robert Marklund <[email protected]>

add workflow and tests for xxhash

51a7ced

Signed-off-by: Robert Marklund <[email protected]>

trollkarlen force-pushed the devel branch from b441a2d to 51a7ced Compare January 22, 2025 22:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add xxHash checksum and fix a comparison bug #167

add xxHash checksum and fix a comparison bug #167

trollkarlen commented Jan 8, 2025

trollkarlen commented Jan 8, 2025

trollkarlen commented Jan 10, 2025

pauldreik commented Jan 10, 2025

trollkarlen commented Jan 10, 2025 via email

trollkarlen Jan 10, 2025

pauldreik commented Jan 12, 2025 •

edited

Loading

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025 •

edited

Loading

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025

pauldreik commented Jan 14, 2025

trollkarlen commented Jan 22, 2025 •

edited

Loading

pauldreik Jan 22, 2025

pauldreik Jan 22, 2025

trollkarlen Jan 22, 2025 •

edited

Loading

pauldreik Jan 22, 2025

trollkarlen Jan 22, 2025

		run: WITH_XXHASH=1 make check
		- name: WITH_XXHASH=1 make distcheck


		allchecksumtypes="md5 sha1 sha256 sha512"

		if [ "$WITH_XXHASH" = "1" ]; then

add xxHash checksum and fix a comparison bug #167

Are you sure you want to change the base?

add xxHash checksum and fix a comparison bug #167

Conversation

trollkarlen commented Jan 8, 2025

trollkarlen commented Jan 8, 2025

trollkarlen commented Jan 10, 2025

pauldreik commented Jan 10, 2025

trollkarlen commented Jan 10, 2025 via email

trollkarlen Jan 10, 2025

Choose a reason for hiding this comment

pauldreik commented Jan 12, 2025 • edited Loading

Large files

Small files

xxh builtin benchmark

Modifying rdfind buffer size

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025 • edited Loading

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025

trollkarlen commented Jan 13, 2025

pauldreik commented Jan 14, 2025

trollkarlen commented Jan 22, 2025 • edited Loading

pauldreik Jan 22, 2025

Choose a reason for hiding this comment

pauldreik Jan 22, 2025

Choose a reason for hiding this comment

trollkarlen Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

pauldreik Jan 22, 2025

Choose a reason for hiding this comment

trollkarlen Jan 22, 2025

Choose a reason for hiding this comment

pauldreik commented Jan 12, 2025 •

edited

Loading

trollkarlen commented Jan 13, 2025 •

edited

Loading

trollkarlen commented Jan 22, 2025 •

edited

Loading

trollkarlen Jan 22, 2025 •

edited

Loading