Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreaded zstd compression support #688

Closed
nalabrie opened this issue Jul 13, 2024 · 2 comments
Closed

Multithreaded zstd compression support #688

nalabrie opened this issue Jul 13, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@nalabrie
Copy link
Contributor

The Unix zstd tool can compress using multiple threads. Number of threads can be specified with the -T flag and the -T0 flag will "attempt to detect and use the number of physical CPU cores" (from the man page).

I suggest having the option to enable zstd multithreading or perhaps making it the default in a fashion similar to the -T0 flag mentioned above.

In my experience, this makes a huge difference when using a high compression level. I just compressed 555MB of data using ouch c --slow and it took around 10 minutes. Then when using a similar maximum compression zstd command on the same data, it took about 30 seconds (sorry for not timing properly). Resulting file size was roughly the same, but I can't be certain how well I matched the settings used by Ouch's --slow flag.

A quick search of the zstd crate docs used by Ouch shows that it supports multithreading.

@nalabrie nalabrie added the enhancement New feature or request label Jul 13, 2024
@philpax
Copy link

philpax commented Jan 18, 2025

Should this be closed now, given that #689 has been merged?

@marcospb19
Copy link
Member

Thanks for noticing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants