Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

du: start printing output immediately #5552

Merged
merged 13 commits into from
Nov 25, 2023
Merged

Conversation

ceteece
Copy link
Contributor

@ceteece ceteece commented Nov 18, 2023

This PR updates du so that it continuously prints output as it is running, as opposed to waiting until all of the data has been collected before printing anything. A separate thread has been added which handles printing data (and errors) as they are produced by the main thread.

Fixes #5516

@ceteece
Copy link
Contributor Author

ceteece commented Nov 18, 2023

This is not ready for review yet, just wanted to get the initial proof of concept out there. It definitely still needs a lot of refinement and cleaning up.

My current solution uses a separate thread for printing, but if that's overkill for this situation then I can switch things back to a single-threaded implementation (please let me know if you have any strong opinions on this!).

Copy link

GNU testsuite comparison:

GNU test failed: tests/du/inacc-dest. tests/du/inacc-dest is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/inotify-dir-recreate. tests/tail/inotify-dir-recreate is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/truncate. tests/tail/truncate is passing on 'main'. Maybe you have to rebase?

src/uu/du/src/du.rs Outdated Show resolved Hide resolved
src/uu/du/src/du.rs Outdated Show resolved Hide resolved
src/uu/du/src/du.rs Outdated Show resolved Hide resolved
Copy link

GNU testsuite comparison:

GNU test failed: tests/tail/inotify-dir-recreate. tests/tail/inotify-dir-recreate is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/retry. tests/tail/retry is passing on 'main'. Maybe you have to rebase?

clint added 2 commits November 20, 2023 21:39
…StatPrinter` to handle printing thread logic
…tion of printing-related variables into `StatPrinter::new`
Copy link

GNU testsuite comparison:

Congrats! The gnu test tests/tail/symlink is no longer failing!
Congrats! The gnu test tests/tail/truncate is no longer failing!
GNU test failed: tests/du/inacc-dest. tests/du/inacc-dest is passing on 'main'. Maybe you have to rebase?

@ceteece
Copy link
Contributor Author

ceteece commented Nov 21, 2023

GNU testsuite comparison:

Congrats! The gnu test tests/tail/symlink is no longer failing!
Congrats! The gnu test tests/tail/truncate is no longer failing!
GNU test failed: tests/du/inacc-dest. tests/du/inacc-dest is passing on 'main'. Maybe you have to rebase?

This test seems to be failing intermittently, likely due to some specific interleaving of prints from the main thread (which still prints some errors) and the printing thread. I haven't been able to reproduce the failure locally yet. Will dig into this more.

Edit:
Looks like this is the failing output:

FAIL: tests/du/inacc-dest
=========================

--- exp	2023-11-21 05:59:01.395501161 +0000
+++ out	2023-11-21 05:59:01.399501125 +0000
@@ -1,7 +1,7 @@
 .
-./a
 ./b
 ./c
 ./d
 ./e
-du: cannot read directory './c': Permission denied
+: cannot read directory './c': Permission denied
+du4	./a
FAIL tests/du/inacc-dest.sh (exit status: 1)

As I suspected, the error message (which is coming from the main thread) is being interleaved with the output from the printing thread. This should be pretty fixable.

…nting a single stat our into its own method in `StatPrinter`
Copy link

GNU testsuite comparison:

Congrats! The gnu test tests/tail/truncate is no longer failing!
GNU test failed: tests/du/inacc-dest. tests/du/inacc-dest is passing on 'main'. Maybe you have to rebase?

@ceteece ceteece marked this pull request as ready for review November 21, 2023 22:58
Copy link

GNU testsuite comparison:

Congrats! The gnu test tests/tail/inotify-dir-recreate is no longer failing!

@cakebaker cakebaker merged commit 84b5e6f into uutils:main Nov 25, 2023
51 of 53 checks passed
@cakebaker
Copy link
Contributor

Good work, thanks :)

@ceteece ceteece deleted the du-print-thread branch February 3, 2025 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

du: should start printing data right away
3 participants