Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache total file size/count #1992

Open
hectorcorrea opened this issue Nov 21, 2024 · 0 comments
Open

Cache total file size/count #1992

hectorcorrea opened this issue Nov 21, 2024 · 0 comments

Comments

@hectorcorrea
Copy link
Member

Calculating the total file size and total file count is slow for large datasets (e.g. datasets with 20-40K files). We should consider caching that information in the Work table along with the last date the values were updated. For example:

work.total_file_size = 1234559
work.total_file_count = 23000
work.total_file_as_of = '2024-11-21 14:00:00'

The process that updates the UploadSnapshot could automatically refresh those values every time it checks for adds/deletes/modifications. That process runs in the background.

The UI could display a little information icon next to the total file size and total file count indicating that the values displayed are current as of the stored data. In the vast majority of cases the values will be up to date, if there is any doubt, the user would be able to click on the information icon to confirm if they are too old (e.g. if a new file was just added but it does not show in the count or size)

See also #1989

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant