Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major bug fixing for s3 file listing #47

Merged
merged 2 commits into from
Oct 22, 2024
Merged

Major bug fixing for s3 file listing #47

merged 2 commits into from
Oct 22, 2024

Conversation

ferd
Copy link
Owner

@ferd ferd commented Oct 21, 2024

If you start a directory monitor as a server in S3 mode and do so by creating a directory object, it will get returned in listing queries. Unfortunately, the list calls will require an is_regular/1 check (which would be "costly") to filter out.

Instead, add an implicit check in S3 where an object with a trailing / is considered to be a directory and is ignored. This is generally safe since MacOS and Linux forbid / in paths and shouldn't happen outside of such manual S3 object creations.

More importantly though, this fixes directory listing pagination within S3's API use. Turns out the translation from integration tests to unit tests was wrong and if in practice a directory in S3 was encountered with over 1000 files, pagination failed and the tracking failed on every restart.

That took a while to figure out.

If you start a directory monitor as a server in S3 mode and do so by
creating a directory object, it will get returned in listing queries.
Unfortunately, the list calls will require an `is_regular/1` check
(which would be "costly") to filter out.

Instead, add an implicit check in S3 where an object with a trailing `/`
is considered to be a directory and is ignored. This is generally safe
since MacOS and Linux forbid `/` in paths and shouldn't happen outside
of such manual S3 object creations.
@ferd ferd changed the title Filter out directories from listing Filter out directory objects from s3 listing Oct 21, 2024
@ferd ferd changed the title Filter out directory objects from s3 listing Major bug fixing for s3 file listing Oct 22, 2024
Turns out the translation from integration to unit tests was wrong and
if in practice a directory in S3 was encountered with over 1000 files,
pagination failed and the tracking failed on every restart.
@ferd ferd merged commit f702dd9 into main Oct 22, 2024
1 check passed
@ferd ferd deleted the small-bugfixes branch October 22, 2024 03:57
@ferd
Copy link
Owner Author

ferd commented Oct 22, 2024

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant