Major bug fixing for s3 file listing #47
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If you start a directory monitor as a server in S3 mode and do so by creating a directory object, it will get returned in listing queries. Unfortunately, the list calls will require an `is_regular/1` check (which would be "costly") to filter out. Instead, add an implicit check in S3 where an object with a trailing `/` is considered to be a directory and is ignored. This is generally safe since MacOS and Linux forbid `/` in paths and shouldn't happen outside of such manual S3 object creations.
ferd
changed the title
ferd
changed the title
Turns out the translation from integration to unit tests was wrong and if in practice a directory in S3 was encountered with over 1000 files, pagination failed and the tracking failed on every restart.
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If you start a directory monitor as a server in S3 mode and do so by creating a directory object, it will get returned in listing queries. Unfortunately, the list calls will require an
is_regular/1check (which would be "costly") to filter out.Instead, add an implicit check in S3 where an object with a trailing
/is considered to be a directory and is ignored. This is generally safe since MacOS and Linux forbid/in paths and shouldn't happen outside of such manual S3 object creations.More importantly though, this fixes directory listing pagination within S3's API use. Turns out the translation from integration tests to unit tests was wrong and if in practice a directory in S3 was encountered with over 1000 files, pagination failed and the tracking failed on every restart.
That took a while to figure out.