datasets: fix dataset clobbering, and file naming - v1 #341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By using a hash of the content, a new file was created everytime the dataset was updated and never cleaned up. To address this, use a filename that doesn't change based on the content. Bug: #6763
To prevent dataset files from difference sources from overwriting each other, give each file downloaded and extracted a prefix based on the URL (a hash). This ensures unique filenames across all rulesets. This mostly matters for datasets, as when datasets are processed we are working with a merged set of filenames, unlike rules which are parsed much earlier when we still have a list of files. Not the most elegant solution, but saves a rather large refactor. Bug: #6833
inashivb
reviewed
Mar 7, 2024
This file has a dataset that has the same name as a dataset in the pawpatrules that I used to test the clobbering. To test the extraneous files, you'd have to update, modify the dataset in the ruleset, and update and again and see the new file creation. Then run with patch and see how it no longer happens. This is the more critical one, as some datasets are pretty large and updated daily. |
|
For example, after a few weeks of rulesets with datasets that update frequently, I have these files in my rules directory: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
instead the filename path:
https://redmine.openinfosecfoundation.org/issues/6763
with a hash of the URL to prevent duplicate filenames from
cloberring each other, in particular dataset files:
https://redmine.openinfosecfoundation.org/issues/6833