filehashes: fix for unique filehash filenames

[jasonish]

Commit 8725e56 gave each downloaded a file a unique name so dataset files from different sources wouldn't clobber each other, but this was applied to all files breaking file hash lists as that code wasn't updated for the new filename scheme.

Update the file hashing code to find the files based on the filename prefix of the rule referencing the file.

Bug: #6854

Test ruleset: https://github.com/jasonish/suricata-test-rules/archive/refs/heads/main.zip

Perhaps in combination with https://rules.pawpatrules.fr/suricata/paw-patrules.tar.gz which has a dataset.

[victorjulien]

Should we have SV test support and tests for SU?

[inashivb]

This does seem to work. Was it always broken with --local?

[inashivb]

Should we have SV test support and tests for SU?

Suricata-update has its own test suite in tests/ , could you please tell what do you aim for the s-v test to look like?
The test suite already covers:

• commands
• configuration options
• modifier confs
• flowbit dependency handling
• signature mapping
• classification config merging
  and more.

I think one thing we could do is ensure that if all free sources are enabled, then suricata-update does not error out.

[jasonish]

This does seem to work. Was it always broken with --local?

Somethings broken with --local?

[jasonish]

I think one thing we could do is ensure that if all free sources are enabled, then suricata-update does not error out.

The integration tests avoid hitting the network now, so not sure how that will work unless they go and hit all the rule sources. Which I'm not that keen to do in automated tests.

[victorjulien]

Should we have SV test support and tests for SU?

Suricata-update has its own test suite in tests/ , could you please tell what do you aim for the s-v test to look like? The test suite already covers:

* commands

* configuration options

* modifier confs

* flowbit dependency handling

* signature mapping

* classification config merging
  and more.

I think one thing we could do is ensure that if all free sources are enabled, then suricata-update does not error out.

Can this issue be properly tested in the framework? If so that should be sufficient.

[jasonish]

This does seem to work. Was it always broken with --local?

Somethings broken with --local?

I see. If you take my test ruleset and extract it to disk and point to it with --local it won't find the hash list files. I believe this has always been broken.

However, datasets are picked up with --local, and were in 1.3.0 as well. So we're good there. Probably won't fix this release, since it was broken already.

[inashivb]

Somethings broken with --local?

yeah. i get the same error as w/o this patch. Haven't tried to check the why though.
See:

$./bin/suricata-update --local ~/Downloads/rules/ #this dir contains all the rules you've mentioned for testing
13/3/2024 -- 21:24:49 - <Error> -- filemd5 file testmyids.md5 was not found
13/3/2024 -- 21:24:49 - <Error> -- filesha1 file testmyids.sha1 was not found
13/3/2024 -- 21:24:49 - <Error> -- filesha256 file testmyids.sha256 was not found

[jasonish]

Can this issue be properly tested in the framework? If so that should be sufficient.

Yes, the integration test was updated to use my test ruleset and verify the extraction of the file hash lists to the proper location.

[inashivb]

Comments were addressed. LGTM 🚀

[inashivb]

I think one thing we could do is ensure that if all free sources are enabled, then suricata-update does not error out.

The integration tests avoid hitting the network now, so not sure how that will work unless they go and hit all the rule sources. Which I'm not that keen to do in automated tests.

Could we keep a copy that we update at certain intervals? ref: https://github.com/OISF/suricata-verify/tree/master/tests/test-ruleparse-etopen-01

[jasonish]

I think one thing we could do is ensure that if all free sources are enabled, then suricata-update does not error out.

The integration tests avoid hitting the network now, so not sure how that will work unless they go and hit all the rule sources. Which I'm not that keen to do in automated tests.

Could we keep a copy that we update at certain intervals? ref: https://github.com/OISF/suricata-verify/tree/master/tests/test-ruleparse-etopen-01

We have a copy of emerging threats. I'm not sure how keeping a copy of all would be helpful.

Now something else that could be interesting is an external tool that validates all public rulesets.. Makesure the links are all there, and which versions of Suricata they parse for. Interesting project I think, but outside of Suricata-Update.

[jasonish]

Merged.