Add more options to file source patterns, e.g. "follow_links", "case_insensitive", "max_depth"

[pablosichert]

Current Vector Version

vector 0.12.0

Use-cases

Give more fine-grained control over file include patterns, e.g. if symlinks should be followed and if the pattern should be considered case (in)sensitive.

Proposal

Add options to the file source config that are propagated to the GlobWalkBuilder and use that builder for the include_patterns.

References

• Curly braces in the include directive, file pattern #5003
• fix(file source): Allow file pattern with curly braces in include directive #5927

[mtekel]

Symlinks are a bit of issue. 0.32.0 sees original file and symlink as two separate and then complains checksum/fingerprint is the same and it spams the log. Interestingly, error messages have repetition supression, but info messages don't.