is it possible to parse the folder paths and add as labels

[dss010101]

i have a folder structure that looks like this

/app/{batch_id}/{partition_id}/logs

i'm using the following to read filter for these logs:
include = ["/app/*/*.log"]

so for example:

/app/1000/0001/logs/run.log
/app/1001/0001/logs/run.log
/app/1001/0002/logs/run.log

I can read and parse the logs fine. But as you can see above the log names are all the same. What is different is the sub paths that has folders serving as key identifiers. i'd like to parse those out into labels associated with every row parsed for the respective log.

Any thoughts on how i can do this?

[jszwedko]

Hi @dss010101 !

I think the easiest way to do this would be use https://vector.dev/docs/reference/vrl/functions/#parse_regex to pull out the bits you are interested in.

[dss010101]

Hi @dss010101 !

I think the easiest way to do this would be use https://vector.dev/docs/reference/vrl/functions/#parse_regex to pull out the bits you are interested in.

right..but i guess what i dont know is how to get at that folder path for a transformer. I know how to parse log lines in the transformer using parse_regex, but is the source file path available in a transformer also? if so by what variable or property do i reference it?

[jszwedko]

Hey! Yes it should be in the event. You can use vector tap or the log function to see the event in your transform. I believe it is under the file key.

[mbneimann-at-work]

Hi @dss010101

You can do it in a remap transformes with split()

source = '''
dirs= split!(.file, "/")
.batch_id = dirs[1]
.partition_id = dirs[2]
'''