Dynamic Prefix: Support configuring uploaded file paths/names within a bucket

[adamvduke]

The current version of the plugin stores all of the uploaded files in the root of the configured bucket. It would be valuable to be able to configure the path and filename within the bucket to allow for ease of longtime archiving without having to scroll through 1000's of files when viewing the contents of the bucket. Maybe a static configuration, or a proc that can be eval'd before uploading the new file?

s3 {
  ...
  path => "{year}/{month}/{day}/{hour}/{host}{uuid}"
  path_proc => '{|time|  "#{time.year}/#{time.month}/#{time.day}/#{time.hour}/#{Socekt.gethostname}#{uuid}'
}

[ph]

I would go with the static configuration (similar to the https://github.com/logstash-plugins/logstash-output-file). In the file ouput you can use the data in the event to generate the path of the file.

[pruthvintss]

Hey, is there an update on how this can be done? would like to configure the path with event fields

[bradphipps]

This would definitely be useful. Outputting to folders within in a bucket based on their type would make data archival/retention/deletion processes easier to manage/safer. Sadly the following config doesn't work: bucket => "logstash-output-bucket/%{type}"

[radupantiru]

+1

[shangliuyan]

+1

[wiibaa]

S3 newbie question,
does this request means that

1. the @Prefix config should be able to use event data like %{type}/ or `%{+YYYY}/%{+MM}/%{dd}/ ??to specify a "folder structure"
2. the filename itself currently generated in this method should be able to use event data ?

If this is right, I could dig in further, most probably looking for 1. first
Thoughts ?

[adamvduke]

It looks like there are several implementations of this feature to varying degrees. I'm currently not focused on this issue, but perhaps someone on this thread could contribute to one of the open PR's to one/more/all of them along.

#44
#59
#70
#81

[FlorinAndrei]

The prefix parameter for the S3 output plugin now (v5.2.0 or newer) supports string interpolation based on field values.

s3 {
  prefix => "logstash-test/%{event}/"
}