Can't get batch working with s3 sink

[extremeandy]

I have the following configured in vector.yaml:

sinks:
  s3:
    type: aws_s3
    inputs:
      - remap_s3_logs
    bucket: my-bucket-name
    batch:
      max_events: 1000000
      timeout_secs: 300
    encoding:
      codec: raw_message
    filename_append_uuid: false
    filename_time_format: "%+"
    key_prefix: "{{ .container_name }}/%Y-%m-%d/{{ .container_name }}-{{ .image_tag }}-"

Everything seems to work except the batch config. I am seeing very small files (~3000 lines, ~50kB).

Documentation seems to suggest that max_bytes is optional, so I didn't set that.

And from what I'm reading, the buffer size should be independent of the batch size, so I didn't configure anything there either.

Am I missing something?

Edit: Setting max_bytes: 500000000 resolved this for me, though docs claim this is optional.

[pront]

Hi @extremeandy, at a first glance your config looks good.

Setting batch.timeout_secs is a valid way to trigger a flush, see (docs here).

How many events reach your s3 sink within a 30 sec interval?
Just a guess, maybe the rate is low and there's not much data to add to the batch.

---

Documentation seems to suggest that max_bytes is optional, so I didn't set that.

You are correct that max_bytes is an optional setting, None by default. This is another way to trigger flushing a batch when it reaches a certain byte size.

And from what I'm reading, the buffer size should be independent of the batch size,

Correct. In a sense that batching determines how we bundle bytes into requests before sending telemetry downstream. However, it's worth keeping an eye on the buffer metrics to see if backpressure building up. Which might not be the case here.

[extremeandy]

Setting batch.timeout_secs is another way to trigger a flush, see (docs here).

That's what I've got configured above, right? 300 seconds == 5 mins.

How many events reach your s3 sink within a 30 sec interval? Just a guess, maybe the rate is low and there's not much data to add to the batch.

Seems like around ~3000.

You are correct that max_bytes is an optional setting, None by default. This is another way to trigger flushing a batch when it reaches a certain byte size.

So I tried setting this anyway (set to 500000000 = 500MB) and it now it is adhering to my 300s flush time. Perhaps max_bytes isn't as optional as the docs claim?

I checked the file sizes when it was flushing too early and they seem to be all roughly 1.1MB.

[pront]

Setting batch.timeout_secs is another way to trigger a flush, see (docs here).

That's what I've got configured above, right? 300 seconds == 5 mins.

Yes 👍. I just wanted to link to https://vector.dev/docs/reference/configuration/sinks/aws_s3/#buffers-and-batches.

Just to better understand the situation, specifically how many buckets we expect here and roughly how many bytes per bucket. You are getting ~3000 events/ 30 secs. Do you happen to have a rough size estimation of each events?

Also, worth checking out the issue Jesse mentioned: #21696 (comment)

[extremeandy]

Hmm, I think I was getting ~3000 events every minute or two. The flush appeared to happen at the 1.1MB mark, not per event count or time. I guess that is an average of 366 bytes/event.

Perhaps #21696 (comment) is related yeah.

But anyway since I was able to fix by setting max_bytes I am not going to follow up further but please let me know if I can provide any more info or you want me to change any settings to give you more data.

[jszwedko]

This may be related to #21696 🤔