Source:file and any sink (?) redundancy options

[ADovgalyuk]

Hello team!
Can you please share your thoughts about redundancy options when I'm using file as a source. I had vector sink before, now switched to clickhouse. But issue still the same. As soon as one sink become unavailable, shortly after vector stoppin upload to second sink as well. Is it smth related to both sinks using the same checkpoint file as source? And when one sink goes to block, second sink can't transfer data as well? Currently I have a following configuration:

[sources.L3_cdr_file]
    type = "file"
    include = [
                "*.log",
                ]
    read_from = "beginning"
    ignore_not_found = true
    oldest_first = true

[transforms.L3_cdr_remap]
    type = "remap"
    inputs = [ "L3_cdr_file" ]
    source = """
        ., err = "L3;"+.message
            """



[sinks.L3_clickhouse_site1]
type = "clickhouse"
inputs = ["L3_cdr_remap"]
endpoint = "http://XX.XX.XXX.X:8123"
database = "statistics"
table = "L3_vector"
skip_unknown_fields = true
compression = "zstd"
auth.strategy = "basic"
auth.user = "vector"
auth.password = "vector"
        [sinks.L3_clickhouse_site1.buffer]
        type = "disk"
        max_size = 8192000000
        when_full = "block"

[sinks.L3_clickhouse_site2]
type = "clickhouse"
inputs = ["L3_cdr_remap"]
endpoint = "http://YY.YY.YYY.Y:8123"
database = "statistics"
table = "L3_vector"
skip_unknown_fields = true
compression = "zstd"
auth.strategy = "basic"
auth.user = "vector"
auth.password = "vector"
        [sinks.L3_clickhouse_site2.buffer]
        type = "disk"
        max_size = 8192000000
        when_full = "block"

How should i configure vector, that in case one sink is unavailable, second still works as usual.

[pront]

Hello @ADovgalyuk,

Your configuration creates two identical clickhouse sinks, but with different endpoints. Both sinks will receive the same set of events, and independently push them downstream. These sinks are independent. There's no order or any synchronization between them.

Do you see any errors?

Tangential note, it might be beneficial if we allowed multiple endpoints to be specified like here.

[jszwedko]

What you are seeing is expected behavior. Backpressure from one sink will affect all other sinks connected to the same inputs as it propagates up. See https://vector.dev/docs/about/concepts/#backpressure for details about how this works as well as pointers to how to use buffers to mitigate the effects of downtime in one sink from affecting others.

[jszwedko]

I see you already have buffers configured. Those should fill up first, before back-pressure is applied.