Multiple connections per destination

[azhiltsov]

I am wondering is it hard to implement multiple connections per destination, so it would not try to push everything via one tcp connection. While using hashing can I use the same host:port:hash trio in order to do so?

[grobian]

You can't at the moment. I wonder why you feel this is necessary though?

[azhiltsov]

sending above 1M points/sec is hitting a limit on the receiver. Both carbon-c-relay and go-carbon barely cope with it via one connection. Multiple connections is the easiest way to fix it. As I understand each connection is processed in its own thread and this imposes the limit.

[grobian]

hmmm, and how would you like to control the amount of connections to use per destination?

[azhiltsov]

config parameter, defaulted to 1?

[ginal]

I am facing the same issue. Pushing millions of metrics per minute through a single tcp connection puts a lot of stress both at the receiving end (carbon-c-relays) and also at the load balancer that is in front of them (in my use case).
In previous versions (3.0 and 1.x) this could be achieved using a hack: by defining multiple hostnames for a single IP in the hosts file, and adding each of those aliases in carbon-c-relay.conf in an "any_of" destination, multiple connections would be opened to the same IP. In 3.2 this is fixed and carbon refuses to start throwing a config error ("cannot share server carbon1:2003 with any_of/failover cluster").

[jaroslawr]

I face the same issue, having one connection to a destination results in also having only one thread doing the write()'s to the connection, the thread saturates its CPU core and becomes a bottleneck. Since this is with both carbon-c-relay and destination server on same machine, I worked around it by doing:

cluster xyz
    any_of
        127.0.0.1:1234
        127.0.0.2:1234
        127.0.0.3:1234
        127.0.0.4:1234
    ;

Would be nice to simply have an option to spawn multiple connections/threads writing to the same destination like @azhiltsov suggested. I think the write() thread might also have lower than optimal throughput because it does a seperate write() for each single metric.