Connection timeout when creating 100 connections

[pabermod]

Hi,

I have an application that creates 100 connections at the same time in different threads. I have a retry for the connection opening. After 1 or 2 retries for each connection it ends up opening it. But i don't understand Why I have a connection timeout.

Here is the .net error:

Error connecting to host team1gij05. RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure ---> RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text='End of stream', classId=0, methodId=0, cause=System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.
at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer, ArrayPool1 pool, UInt32 maxMessageSize) at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame() at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration() at RabbitMQ.Client.Framing.Impl.Connection.MainLoop() at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout) at RabbitMQ.Client.Impl.ModelBase.ConnectionStartOk(IDictionary2 clientProperties, String mechanism, Byte[] response, String locale)
at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
--- End of inner exception stack trace ---
at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist)
at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
--- End of inner exception stack trace ---
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)

Log from rabbitmq for one of the connections:

2023-01-18 05:03:06.667000-08:00 [info] <0.2730.0> accepting AMQP connection <0.2730.0> (192.168.66.132:54932 -> 192.168.66.147:443)
2023-01-18 05:03:36.622000-08:00 [error] <0.2730.0> closing AMQP connection <0.2730.0> (192.168.66.132:54932 -> 192.168.66.147:443):
2023-01-18 05:03:36.622000-08:00 [error] <0.2730.0> {handshake_timeout,frame_header}

my current rabbitmqconfiguration after trying to modify some settings:

ssl_options.cacertfile = C:/Users/Administrator/Downloads/ca_certificate.pem
ssl_options.certfile = C:/Users/Administrator/Downloads/server/server_certificate.pem
ssl_options.keyfile = C:/Users/Administrator/Downloads/server/private_key.pem
ssl_options.password = mypassword
ssl_options.versions.1 = tlsv1.2

tcp_listen_options.send_timeout = 120000

tcp_listen_options.backlog = 4096
tcp_listen_options.nodelay = true

### PEER VERIFICATION
ssl_options.verify = verify_none
ssl_options.fail_if_no_peer_cert = false

### DISABLE NON-SSL CONNECTIONS
#listeners.tcp = none

### FLUSH CURRENT WAL FILE TO A SEGMETN FILE ON DISK ONCE IT REACHES 64 MiB IN SIZE
raft.wal_max_size_bytes = 64000000

### MEMORY THRESHOLD ADJUSTED TO 30% OF ALL
#vm_memory_high_watermark.relative = 0.3

#management.tcp.port = 1521
#channel_max = 10002

num_acceptors.tcp = 200
num_acceptors.ssl = 200

handshake_timeout = 30000```

Can anyone help me?

Regards

[michaelklishin]

{handshake_timeout,frame_header}

in the server logs means that the connected client did not send a protocol header appropriate for the port. Usually this is the case when a non-TLS client connects to a TLS port (or a TLS-enabled client connects to a non-TLS port but usually the error would be different).

[pabermod]

I'm opening each connection in a different Thread. Maybe I'm starving the threadpool?

[michaelklishin]

Sorry but the error in the logs is very specific and only logged in one case.

If you use the .NET client, IIRC there is a runtime limit on the number of tasks or something like that that you have to bump.

[pabermod]

It seems this is the issue. Anyway, shouldn't there be async methods in the rabbitmq .net client library for opening connection, channels, etc?

[michaelklishin]

@pabermod you haven't even mentioned what version of what client you use.

The .NET client is halfway migrated to async APIs but there are still limits in the runtime that can be exceeded with a lot of connections from a single process. Most applications do not need to open
100 or more connections, usually only load testing tools do that.

[pabermod]

I'm uisng version 6.4.0

I will try to modify my application to only open 1 connection and create 100 channels

[michaelklishin]

In other words, the root cause is very likely on the client or any proxy/intermediary side, not on the RabbitMQ node side.

[lukebakken]

@pabermod -

rabbitmq-dotnet-client/projects/TestApplications/MassPublish/Program.cs

Line 21 in 99a2b69

ThreadPool.SetMinThreads(16 * Environment.ProcessorCount, 16 * Environment.ProcessorCount);

Version 7 of this client will not have the issue you report. At this time, if you want 100 connections, you'll have to increase the thread pool size on application start.

[pabermod]

Do you have an estimated release date for version 7?

[michaelklishin]

It's unlikely to happen in the next month, maybe even two.

[pabermod]

Increasing the thread pool does the trick. Thank you both!

[lukebakken]

@pabermod if you have time to test version 7 alphas, it would be really appreciated

https://www.nuget.org/packages/RabbitMQ.Client/7.0.0-alpha.6

I'm hoping to get the final release done this month, but only if there is enough community testing (with reports back!)