hitting networking issues when scaling network beyond 100 nodes

[gdsoumya]

I am trying to scale my libp2p node to see how large of a network I can support. I am using libp2p for a simple dkg protocol. For normal test with smaller network size <=100 it works just fine but as soon as I start scaling to 200 or more peers the nodes starts throwing errors when attempting to connect to peers. The errors are mostly with i/o timeouts and failing dials to peers, some times I also get security protocol specific errors. Here are some for the example errors I have seen on the node trying to establish connection to another node:

1. failed to dial: failed to dial 16Uiuxxxxxh: all dials failed
2. failed to negotiate security protocol: error reading handshake message: noise: message is too short

The 2nd error was also coming for TLS too when I was using TLS instead.

The above errors relate to this line of the code :

if err := d.host.Connect(ctx, *peerinfo); err != nil {...}

And in some cases when the connection does succeed I get errors opening a stream NewStream() : timed out: context deadline exceeded

I have increased all possible limits as far as I could find in documentation for libp2p, here's my node config :

listenAddrs := libp2p.ListenAddrStrings(fmt.Sprintf("/ip4/0.0.0.0/tcp/%v", cfg.ListenPort))
	transports := libp2p.ChainOptions(
		libp2p.Transport(tcp.NewTCPTransport),
	)

	yamuxTransportCfg := *yamux.DefaultTransport
	
        // overrides to increase limits and timeouts
	yamuxTransportCfg.ConnectionWriteTimeout = 2 * time.Minute
	yamuxTransportCfg.LogOutput = os.Stdout
	yamuxTransportCfg.EnableKeepAlive = true
	yamuxTransportCfg.KeepAliveInterval = 10 * time.Second
	muxers := libp2p.Muxer(yamux.ID, &yamuxTransportCfg)

	// facing this issue https://github.com/ipfs/kubo/issues/7516 with tls so preferring noise
	security := libp2p.ChainOptions(
		libp2p.Security(noise.ID, noise.New),
		libp2p.Security(libp2pTLS.ID, libp2pTLS.New),
	)

	rcmgr.MustRegisterWith(prometheus.DefaultRegisterer)

	str, err := rcmgr.NewStatsTraceReporter()
	if err != nil {
		return nil, err
	}

        // increased watermarks
	connManager, err := connmgr.NewConnManager(1000, 2000)
	if err != nil {
		return nil, err
	}

	// set resource mngr limits to infinite
	rcm, err := rcmgr.NewResourceManager(rcmgr.NewFixedLimiter(rcmgr.InfiniteLimits), rcmgr.WithTraceReporter(str))
	if err != nil {
		return nil, err
	}
	node, err := libp2p.New(
		libp2p.Identity(cfg.PrivKey),
		transports,
		listenAddrs,
		muxers,
		security,
		libp2p.ResourceManager(rcm),
		libp2p.ConnectionManager(connManager),
		libp2p.PrometheusRegisterer(prometheus.DefaultRegisterer),
	)
	if err != nil {
		return nil, err
	}

And here's the msg send logic :

func (d *Protocol) SendMessage(ctx context.Context, pid string, data []byte) error {
	info, ok := d.PeerRegistry.GetPeer(pid)
	if !ok {
		return fmt.Errorf("peer not found")
	}
	peerAddr, err := multiaddr.NewMultiaddr(fmt.Sprintf("%v/p2p/%v", info.RemoteAddr, pid))
	if err != nil {
		return err
	}
	peerinfo, err := peer.AddrInfoFromP2pAddr(peerAddr)
	if err != nil {
		return err
	}

       // attempting to ensure the connection isn't cleaned up while sending message
	d.host.ConnManager().Protect(peerinfo.ID, "smpc")
	defer d.host.ConnManager().Unprotect(peerinfo.ID, "smpc")

	// temporarily add to peerstore for connection
	if err := d.host.Connect(ctx, *peerinfo); err != nil {
		return fmt.Errorf("failed to connect to peer %v", err)
	}

	stream, err := d.host.NewStream(ctx, peerinfo.ID, protocol.ID(SmpcProtoMessage))
	if err != nil {
		return err
	}
	defer func() {
		if err := stream.Close(); err != nil {
			funcLogger.Error("failed to close stream:", err)
		}
	}()

	_ = stream.SetDeadline(getSendDeadLine(ctx))
	
	// Create a buffer stream for non-blocking read and write.
	rw := bufio.NewReader(stream)
	
	if err := d.Request(data, stream); err != nil {
		funcLogger.Debug("failed to send message", err.Error())
		return err
	}

	response := pb.ProtocolAcknowledgement{}
	if err := d.ParseResponseStream(&response, rw); err != nil {
		funcLogger.Debug("failed to send message ", err.Error())
		return err
	}

	if response.AckString != SmpcAck {
		funcLogger.Debug("failed to send message", response.AckString)
		return fmt.Errorf("did not receive ack, received %v", response.AckString)
	}
	return nil
}

Here's what I did on the infrastructure setup :

1. I made sure to set the ulimit on the actual machines to a pretty large value in case the system fd limits were causing issues.
2. I am running the tests on gcp and I upgraded the instances used to have larger cpu/memory limits and also bandwidth limits.
3. Each vm runs only 1 node process so there's no resource sharing involved.
4. Nodes are connecting over the public internet to replicate real world setup and are also geographically separated to some extent.
5. All nodes have public ips and ports exposed for libp2p so no need for relays or anything else, every node can be directly connected to.

For the dkg protocol itself the interactions between the peers are pretty simple :

1. at the start of the process each node directly pings every other node so for eg. in a network of 100 nodes each node connects to the rest 99 nodes and sends an initial package which is usually the largest package of the protocol. Each message is about 7kb of data.
2. once all nodes have sent and received the initial message from all other nodes it then sends 1 last message to every other node and finishes the protocol.

In case of smaller networks (<=100) all the above is more than enough and it works without any issues I don't even need to change the ulimits on the node nor do I need to set the limits for the resource mngr etc. for it to work. But once we scale to a 300 node setup the system just can't handle it even with all the increased limits, the nodes start to fail at the first step of the protocol itself, though not all nodes show the error only some of the nodes do about 30% of the nodes. I am guessing the node couldn't handle 300 connections simultaneously, I do not think it's an infra problem anymore as the vms are pretty beefy and so is the bandwidth. My best guess is that I am probably not doing something right in the code, I have already tried the obvious things like ensuring I am closing streams, ensuring contexts are set properly, added retry logic for connections etc. I also ensured the nodes are actually still accessibly by exposing a http server in the same node process and I was able to hit the http endpoint on the receiver node without issues from the sender node while the p2p connection kept failing.

Version Info :

github.com/libp2p/go-buffer-pool v0.1.0
github.com/libp2p/go-flow-metrics v0.1.0
github.com/libp2p/go-libp2p v0.36.2
github.com/libp2p/go-libp2p-asn-util v0.4.1
github.com/libp2p/go-libp2p-testing v0.12.0
github.com/libp2p/go-msgio v0.3.0
github.com/libp2p/go-nat v0.2.0
github.com/libp2p/go-netroute v0.2.1
github.com/libp2p/go-reuseport v0.4.0
github.com/libp2p/go-yamux/v4 v4.0.1
github.com/libp2p/zeroconf/v2 v2.2.0

[elecbug]

Can I get dial part code? maybe I help you.

[elecbug]

This answer is particularly to solvefailed to negotiate security protocol: error reading handshake message: noise: message is too short

[gdsoumya]

@elecbug I do have a wrapper around SendMessage to attempt for retries on failure which i guess should also handle the case you are pointing to? From my observation the node even after multiple retries with exponential backoffs never completes the connection. Also the peer it's trying to connect to I am observing the logs for that and I do not think it was trying to connect to this node at that time.

[elecbug]

Also one thought about the first question is, your network's IP range is set subnet mask to 24? I made a similar mistake before, but if you set the subnet too small, you might not be able to form more than 255 nodes (which is actually less than that).

[elecbug]

I do have a wrapper around SendMessage to attempt for retries on failure which i guess should also handle the case you are pointing to?

What I learned about this issue is also because I've asked questions in the past. I don't know if there's an answer you want here, but you should take a look.

[gdsoumya]

That's good point, but as I mentioned in the post all nodes were talking over the internet and I was able to ssh into each one of them with the public ip so I don't think that will be a problem. I was also running the nodes in different regions so the subnets were also different