Add exponential backoff to websocket client

[bgins]

General Description

Implement exponential backoff for the websocket client.

Which system(s) or functionality does this affect

This change affects:

• Job Creation
• Resource Provider
• Mediation
• Solver

Describe the changes, and how this affects/ interacts with each system.

Our existing websocket client has a retry strategy where it attempts to connect every two seconds:

lilypad/pkg/http/websocket_client.go

Lines 17 to 29 in 769deb7

	connectFactory := func() *websocket.Conn {
	for {
	log.Debug().Msgf("WebSocket connection connecting: %s", url)
	conn, _, err := websocket.DefaultDialer.Dial(url, nil)
	if err != nil {
	log.Error().Msgf("WebSocket connection failed: %s\nReconnecting in 2 seconds...", err)
	time.Sleep(2 * time.Second)
	continue
	}
	conn.SetPongHandler(nil)
	return conn
	}
	}

The repeated connection attempts can flood the solver with connection attempts at scale.

Let's implement exponential backoff when making the connection. We could start with a base interval of one second and an exponent of 1.5. Using an exponential backoff calculator, this would look like:

In addition, we should set a maximum number of attempts before giving up and exiting the program. We should exit with a clear error message so the user can report an issue connecting to us.

---

The job creator, resource provider, and mediator connect to the solver over a websocket. This feature impacts the communication between each client and the solver.