graceful shutdown

[yosiat]

Hi,

I wrote a redcon server and I need to support graceful shutdown, from what I observed calling "Close" only closes the server connection and don't wait for in-memory requests to flush.

Is there a recommended way to implement graceful shutdown?

[tidwall]

Hi, you're right. Calling Close only closes the network listener, which stop the server from accepting new connections.

But existing connection will continue to run until they're network socket has been closed.

The only way I can think of with the current implementation is to track the connections using the accept and close callbacks and to wait for the all connections to be closed using a WaitGroup. Also to use the SetIdleClose function to automatically close idle connections.

For example, here we'll create a new server, which will automatically close after 5 seconds.

func main() {
	// Create a new Server
	var wg sync.WaitGroup // connection wait group
	var closed int32      // atomic flag
	s := redcon.NewServer(":6380",
		// handler
		func(conn redcon.Conn, cmd redcon.Command) {
			if atomic.LoadInt32(&closed) != 0 {
				// server closed, close connection
				conn.Close()
			} else {
				// TODO: handle incoming command
				conn.WriteString("hello")
			}
		},
		// accept
		func(conn redcon.Conn) bool {
			if atomic.LoadInt32(&closed) != 0 {
				// Server closed, do not accept this connection
				return false
			}
			// Add connection to a wait group
			wg.Add(1)
			return true
		},
		// close
		func(conn redcon.Conn, err error) {
			// Remove connection from wait group
			wg.Done()
		},
	)
        // Set a max amount of time a connection can stay idle.
	s.SetIdleClose(time.Second * 10)
	go func() {
		// Close the server after an minute
		for i := 5; i > 0; i-- {
			println("closing in", i)
			time.Sleep(time.Second)

		}
		// Set the closed flag and wait for the connections to be closed.
		atomic.StoreInt32(&closed, 1)
		println("waiting for connections to close")
		wg.Wait()
		// No more live connections, close the server
		s.Close()
	}()
	if err := s.ListenAndServe(); err != nil {
		log.Fatal(err)
	}
	// The server is now closed
	println("closed")
}

[tidwall]

Now if you connect to the server using redis-cli within the 5 seconds. You will see

closing in 5
closing in 4
closing in 3
closing in 2
closing in 1
waiting for connections to close

a pause up to 10 second because of the SetIdleClose, then:

closed

[tidwall]

Ideally this logic would exist in the library. Maybe this is something that would work with #52

[yosiat]

@tidwall tested this locally and it works, thanks for the explanation and proposed solution!

Would be interested to see how graceful solution can be done with context, will follow to see that :)