How to implement graceful shutdown of server with automatic client reconnection

[giovanni-bertoncelli]

Question

I was wondering if there is any way to implement a graceful shutdown of socket.io and at the same time guarantee the client the chance to retry automatically (without any client custom implementation) the reconnection.
My use case is:

• I have potentially multiple replicas with a IO server each (with redis adapter)
• My clients uses the reconnect: true option
• I scale down one of my replicas: I want the clients currently connected to that replica to automatically try to reconnect to another replica.

I'd like to know if there is any way to implement this, ideally also with the Connection Recovery feature.

What I've already tried

fastify.addHook('preClose', () => {
    await new Promise(res => fastify.io.close(res))
})

• I noticed that stragely, in the io.close() method (or the io.local.disconnectSockets() one), there is no way to specify whenever the clients that receive the disconnection event should retry to reconnect (even if the transport is closed)
• Another similar observation: there is no way to customize the disconnect event reason, so in this case the clients receive always a transport close disconnect reason, that is actually fatal.

[tusharrusia]

Hi @giovanni-bertoncelli , did you find an answer to your problem ?

I am stuck in a similar situation. I have a socket connection between my server which is running in AWS ASG group with EC2 instances and frontend portal running separately. Now the problem is the socket connection disconnect when the any instance downscale and then reconnects to some other instance immediately due to socket reconnection mechanism. Now the problem is I have to update some fields of the user if the socket disconnection is initiated by the user in the frontend portal and basically I was listening to ' disconnect ' event in backend for that. But now when the instance downscale the same fields are updated, which I want to prevent.

I went through the disconnect reasons in socket.io documentation and felt that instance downscaling may come under the disconnect reason as 'server shutting down' but when I tested it the reason came to be 'transport close' which is the same reason when the tab is closed in frontend and therefore I am not able to differentiate between the two cases.

[tusharrusia]

Checked for SIGINT also, didn't received it. I am running node directly using node index.js
Here is a basic template that I am running:

const express = require('express');
const axios = require('axios');

const app = express();
const port = 5000;

app.get('/', async (req, res) => {
  try {
    const response = await axios.post(
      'https://webhook.site/c33a45bb-5e80-4f77-b8ac-891d3c433378',
    );

    res.send(
      `Status: ${response.status}, Data: ${JSON.stringify(response.data)}`,
    );
  } catch (error) {
    console.error('Error making POST request:', error);
    res.status(500).send('Error making POST request');
  }
});

const server = app.listen(port, () => {
  console.log(`Example app listening at http://localhost/:${port}`);
});

process.on('SIGINT', async () => {
   await axios.post(
      'https://webhook.site/c33a45bb-5e80-4f77-b8ac-891d3c433378',
    );
  console.log('SIGINT signal received: closing HTTP server');
  server.close(() => {
    console.log('HTTP server closed');
  });
});

[giovanni-bertoncelli]

Try running the Dockerfile with tini

[tusharrusia]

I don't use docker, SIGINT signal should be received irrespective of using docker, correct ? When instance is downscaled it should send this signal to all process IDs ?

[darrachequesne]

@tusharrusia server.close() is not sufficient, as it won't close ongoing requests (like long-polling requests or WebSocket).

Reference: https://nodejs.org/api/http.html#serverclosecallback

You need to call io.close() (on the Socket.IO server).

Reference: https://socket.io/docs/v4/server-api/#serverclosecallback

[tusharrusia]

@darrachequesne thanks for pointing out that, but the main problem I am not able to solve is to know inside the code that the instance is going to terminate by listening to some process signal.