Delete routing table gracefully

[axot]

We observed connection time out in our client-side when scaling in or performing a rolling update of MIGs,

The workflow is,
Client -> ILB -> MIGs instance.

Before ACPI G2 soft off signal received[1], the forward ip(IP of ILB) was already deleted in routing table. This caused these long connections will be a timeout.

[1]

Related source code,
https://github.com/GoogleCloudPlatform/guest-agent/blob/master/google_guest_agent/addresses.go#L407

Is there any way to perform this more graceful? Any advice will very helpful.

[axot]

Hi, @gaohannk @hopkiw, could you take a look for this case?

[hopkiw]

what does 'more graceful' mean? i don't follow your request at all

[axot]

Hi @hopkiw, thank you for follow up this issue,

Because of forwarded ip in routing table will be deleted before shutdown-script,
this impact these active connections, we designed to send FIN or RST in our shutdown-script logic,
but these packets could not be arrived to client-side.

My opinion is, if google want to delete forwarded ip in routing table before shutdown-script,
it should send RST packet to all active connections, or let customer to handler these connections
in shutdown-script, does it make sense to you?

Thanks.

[hopkiw]

We don't delete routes unless they've been removed from the metadata server. Does pressing the 'power button' cause routes to be deleted from the metadata server? I will attempt to reproduce this.

The guest agent does not have any way to issue RST to destination of existing socket connections owned by other processes on the system.

[axot]

We found this behavior will happen with the following setups.

1. Create MIGs.
2. Add a L4 TCP ILB in front of MIGs with connection draining enabled. eg: 60 seconds timeout.
3. Establish long tcp connections from client to MIGs instances via ILB.
4. Perform scale down on MIGs, eg: 3 to 2 instances.

You will see forwarded IP(ILB's IP) will be deleted before 'power button' event.

If you test press the 'power button' directly on an instance this issue will not happen.

[axot]

Is there any workaround to improve this behavior?
How about disable network_enabled once the agent has been completed to configure the network?

[hopkiw]

MIG scale down sends the usual instances.terminate signal; the guest agent shuts down first, prior to running shutdown scripts. so i ran a shutdown script that shows the routes still existed after the shutdown of the guest agent.

[axot]

Thank you for the investigation, one thing I want to confirm is did you setup a L4 ILB in front of your MIG instance with connection draining enabled?