implement path migration

[kazuho]

Design:

• We issue 4 CIDs and therefore support 4 paths.
• conn->paths[0] is the path that will always exist, this path is always validation-complete once the handshake is complete and is the only path on which non-probing packets are sent.
• paths[1] to paths[3] are backup paths.
• When receiving a packet (regardless of it being probing or non-probing) from a new path, we pick the least-recently-used backup path that is not currently validating, and set its path to that of the new path, then start path validation.
• When receiving a packet on one of the backup paths, we consult if we have received non-probing packets on that backup path and if the path is validation-complete. If both conditions are met, this backup path is promoted to path[0].

Notes:

• At the moment, loss detection and congestion controller are not reset when migrating. We will address the issue as part of adding support for multipath.

Config knobs:

• max_probe_packets - maximum number of probe packets to send before calling a path unreachable (default: 5)
• max_path_validation_failures - once path validation fails for the specified number of paths, packets arriving on new tuples will be dropped (default: 100)

Connection stats:

• num_paths.created - number of paths created
• num_paths.validated - number of paths that were validated
• num_paths.validation_failed - number of paths that failed to validate
• num_paths.migration_elicited - number of paths on which migration was elicited (i.e., non-probing packets were received)
• num_paths.promoted - number of paths that were promoted (for sending data)
• num_paths.closed_no_dcid - number of paths that were closed due to Connection ID being unavailable
• num_packets.sent_promoted_paths - number of packets sent on any promoted path
• num_packets.ack_received_promoted_paths - number of packets being acked among ones that were sent on any promoted path

[kazuho]

ToDo:

• Add test case that involves path migration failures; e.g., set max_path_validation_failures to zero?
• Consider CC behavior upon migration.

As to the second point, when a server receives a HTTP/3 request after quiescence with NAT rebinding involved, the server will send a HTTP/3 response to the old address while sending a probe to the new address. As the client has migrated to the new address, the packet carrying HTTP/3 response gets lost. This would be the most popular pattern of path migration and we are going to see negative effect on HTTP/3 throughput, unless we need to do something.

Regarding what to do, for CC, one way of moving forward would be to reset whenever the path is migrated, potentially adopting the send rate on the old path as the jumpstart rate on the new path. Also, we might look into flushing the sentmap of the old path immediately when the new path is validated. Doing so allows the retransmission logic to kick in at an earlier moment.