separate Path IDs from Connection IDs #214
Comments
|
Marteen, The current version of the draft actually does away with the concept of path identifier, and only uses the connection ID. This is the result of about two years of discussions, during which several different concepts were tried. The basic problem is that identification of paths by four tuples does not work well, because of NAT. You never know when a NAT rebinding takes places, and that introduces lots of complexity when trying to use a "hard" notion of path identifiers. On path management: yes, there is a tie between life of a CID and life of the associated number space. This is explained in some detail in 7.1. Number Spaces. That section explains the "loose" definition of path identifiers that we converged on. The requirement to check that the peer has connection ID available is already there in RFC 9000. See for example the text in section 8.2: "Sending NEW_CONNECTION_ID and PATH_CHALLENGE frames in the same packet, if the peer's active_connection_id_limit permits, ensures that an unused connection ID will be available to the peer when sending a response." You do not actually need to tie multiple number spaces together for loss recovery: you can merely rely on timers after closing a number space. But you may want to optimize: I have in fact added optimization code in Picoquic, so that receiving ACKs of packets sent later on the same path triggers packet-number based loss detection. This is not very hard. |
Only using CIDs is what I'm objecting to. I believe this makes the design more complicated than it needs to be.
I'm not proposing to bind the Path ID to a 4-tuple. In my mental model, a NAT rebinding doesn't open a new path (although it might require re-verification of the path as described in RFC 9000, Section 9). The client has no way of know when a NAT rebinding will happen / has happened, so it won't be able to anything about it: It will just continue sending with the old CID. That old CID would still be bound to the old Path ID. In my proposal, the only thing that would open a new path is the client consciously making the decision to do so: by using a CID associated with a fresh Path ID (in the common case, on a new 4-tuple). Actually, I'd consider it a nice property that the server can now distinguish between a NAT rebinding (path ID stays constant) and initiation of a new path (new path ID is used).
It definitely breaks packet-number space loss recovery. I'm not sure how I'd retrofit this in my implementation without introducing a lot of complexity. I think we should strive for a design where no changes to the loss recovery logic are needed (other than initializing a new loss recovery context per path). |
|
The use of the CID derives from a desire to not change the packet format, and to keep the path validation the same as RFC 9000. One of the key properties is that there is only one encryption key at a given time. Yes, this requires some coordination between paths, and the current proposal might not be optimal. Your proposal amounts to an explicit setup of path, after extending the definition of connection identifiers. You are making implementation arguments, but that cuts both ways: what might simplify your implementation, will create additional complexities in implementations that have already implemented the current scheme. I would much rather leave the spec as is, and focus on the "key update" issue. The current spec is safe but heavy: do not rotate the key "again" unless rotation has been observed on all paths. I am pretty sure that we can find a less onerous way to do that. Let's resolve that as its own issue. |
Neither of those would be changed by the introduction of explicit path IDs.
I understand that this is the current proposal. This issue doesn't propose to change that, #215 does. We could decouple path IDs from CIDs without changing that property, and I've been arguing that this would be an improvement in itself. That's why I split it into two issues. That said, I don't see why having a single encryption key would an advantage. Initializing a new encryption context per path seems pretty clean to me (just like we initialize a new loss detection context per path), but let's keep that discussion on #215.
I couldn't disagree more. Many production-level stacks haven't even started working on their MPQUIC implementation, so optimizing the design to minimize the diff for the stacks that already implemented the draft current seems premature. I'd prefer to find the cleanest design, starting from RFC 9000. I also disagree that my argument is just an implementation argument. Saying that it shouldn't be possible (or making it super hard) to rotate CIDs on existing paths, making "Retire Prior To" practically useless, etc. are significant departure from RFC 9000. Arguably much larger than adding a Path ID varint to the NEW_CONNECTION_ID frame. |
|
As for loss recovery: the default solution will be to use timers, and only consider the packets of an old number space lost if no MP-ACK arrives before the timer expires. This worse than using PN-based loss detection, but it is not a bad default. In situations such as CID rotation or NAT rebinding, there are no packet losses and the MP-ACK does arrive in time. In abnormal situations such as abandoning a path because it does not work, losses are probably happening already and there should not be a lot of packets in transit. I am ready to bet that the performance impact will be minimal. |
|
Frankly, I don't think that this designs make it "super hard" to rotate CID on existing paths. Can you quantify the hardness? |
|
Marten, FWIW your proposal has actually the same basis as an old proposal that proposes to associate CIDs to an (internal) path identifier when proposing them to your peer. I think such a proposal is sensible, and I agree on the properties listed in the initial message. However, it won't be "as minimal as possible with regards to RFC9000" and might introduce additional complexity relative to the Path ID changes/rotations (i.e., closing some "paths" and opening new ones after)... This would thus require some consensus. In the meantime, I am open to review any proposal in order to evaluate "how hard" such a change would be. |
|
Pure question, but can Multipath be used on top of QUIC-aware UDP proxy without having the notion explicit path ID that is communicated over the wire? QUIC-aware UDP proxy allows the proxy to coalesce multiple connections onto one address-port. Let's say that a client speaking Multipath QUIC establishes a connection to a target (i.e., server) through a QUIC-aware UDP proxy. Then, the client opens a new path to the target through the same proxy. Now, from the client's point of view, there are two paths. But from the target's point of view, there will be only one path if the proxy chose to use the same local address / port for sending packets to the target. Speaking broadly, to me it seems that there's an assumption in the Multipath draft that if an endpoint opens a new path (i.e., by selecting a different 2-tuple locally), then that would be visible to the peer. Is it a good idea to have such an assumption and built a protocol that relies on that? cc @tfpauly. |
|
I think the proxy can open multiple sockets to the target in this case, and put the different paths over those different sockets. Those sockets can be reused for other mpquic connections as long as they always use different CIDs. However, having explicit CID mappings does make things simpler. |
|
@kazuho Yes, asking the proxy to "please send these packets from a new address" would require a new socket. But then, this feels like an extension to Masque, because Masque UDP proxies would typically have only one IP address. I think that if the client wants a real multipath scenario, it will have to route packets through several proxies. That, or writing an extension to Masque in which the proxy announces availability of several outgoing paths, and let the clients select which one they want for a particular UDP socket. |
|
@tfpauly @huitema Thank you for your comments. I think we are in agreement that the issue can be fixed on the Masque-side rather than communicating path IDs in the wire-protocol of Multipath QUIC. At the same time, I agree with @tfpauly that for clients that implement both Multipath QUIC and QUIC-aware UDP proxy draft, having the concept of path ID would simplify things. I think this is a good input to this discussion. |
|
@kazuho it is good that we start thinking about Masque and multipath now. I think that when it addresses multipath, the Masque WG will have to consider at least three scenarios:
|
|
@marten-seemann, this is a very good proposal! It is well argued and solves explicit problems as well as reduces complexity in a clean and simple way. |
|
I worked on a proof-of-concept integration of the above idea in my implementation, so I have a better idea of the overall proposal now. In the current state, I looked at a notion of symmetric Path ID, given that paths must be bidirectional per RFC9000. Overall, I agree with all the properties in the initial message, and note some additional properties. A nice part of this proposal is that it decouples the notion of "network paths" (i.e., the 4-tuple, having some validation state,...) from "logical QUIC paths" and "application packet number spaces" (i.e., acknowledgments, number of concurrent active paths, recovery states, active/standby status,...). Advertising in advance using MP_CONNECTION_ID frames to which "Path ID" the provided CID simplifies the "path handling" when the CID used over a same 4-tuple changes (as they belong to the same packet number space, unlike the current state of the draft). Another nice property relates to the "connection migration". The current draft "overrides" the handling defined by RFC9000 to let the server use any available network path. This means that if a client probes an network path just to check possible connectivity, it opens the door to that network path usage by the server, which may not always be wanted (the client can still advertise a Standby status over that path with PATH_STATUS/PATH_STANDBY, but this is only a scheduling suggestion, the client has no way to enforce this at server-side). With the Path ID proposal, the connection migration mechanism remains on a per-Path ID basis, enabling clients to probe a network path (using probing packets) without having the server moving the traffic towards that probed network path. Furthermore, basing the “path abandon” mechanism on such Path ID sounds more robust than identifying paths through CIDs. A “PATH_CLOSE” frame could be defined and would force retiring all the CIDs associated to that Path ID and hence control the number of simultaneous network paths in use for non-probing packets. For the advertisement of status of the paths, they would also remain explicit, even if the CID used over a path changes. Finally, using Path IDs would enable per-path keys, though as a first step we could stay with the current nonce adaptation and includes the Path ID in the nonce instead of the CID sequence number. Of course, this introduces some additional wiring (MP_NEW_CONNECTION_ID and MP_RETIRE_CONNECTION_ID frames, the active_path_limit TP that must be advertised by both peers) and the multipath extension would be "less minimal". But this approach surely has its advantages. |
|
Regarding rewiring some of the frames, I think we can avoid this. NEW_CONNECTION_ID frames carry information specific to a path. If we want to issue new CIDs for a path, we would surely do it on that same path. If for some reason we want to send them on another connection, we could tell the endpoint that the next frames refer to another path. And we would tell that with a new frame, which could be called INTENDED_PATH. For instance, we want to send CIDs 1234 and 5678 for path B on the path A, along with some data. The frames packed in the next packet on the path A would look as follows: INTENDED_PATH {Path ID: B} If we want to combine multiple extensions with MP, we might need to rewire all of their frames. Using INTENDED_PATH frame, we won’t. |
|
I don't really understand why defining a new frame type is such a big deal. Frame types are cheap, we have 2^62 of them.
This proposal on the other hand would be a major inconsistency with RFC 9000's frame semantics. RFC 9000 is very careful to 1. not assign any meaning to the order that frames occur in a packet and 2. is explicitly designed such that every frame can be interpreted on its own. |
|
Having almost implemented Multipath draft as-is, I think I share the view with what @huitema says in #214 (comment). To me it path IDs seem to be an complexity that might not be necessary. The complexities that I'm concerned include:
Compared to these, I do not think there is a lot of benefit in introducing Path IDs. Sure, we might no longer need to iterate (or do a hash lookup of) certain structures, but we have to iterate others anyways. As we have multiple packet number spaces, we have to do a lookup of ack queue (when receiving packets) and also a lookup of loss recovery state (when receiving acks). Considering that these N:1 mapping is going to exist anyways, I'm not sure if getting rid of only some is worth having more diversion from QUIC v1. As stated in #50 (comment), I tend to think that QUIC v1 already is a multipath protocol with the exception that only one path can be used for sending data at a time; my preference seems to go to just fixing that problem rather than introducing a new concept of having multiple path-groups with each group having only one active path. |
@kazuho, I'm not concerned about hash table lookup latencies. I agree that (1) and (3) would go away iff you never rotate CIDs on an existing path. While rotating CIDs can be seen as an anti-ossification measure, I think I remember that @nibanks is using Retire Prior To field in the NEW_CONNECTION_ID frame with his RSS implementation in MsQuic. It seems like this feature of RFC 9000 is effectively rendered useless, unless one wants to risk running into the corner cases described in section 4.3.3 around having established paths that don't have an associated CID (and therefore can't be addressed by PATH_* frames). I haven't implemented it yet in quic-go, but I imagine that this corner case would be rather annoying to deal with. (3) introduces a very big diff to RFC 9000: Loss recovery for the two packet number spaces created by rotating the CID on a path seems pretty complicated: You'd now either have to declare the packet sent with the old CID lost if it's not acknowledged, or implement loss recovery that spans the two packet number spaces. It's probably doable, but it's not trivial. It's also fundamentally different from QUIC v1, where rotating a CID on an existing path doesn't create a new packet number space.
You're right, you'd need to allocate a tiny bit more space for CIDs. I don't expect that there are many situations where you'd have more than 5 paths (that already seems quite a lot), and you don't need more than 2 or 3 CIDs per path, even if you want to allow for frequent rotation of CIDs. That's not a lot of state. |
|
@marten-seemann Thank you for sharing your thoughts.
So I think i disagree with this view. FWIW, we also rotate CID encryption keys, I assume every server that support clustering (load balancing) does. But that does not happen frequently, I believe that is the same for others as well. Therefore, problem (1) is IMO not an issue in practice. Note that, in order to handle packets arriving late, an endpoint has to be capable for receiving packets with CIDs that it has issued in the past for at least 3 PTO. Therefore, even in the case of an endpoint rotating CIDs at full speed, there will be high chance of PATH_STATUS getting through. Re (3), when changing the CID of an existing path, all an endpoint needs to do is remember the delta (base offset) between the packet number sent on the wire and that being remembered by the loss recovery logic / congestion controller. To give an example, if an endpoint has sent using CID=X packets up to PN=1000, and wants to switch CID to Y, all it needs to remember is an offset of 1001. The first packet sent with CID=Y will carry PN=0, but internally, it would be handled as PN=1001 by adding the offset. The remaining one is (2), but I do not consider that as an issue. In practice, a new CID is used when either of the following happens: i) a client tries to use a new path, ii) a server sends a packet to a new client address. To support these two cases, all that have to be done by the endpoints are provide as many as active_connection_id_limit CIDs, and by the clients to always keep one spare CID or two (see #221). Note that while QUIC v1 does not prohibit endpoints rotating CIDs for no reason, endpoints behaving as such risk themselves of losing CIDs, as the peer might run out of CIDs and stop providing more. PS.
The amount of state that you need to retain depends on what you want to do e.g., in response to an apparent rebinding. As discussed in #50 (comment), for a server, it makes sense to bundle data in a probe sent in response to an apparent rebinding. In a request-response protocol, doing so saves an RTT. But to do so, you need more state per CID not to mention the complexity of reusing CCs across multiple paths; things are much easier with current design that allows a server to start using a new new packet number space for a new path that the server observes (rather than letting the proposed "path ID" approach that lets only the client designate packet spaces). |
|
The problem with the explicit path ID approach is that you can't guarantee in all cases that the client and server have the same view of the open paths. The complicated case is when the client changes the CID and at the same time a NAT rebinding happens. This seems like a corner case but it's completely not unlikely because NAT rebinding usually happen after some idle time and that might also be the reason for the client to re-start sending with a fresh CID. This was also discussed at length in issue #169 We need to address this case and that makes the explicit path ID approach complicated. Having an explicit path ID seems logical easier and for sure it make sense for the mental model, however, encoding it in the protocol design means actually additional complexity. Regarding your issue list @marten-seemann: Note that we need to resign PATH_STATUS anyway (see issue #186). However, first, PATH_STATUS is only a recommendation. If you don't have PATH_STATUS information about a used CID, that doesn't automatically mean that you should use it. So If you see a CID change on a path, I don't think this should automatically change your scheduling. The other option is in the loose path model that you could even send PATH_STATUS information for issued but not yet-used CIDs in advance and thereby per-set that information. Or if you could also send the PATH_STATUS frame in the packet with a new CID or even repeat it multiple times if you really want to be sure. Again, we need to redesign PATH_STATUS and we need to consider the issue you describe; this is a known issue that wasn't fully resolved with the last revision. You second point is discussed in issue #205 and it is incorrect to have a MUST requirement for the availability of CIDs on the other end. This is something you can't guarantee but this is already the case in RFC9000. So I think it's rather an editorial issue to remove this requirement from the current draft. Regarding loss recovery, I think @huitema answer this point already. The expectation is that you do not do loss recovery between to packet number spaces. If the CID and therefore the packet number changes and a loss might happen in that RTT, then recovery might be slower as you have to wait for a time-out. However, that's such a rare case that I wouldn't recommend to optimise it because of the complexity you describe above. |
I don’t see any problem here, if you say that a NAT rebinding doesn’t change the path ID. This is consistent with RFC 9000 Section 9.5, which states that when a server receives a packet with the same CID from a different source address (which would be the effect of a NAT rebinding), it MAY continue using the same CID for responses. There’s also no problem when the client rotated CIDs at the same time the NAT rebinding happens. Since in my proposal, CIDs are scoped to the path, there’s no ambiguity. It’s actually advantageous for the server to being able to distinguish between intentional migrations and NAT rebindings (which is only possible with my proposal), since it allows the server to retain congestion control state (Section 9.4 of RFC 9000) if the NAT only changed the port number. |
|
Okay, if you issue your CIDs per path that would address the problem. We didn't considering this option in the previous discussion, as we didn't want to change the CID management of RFC9000. Changing that will add some complexity (probably not that much but anyway), so the question is: is that complexity worth any potential benefits? As explained above, I don't think the three issues you mention are that big of a problem. |
|
I agree that adding a field to the CID management frames of RFC 9000 adds a little bit of complexity (but then again, modifying the frame parsing code is trivial and super easy to write tests for). We do however gain some simplifications, some of them significant. After the discussion we've had so far, I believe the following list summarizes the advantages we'd gain from adopting path IDs.
|
|
From my implementation experience, I very much agree with @marten-seemann and @qdeconinck. Ericsson's in-house MP-QUIC implementation, called Rask, is designed from start for multipath. One of the most important things when structuring the stack is to clearly define what is per-path and what is at the common connection level, and I think that Marten has hit it more or less spot on! The path "concept" also needs to be well defined, and the path "identity" (with a single identifier) should be stable and easy to reason about also through NAT rebindings and CID updates--just like in Marten's design. Also, I think that it's trivial to add new frame types and it is much better to be explicit in signalling than to add new, different semantics to existing frames with the misdirected argumentation that it would "reduce the difference with RFC 9000". I have had plans to add support for Marten's design to Rask, but didn't come around to do it partially due to the lack of another implementation to interop test with. Now that Quentin has an experimental implementation, it would be interesting to do the (rather limited) work to add support also to Rask. Unfortunately, it will not happen before the IETF meeting. |
|
The challenge with introducing a separate Path ID is that it requires a fundamentally stable path definition to begin with. Given the inherently unstable nature of the current 4-tuple path definition, it remains unclear how suddenly we can now achieve a stable path identification. The latest draft, which moves away from the concept of a path ID, seems to align better with reality by accepting the unstable nature of path definitions. When there’s a change in the 4-tuple, we simply rotate the Connection IDs (CIDs) and shift the packet number space. As outlined in RFC 9002, loss recovery can be done on a per-packet-number-space basis. It’s also important to recognize that events like NAT rebindings and CID rotations are infrequent events in real-world scenarios. Even in vehicular communications that use multi-path, frequent NAT rebindings are not typically observed. From a performance perspective, I would argue our primary concerns should be with scheduling and redundancy rather than the infrequent changes in path characteristics. |
|
We definitely need a stable path definition! In the end, a path is a physical entity where packets flow. Paths have different properties, and the endpoints must be able to understand and reason about which physical path they are using. The path properties include latency and packet loss, but also things like demands for energy or momentary power and economic cost. In a system setting, like a cellular network, also the competing devices' possibility to move load to other paths can be relevant. |
|
I came into an idea about how to fix the issue people have discussed a lot here in this issue. Actually there’re 3 part the CID/Path_ID design affects:
In these parts:
And I totally understand there is cost when we introduce a Path ID definition. My key point is: “Packet number space per Path”, is better than “per CID”. |
|
I very much agree with what is suggested above. The single packet number space per path is in practice a path identity. It is then very easy to attach a path identifier to that path. This path identifier will be unique and stable during the full lifetime of the path. It is looked up and used for signalling. A previous version of the multipath draft used the sequence number of the CID that was used to set up the path as path identifier, and that still looks like a good idea to me. For nonce generation, the CID sequence number can be used if that is seen as simpler than the path identifier (for our Rask implementation it doesn't matter). My key point is: One packet number space per path, as suggested above, is fundamental. Luckily, it very easily leads to a stable path identifier that can be used for signalling without the race conditions and misunderstandings that a changing signalling identifier implies. |
Could you elaborate how we would do that? In current form of QUIC v1 and Multipath, CIDs are issued by an endpoint and the receiver chooses which CID to use. Therefore, when the receiver switches to a new CID but continue using the same packet number space, the receiver has to send a signal indicating to which packet number space the new CID is associated. My vague recollection is that this additional signalling was considered bad and that led to what we have now. |
|
The crypto requirement is that all packets have a unique nonce. That is met by incorporating the CID sequence number in the nonce, and ensuring that the sequence number be unique for all packets with the same CID. You might want to use the same sequence number space for all CID in the same "logical space". You might even implement that as a single sequence number space for the entire set of path. That will work just fine, as long as you do not send more that 2^32 packets. But you have a problem if you have sent packet number 0x5FFFFFFFF on one path, and then rotate the CID and send packet 0x60000000 with the new CID. The sequence number will be set in the packet to at most 4 digits -- 0x00000000. This has to be expanded to the "correct" 64 bit value. Since this is a new CID, the receiver has no choice but to expand it to 64 bits of zeroes. And decryption will fail. |
|
If you want a path identifier independent of the CID and a number space "per path", then the packets can only be decrypted if the receiver knows the mapping from CID to number space identifier. That implies either changing the packet format to include an explicit number space identifier in the header, or having a protocol to negotiate that mapping before using the CID. We don't really want to add change the packet format. Adding a path identifier would increase overhead, and break format compatibility with RFC 9000. The path identifier would have to be encrypted, to prevent external observers to tie two successive identifiers to the same underlying connection. So, significant overhead, significant complexity. We could conceive a protocol in which parties negotiate in advance that CID number N is used for number space number P, but that too would be significant additional complexity. |
Considering a more rigorous mathematical framework, crafting a stable path ID is quite complex. A formal definition necessitates four components: (1) path attributes, denoted by A, (2) an identifier within a specific domain D, and (3) a mapping function F that assigns A to ID, such that ID = F(A). (4) a mapping function G that maps an ID to all the stateful information of a path. With the "loose path ID" model, the current draft adheres to RFC9000's method of defining a path via a 4-tuple. In this schema, the formal definition is straightforward: (1) the attribute A is the 4-tuple, (2) the ID corresponds to a value within the CID sequence number domain, (3) a mapping between the 4-tuple and the ID, and (4) a mapping between the ID and all the stateful information of a path are established. However, in cases requiring a stable path ID, we must reconsider what constitutes the attributes A. The 4-tuple alone is inadequate, necessitating additional context that encapsulates the notion of path continuity. Rigorously define the concept of continuity it is not straightforward. Furthermore, incorporating continuity into the definition of a path extends beyond the original scope set by RFC9000. |
|
Hmm, maybe we are talking past each other here, @yfmascgy. Your mapping analysis above, where the CID sequence number maps to "the stateful information of a path", looks good. The only thing that I suggest is that a stable unique identifier is added to this "stateful information of a path". This identifier would not be the "ID" but just added per-path metadata. When the path is referenced to in signalling, this path identifier is fetched from "the stateful information of a path" and used in the signalling frames. |
Assigning CIDs to paths before they are used is a simple and nice way for the peer to understand the (physical) path identity at a CID update. It also happens to be pretty much exactly how regular unipath QUIC handles CID updates, the endpoints have agreed on the semantics of the CIDs before they are used. RFC 9000 allows for multiple concurrent connections to use the same IP address and port and then the pre-agreed connection IDs are used to map the packet to a connection. This very issues where we are discussing is actually a suggestion from @marten-seemann for just such a design which I think is an excellent idea. |
|
To add another monkey wrench into the discussion ... Changing the packet number space semantics of RFC 9000 may make it difficult to use MPQUIC in combination with a QUIC-aware UDP proxy. If the proxy is operating in forwarding mode, an uplink QUIC short header packet received over a (virtual) CID on the network segment between the proxy and a client is mapped to a destination CID on the network segment between the proxy and a server. Nothing else in the QUIC packet is changed and parts of the QUIC packet header - including the packet number - are protected by a header protection key known only to the client and server. If MPQUIC is used between the proxy and client, and uni-path QUIC (RFC 9000) is used between the proxy and server, then a change in the path between proxy and client cannot affect the packet numbering. In other words, I think MPQUIC would need to preserve the packet numbering spaces defined by RFC 9000 and not introduce a new set of packet number spaces that would prevent interop with RFC 9000 compliant servers - i.e. both number space per CID and number space per path seem like non-starters to me. |
|
Let me give another example, one where MPQUIC might be used in a DETNET deployment. A client maintains multiple paths towards a packet elimination function (PEF) that has MPQUIC functionality and eliminates duplicate copies of received packets. The client may replicate a QUIC packet and send a different copy of the packet over each of the different paths. If the same packet numbering space is used across all paths, it is trivial for the PEF to identify and discard duplicate packets (and to perform reordering if necessary). In fact, I don't think anything other than basic RFC 9000/9002 functionality is required to do this. Again, I caution against changing the packet number space semantics of RFC 9000 by introducing either number space per CID or number space per path. |
|
@BillGageIETF In QUIC packets, the packet number field is encrypted. UDP proxies and NAT cannot decrypt it, and cannot do any kind of treatment based on packet numbers. I would not worry about that. Not only that, but the use of nonce in encryption guarantees that all packets are different. The PEF can only "eliminate duplicate copies" if there are actual duplicates, maybe caused by a malfunction by an on path agent. I would not worry about that either. And then, since the packet numbers are encrypted, the middleboxes have no basis to perform any reordering. So, no worry about that either. QUIC is specifically design to limit interference by middle-boxes, including PEF, because middleboxes like PEF cause ossification and prevent innovation. Because of the protections built in QUIC, we can freely innovate and develop multipath functions. |
I was describing a DETNET scenario where the PEF is a QUIC endpoint, not an intermediate node or middlebox. Packet replication over diverse paths is a common technique for applications requiring high(er) reliability and low(er) latency. In this case, replication of the QUIC packet occurs after encryption so they are, in fact, exact copies. The different copies would be encapsulated in different IP/UDP datagrams associated with different sockets, corresponding to the different paths established (using MPQUIC) between the client and the PEF. A single packet number space would greatly simplify implementation. |
|
@BillGageIETF in any case, what you are discussing here is not related to the issue #214 discussed above, which is about path identification. You should open a specific issue with your suggestion. |
The question here is how do you formally define "stableness"? What is the path attributes A? Given two path attributes A1 and A2, how do you decide if they are the same path or not? |
|
We must also remember one fundamental reason for using multipath: it's for seamless failover. The current draft is a minimally scoped extension, allowing for initial single-path communication akin to how QUICv1 operates. Should your path begin to falter, you can swiftly establish a new one with any available spare Connection ID (CID). Yet, if you assign CIDs to specific paths, this creates an extra dependency, necessitating extra information exchange on the initial path before a new one can be launched. If the original path is deteriorating, causing such information exchange to fail, this can impede the failover process. Experience has taught us that in a production environment, it's wiser to minimize dependencies to avoid complications and failures. |
|
During the meeting in Prague, the WG decided to study what it would take to change the draft in line with the "path-id" suggestion. I tried to write a draft describing a "path-id extension" that would allow use of a unique path id. I think I did a fair spec based on the ideas floating around, but hey, I could be wrong. The draft is:
My next step is to implement that in picoquic, to assess the implementation issues, perf, etc., and do a comparison with what we have. |
@huitema: I'm not sure it's so easy to separate the issues. Architecturally, there are two models for multi-path operations: model (A) is a collection of uni-path QUIC constructs while model (B) is a uni-path QUIC construct operating over a collection of paths. Model (A) is like MPTCP and appears to be the model of the current MPQUIC design. Model (B) is like a TCP connection operating over a layer 2 link aggregation group. In (B), a TCP segment can be transmitted in an IP datagram over any of the links in the LAG. If we apply (B) principles to MPQUIC, then path management is distinct from connection management. Conceptually the MPQUIC stack is an RFC 9000 entity sitting on top of a path management entity with a shim entity between them to direct a QUIC packet over one of the available paths. In (B), a QUIC packet can be sent over any of the available (and unrestricted) paths. Since connection identifiers are independent of path, a QUIC packet received over any path is processed in the same way as a packet received over the single path construct of RFC 9000 - i.e. there is a single application data packet number space and an ACK received over any path is unambiguous (no need for MP-ACK). These are basically the principles used for connection migration in RFC 9000. The difference is that MPQUIC provides multiple paths that can be simultaneously active and path usage is explicitly managed. Clearly congestion control must be path-specific but connection management and packet loss recovery are not path-specific. In this model, it is hard to imagine a solution that does not include an explicit path identifier that is independent of connection identifiers. |
|
@BillGageIETF we already had the debate between single number space per connection versus number space per path. There are good arguments for each design, they were considered and debated at length -- see issue #96 for the debate; also see previous versions of the draft. If you want to reopen that debate, you need to do it in a separate issue because the arguments are unrelated to what we are discussing here. |
|
The discussion in this thread is whether there is value in separating path IDs from connection IDs. I am supporting Marteen's suggestion to introduce an explicit path identifier. I guess I am also suggesting benefits of this separation. With explicit path identifiers, you can manage paths without referencing connection identifiers. You can also manage paths separately from connections. With the latter, there is no need to split the application data packet number space. |
|
This document is in response to discussions of issue 214 in the QUIC
multipath GitHub: #214
The current MPQUIC draft (-06) binds a connection identifier to a path
by using the sequence number of a connection identifier as an implicit
path identifier. To simplify implementation, the current MPQUIC draft
introduces the concept of multiple application data packet number spaces
with a different number space for each connection (path). This is in
contrast to RFC9000 where there is a single application data packet
number space.
Issue 214 proposed separating path IDs from connection IDs. This
document uses that separation of identifiers to propose a different path
model for Multipath QUIC using explicit path identifiers, enabling a
multipath management framework that retains the principles and
operations of RFC9000.
The multipath operations described in this document do not change the
basic operations described in RFC9000. In particular, none of the
following procedures described in RFC9000 are affected by the use of
multiple paths:
+ connection management (e.g. the use of NEW_CONNECTION_ID frames and
subsequent rotation of connection identifiers);
+ key management (e.g. use of key phase bit) and derivation of AEAD
parameters;
+ packet loss detection and loss recovery (e.g. using type 0x02 ACK
frames without ECN counts).
However, changes to RFC9002 procedures are required to deal with
path-dependent characteristics such as path MTU size, RTT and congestion.
Cheers ...
/bill
On 2023-11-11 5:49 p.m., ***@***.*** wrote:
A new version of Internet-Draft draft-gage-quicmp-pathmodel-00.txt
has been
successfully submitted by Bill Gage and posted to the
IETF repository.
Name: draft-gage-quicmp-pathmodel
Revision: 00
Title: An Alternate Path Model for Multipath QUIC
Date: 2023-11-11
Group: Individual Submission
Pages: 14
URL:
https://www.ietf.org/archive/id/draft-gage-quicmp-pathmodel-00.txt
Status: https://datatracker.ietf.org/doc/draft-gage-quicmp-pathmodel/
HTML:
https://www.ietf.org/archive/id/draft-gage-quicmp-pathmodel-00.html
HTMLized:
https://datatracker.ietf.org/doc/html/draft-gage-quicmp-pathmodel
Abstract:
The path model used in the current MPQUIC draft binds a connection
identifier to a path. In fact, the sequence number of a connection
identifier is used as an implicit path identifier. This has a number
of consequences that may cause MPQUIC to diverge from the principles
of RFC9000. One of these consequences, for example, is to associate
each connection with a different application data packet number space
rather than maintaining a single application data packet number space
across all connections as defined in RFC9000.
This document proposes a different path model for Multipath QUIC
using explicit path identifiers, enabling a multipath management
framework that retains the principles and operations of RFC9000.
The IETF Secretariat
|
The current version of the draft introduces a very tight coupling between paths and connection IDs. This feels unnatural, since QUIC was designed such that CIDs can be rotated (potentially very) frequently, even in the absence of any path migration / probing.
The tight coupling in this draft leads to a whole range of problems:
I believe the design would become significantly easier if CIDs were scoped to the path to begin with:
active_connection_id_limittransport parameter could be reinterpreted to mean the maximum number of active connection IDs per path. It could be accompanied by aactive_path_limittransport parameter to allow the client to limit the number of paths.I believe this would resolve the problems described above. In addition, this has a few nice properties:
It has not escaped my attention that this proposal suggests another strategy for key management, which - among others - would simplify the key update problem. I'll follow up in a separate issue.
The text was updated successfully, but these errors were encountered: