Storage/RPC Related Resource Management Updates for Connectionless EPs #10837
+34
−17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
soumagne
approved these changes
Mar 3, 2025
j-xiong
reviewed
Mar 3, 2025
Previously, any resource management error would result in an endpoint being disabled. Storage libfabric users have the expectation that connectionless endpoints will not be disabled in an RDMA operation results in an error. For example, if a client sends a server a bad remote key and the server issues an RMA read/write to this key, by the current resource management definition, the endpoint will be disabled. RDMA failure to a single connectionless peer should not an impact endpoints connectively to other peers. Fix this by clarifying exactly which errors will cause a connectionless endpoint to transition into a disabled state. Signed-off-by: Ian Ziemba <[email protected]>
Resource management unreachable EP addresses the issue of issuing RDMA operations to a connectionless EP which cannot be reach. Such examples include no-route-to-host or target NIC down. Defining this behavior is important for storage use-cases where NICs may unexpectedly disappear. Signed-off-by: Ian Ziemba <[email protected]>
j-xiong
approved these changes
Mar 4, 2025
shefty
reviewed
Mar 4, 2025
| the endpoint must be re-enabled before it will accept new data transfer | ||
| operations. For connected endpoints, the connection is torn down and | ||
| must be re-established. | ||
| When a resource management error occurs on an connected endpoint, the endpoint |
| must be re-established. | ||
| When a resource management error occurs on an connected endpoint, the endpoint | ||
| will transition into a disabled state and the connection torn down. While | ||
| transitioning to disabled, any queued and inflight operations will be dropped. |
There was a problem hiding this comment.
A disabled endpoint will drop any queued or inflight operations."
| When a resource management error occurs on an connected endpoint, the endpoint | ||
| will transition into a disabled state and the connection torn down. While | ||
| transitioning to disabled, any queued and inflight operations will be dropped. | ||
| Connection must be re-established for endpoint to be usable. |
There was a problem hiding this comment.
I'd consider removing this sentence. It may not be possible to re-reconnect the same endpoint, forcing the use of a new one.
man/fi_domain.3.md
Outdated
| transitioning to disabled, any queued and inflight, local transmit operations | ||
| will be dropped. Endpoints targeting a disabled EP must adhere to the Target EP | ||
| behavior. If the endpoint becomes disabled, the endpoint must be re-enabled | ||
| before it will accept new data transfer operations. |
There was a problem hiding this comment.
The behavior of resource management errors on connectionless endpoints depends
on the type of error. If RM is disabled and one of the following errors
occur, the endpoint will be disabled: Tx Ctx, Rx Ctx, Tx CQ, or Rx CQ. For other errors
(Target EP, No Rx Buffer, etc.), the operation may fail, but the endpoint will remain enabled.
A disabled endpoint will drop or fail any queued or inflight operations. ...
| operations targeting an unreachable endpoint will have operation dropped. For | ||
| FI_EP_RDM, target operations targeting an unreachable endpoint will result in | ||
| a transmit error. A provider may choose to set the completion error code to | ||
| FI_EHOSTUNREACH signaling to user the target endpoint is unreachable. |
There was a problem hiding this comment.
I'd hesitate to call out a specific error code, as that depends on getting some sort of feedback from the network or target node to be accurate.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Storage libfabric users using a connectionless EP do not expect the EP to become disabled if an RDMA operation fails. The current libfabric documentation states that if various resource management enabled errors occur on an endpoint, the endpoint becomes disabled and must be re-enabled to be reusable. While this makes sense for a connected endpoint (e.g., TCP socket and IB RC QP), this does not make sense for connectionless endpoints. Consider the following RDM EP example:
The above shows that a single RPC client crashing can trigger server endpoint becoming disabled. This would impact RPC server connectivity to all other up RPC clients.
This PR addresses this problem by