From 3c34a87ae9eefd94f784e5f38b5b68842362073c Mon Sep 17 00:00:00 2001 From: Jianxin Xiong Date: Tue, 14 May 2024 18:13:07 -0700 Subject: [PATCH] core: Add endpoint options for max message size and inject size Today, `ep_attr->max_msg_size` and `tx_attr->inject_size` define the transport's size limitation for a single data transfer operation. It lacks the flexibility to allow providers having different limits on different categories of operations (msg/tagged/rma/atomic). At the same time, applications can't express usage models that have very different size limits (e.g, only need small send/recv, but want large RMA). Here a set of options are added to the fi_getopt()/fi_setopt() interface. These options allow the maximum message size and inject size be retrieved or set independently for msg/tagged/rma/atomci operations. The existing definition of `ep_attr->max_msg_size` and `tx_attr->inject_size` remain unchanged -- they are the upper bound of the limits for all the data transfer functions. A typical application will call fi_getopt() to retrieve the default values first, and call fi_setopt() if lower values are desired. If -FI_ENOPROTOOPT is returned, `ep_attr->max_msg_size` and `tx_attr->inject_size` should be used. Signed-off-by: Jianxin Xiong --- include/rdma/fi_endpoint.h | 8 +++++ man/fi_endpoint.3.md | 64 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/include/rdma/fi_endpoint.h b/include/rdma/fi_endpoint.h index ad2a9387dfe..d01ca770f40 100644 --- a/include/rdma/fi_endpoint.h +++ b/include/rdma/fi_endpoint.h @@ -70,6 +70,14 @@ enum { FI_OPT_XPU_TRIGGER, /* reserved for compatibility */ FI_OPT_CUDA_API_PERMITTED, /* bool */ FI_OPT_SHARED_MEMORY_PERMITTED, /* bool */ + FI_OPT_MAX_MSG_SIZE, /* size_t */ + FI_OPT_MAX_TAGGED_SIZE, /* size_t */ + FI_OPT_MAX_RMA_SIZE, /* size_t */ + FI_OPT_MAX_ATOMIC_SIZE, /* size_t */ + FI_OPT_INJECT_MSG_SIZE, /* size_t */ + FI_OPT_INJECT_TAGGED_SIZE, /* size_t */ + FI_OPT_INJECT_RMA_SIZE, /* size_t */ + FI_OPT_INJECT_ATOMIC_SIZE, /* size_t */ }; /* diff --git a/man/fi_endpoint.3.md b/man/fi_endpoint.3.md index 78ce6947e89..18b6798bee3 100644 --- a/man/fi_endpoint.3.md +++ b/man/fi_endpoint.3.md @@ -568,6 +568,70 @@ The following option levels and option names and parameters are defined. shared memory will not be used and the implementation of intra-node communication is provider dependent. +- *FI_OPT_MAX_MSG_SIZE - size_t * +: Define the maximum message size that can be transferred by the endpoint + in a single untagged message. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `ep_attr->max_msg_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `ep_attr->max_msg_size` should be used. + +- *FI_OPT_MAX_TAGGED_SIZE - size_t * +: Define the maximum message size that can be trnasferred by the endpoint + in a single tagged message. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `ep_attr->max_msg_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `ep_attr->max_msg_size` should be used. + +- *FI_OPT_MAX_RMA_SIZE - size_t * +: Define the maximum message size that can be transferred by the endpoint + via a single RMA operation. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `ep_attr->max_msg_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `ep_attr->max_msg_size` should be used. + +- *FI_OPT_MAX_ATOMIC_SIZE - size_t * +: Define the maximum data size that can be transferred by the endpoint + via a single atomic operation. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `ep_attr->max_msg_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `ep_attr->max_msg_size` should be used. + +- *FI_OPT_INJECT_SIZE - size_t * +: Define the maximum message size that can be injected by the endpoint + in a single untagged message. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `tx_attr->inject_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `tx_attr->inject_size` should be used. + +- *FI_OPT_TAGGED_INJECT_SIZE - size_t * +: Define the maximum message size that can be injected by the endpoint + in a single tagged message. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `tx_attr->inject_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `tx_attr->inject_size` should be used. + +- *FI_OPT_RMA_INJECT_SIZE - size_t * +: Define the maximum data size that can be injected by the endpoint + in a single RMA operation. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `tx_attr->inject_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `tx_attr->inject_size` should be used. + +- *FI_OPT_ATOMIC_INJECT_SIZE - size_t * +: Define the maximum data size that can be injected by the endpoint + in a single atomic operation. The size is limited by the endpoint's configuration + and the provider's capabilities, and must be less than or equal to + `tx_attr->inject_size`. + Providers that don't support this option will return -FI_ENOPROTOOPT. In that + case, `tx_attr->inject_size` should be used. + ## fi_tc_dscp_set This call converts a DSCP defined value into a libfabric traffic class value.