From a09d82968f7a7e8cf26d10b823ec76cb1352b959 Mon Sep 17 00:00:00 2001 From: Ian Ziemba Date: Tue, 19 Sep 2023 12:17:59 -0500 Subject: [PATCH] man: Doc FI_AV_AUTH_KEY FI_AV_AUTH_KEY is used to enable multiple auth keys per connectionless endpoint. Signed-off-by: Ian Ziemba --- man/fi_av.3.md | 75 ++++++++++++++++++++++++++++++++++++++++++++ man/fi_cq.3.md | 5 +++ man/fi_domain.3.md | 10 +++++- man/fi_endpoint.3.md | 4 +++ man/fi_mr.3.md | 33 +++++++++++++++++++ 5 files changed, 126 insertions(+), 1 deletion(-) diff --git a/man/fi_av.3.md b/man/fi_av.3.md index 2c9c3eb7453..3b157977bf2 100644 --- a/man/fi_av.3.md +++ b/man/fi_av.3.md @@ -24,6 +24,12 @@ fi_av_lookup fi_av_straddr : Convert an address into a printable string. +fi_av_insert_auth_key +: Insert an authorization key into the address vector. + +fi_av_lookup_auth_key +: Retrieve an authorization key stored in the address vector. + # SYNOPSIS ```c @@ -58,6 +64,12 @@ fi_addr_t fi_rx_addr(fi_addr_t fi_addr, int rx_index, const char * fi_av_straddr(struct fid_av *av, const void *addr, char *buf, size_t *len); + +int fi_av_insert_auth_key(struct fid_av *av, const void *auth_key, + size_t auth_key_size, fi_addr_t *fi_addr, uint64_t flags); + +int fi_av_lookup_auth_key(struct fid_av *av, fi_addr_t addr, + void *auth_key, size_t *auth_key_size); ``` # ARGUMENTS @@ -97,6 +109,14 @@ const char * fi_av_straddr(struct fid_av *av, const void *addr, *flags* : Additional flags to apply to the operation. +*auth_key* +: Buffer containing authorization key to be inserted into the address + vector. + +*auth_key_size* +: On input, specifies size of auth_key buffer. On output, stores number + of bytes written to auth_key buffer. + # DESCRIPTION Address vectors are used to map higher-level addresses, which may be @@ -328,6 +348,16 @@ that calls to fi_av_insert following a call to fi_av_remove always reference a valid buffer in the fi_addr parameter. Otherwise it may be difficult to determine what the next assigned index will be. +If the address vector is configured with authorization keys, the fi_addr +parameter cannot be NULL and is used to define the authorization key associated +with the address being inserted. Acceptable inputs are the fi_addr_t's generated +from `fi_av_insert_auth_key`. Only `fi_addr[0]` can be used to pass in an +authorization key fi_addr_t. Return returned fi_addr_t's will map to endpoint +address against the specified authorization key. These fi_addr_t's can be +used as the target for local data transfer operations. If the endpoint +supports `FI_DIRECTED_RECV`, these fi_addr_t's can be used to restrict +receive buffers to a specific endpoint address and authorization key. + *flags* : The following flag may be passed to AV insertion calls: fi_av_insert, fi_av_insertsvc, or fi_av_insertsym. @@ -414,6 +444,11 @@ accessed. Inserted addresses are not required to be removed. fi_av_close will automatically cleanup any resources associated with addresses remaining in the AV when it is invoked. +If the address being removed came from `fi_av_insert_auth_key`, the address +will only be removed if all endpoints, which have been enabled against the +corresponding authorization key, have been closed. If all endpoints are not +closed, -FI_EBUSY will be returned. + Flags are reserved for future use and must be 0. ## fi_av_lookup @@ -450,6 +485,46 @@ size of the buffer needed to store the address. This size may be larger than the input len. If the provided buffer is too small, the results will be truncated. fi_av_straddr returns a pointer to buf. +## fi_av_insert_auth_key + +This function associates authorization keys with an address vector. This +requires the domain to be opened with `FI_AV_AUTH_KEY`. `FI_AV_AUTH_KEY` +enables endpoints and memory regions to be associated with authorization +keys from the address vector. This behavior enables a single endpoint +or memory region to be associated with multiple authorization keys. + +When endpoints or memory regions are enabled, they are configured with +address vector authorization keys at that point in time. Later authorization +key insertions will not propagate to already enabled endpoints and memory +regions. + +The `auth_key` and `auth_key_size` parameters are used to input the +authorization key into the address vector. The structure of the +authorization key is provider specific. + +The output of `fi_av_insert_auth_key` is a fi_addr_t handle representing +all endpoint addresses against this specific authorization key. For +endpoints enabled with FI_DIRECTED_RECV, authorization key fi_addr_t's +can be used to restrict incoming messages to only endpoint addresses +within the authorization key. + +These authorization key fi_addr_t's can later be used an input for +endpoint address insertion functions to generate an fi_addr_t for a +specific endpoint address and authorization key. + +Flags are reserved for future use and must be 0. + +## fi_av_lookup_auth_key + +This functions returns the authorization key associated with a fi_addr_t. +Acceptable fi_addr_t's input are the output of `fi_av_insert_auth_key` and +AV address insertion functions. The returned authorization key is in a +provider specific format. On input, the auth_key_size parameter should +indicate the size of the auth_key buffer. If the actual authorization key +is larger than what can fit into the buffer, it will be truncated. On +output, auth_key_size is set to the size of the buffer needed to store the +authorization key, which may be larger than the input value. + # NOTES An AV should only store a single instance of an address. diff --git a/man/fi_cq.3.md b/man/fi_cq.3.md index a990142cb08..23cf4899a70 100644 --- a/man/fi_cq.3.md +++ b/man/fi_cq.3.md @@ -448,6 +448,11 @@ Notable completion error codes are given below. passed directly into an fi_av_insert call to add the source address to the address vector. + For API versions 1.20 and later, if the EP is configured with + FI_AV_AUTH_KEY, src_addr will be set to the fi_addr_t authorization key + handle corresponding to the incoming data transfer. Otherwise, the + value will be set to FI_ADDR_NOTAVAIL. + ## fi_cq_signal The fi_cq_signal call will unblock any thread waiting in fi_cq_sread diff --git a/man/fi_domain.3.md b/man/fi_domain.3.md index 50c2e0fe176..a3639d30a76 100644 --- a/man/fi_domain.3.md +++ b/man/fi_domain.3.md @@ -743,6 +743,9 @@ The default authorization key to associate with endpoint and memory registrations created within the domain. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If domain auth_key_size is set to the value FI_AV_AUTH_KEY, domain auth_key +is ignored. + ## Default authorization key length (auth_key_size) The length in bytes of the default authorization key for the domain. If set to 0, @@ -751,6 +754,11 @@ registrations created within the domain unless specified in the endpoint or memory registration attributes. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If the size is set to the value FI_AV_AUTH_KEY, all endpoints and memory +regions will be configured to use authorization keys associated with the AV. +Providers which support authorization keys and connectionless endpoint must +support this option. + ## Max Error Data Size (max_err_data) : The maximum amount of error data, in bytes, that may be returned as part of a completion or event queue error. This value corresponds to the @@ -776,7 +784,7 @@ for additional information. ## Max Authorization Keys per Endpoint (max_ep_auth_key_cnt) : The maximum number of authorization keys which can be supported per connectionless - endpoint. + endpoint. If this value is non-zero, providers must support FI_AV_AUTH_KEY. # RETURN VALUE diff --git a/man/fi_endpoint.3.md b/man/fi_endpoint.3.md index b57dba4bc12..1fb97cf8130 100644 --- a/man/fi_endpoint.3.md +++ b/man/fi_endpoint.3.md @@ -911,6 +911,8 @@ The length of the authorization key in bytes. This field will be 0 if authorization keys are not available or used. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If the domain is opened with FI_AV_AUTH_KEY, auth_key_size is ignored. + ## auth_key - Authorization Key If supported by the fabric, an authorization key (a.k.a. job @@ -923,6 +925,8 @@ cross traffic. The domain authorization key will be used if auth_key_size is set to 0. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If the domain is opened with FI_AV_AUTH_KEY, auth_key is ignored. + # TRANSMIT CONTEXT ATTRIBUTES Attributes specific to the transmit capabilities of an endpoint are diff --git a/man/fi_mr.3.md b/man/fi_mr.3.md index b73ac2aa3e7..524065439d0 100644 --- a/man/fi_mr.3.md +++ b/man/fi_mr.3.md @@ -373,12 +373,18 @@ must remain valid until the registration operation completes. The context specified with the registration request is returned with the completion event. +For domains opened with FI_AV_AUTH_KEY, fi_mr_reg is not supported +and fi_mr_regattr must be used. + ## fi_mr_regv The fi_mr_regv call adds support for a scatter-gather list to fi_mr_reg. Multiple memory buffers are registered as a single memory region. Otherwise, the operation is the same. +For domains opened with FI_AV_AUTH_KEY, fi_mr_reg is not supported +and fi_mr_regattr must be used. + ## fi_mr_regattr The fi_mr_regattr call is a more generic, extensible registration call @@ -525,6 +531,11 @@ struct fi_mr_attr { } device; void *hmem_data; }; + +struct fi_mr_auth_key { + struct fid_av *av; + fi_addr_t key; +}; ``` ## mr_iov @@ -615,6 +626,9 @@ The size of key referenced by the auth_key field in bytes, or 0 if no authorizat key is given. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If the domain is opened with FI_AV_AUTH_KEY, auth_key_size must equal +`sizeof(struct fi_mr_auth_key)`. + ## auth_key Indicates the key to associate with this memory registration. Authorization @@ -624,6 +638,9 @@ region. The domain authorization key will be used if the auth_key_size provided is 0. This field is ignored unless the fabric is opened with API version 1.5 or greater. +If the domain is opened with FI_AV_AUTH_KEY, auth_key must point to a +user-defined `struct fi_mr_auth_key`. + ## iface Indicates the software interfaces used by the application to allocate and manage the memory region. This field is ignored unless the application has @@ -681,6 +698,22 @@ FI_HMEM_ZE interfaces. The driver and device index values represent their 0-based positions in arrays returned from zeDriverGet and zeDeviceGet, respectively. +## av + +For memory registration being allocated against a domain configured with +FI_AV_AUTH_KEY, av is used to define the fid_av which contains the authorization +keys to be associated with the memory region. If the domain is also opened with +FI_MR_ENDPOINT, the specified AV must be the same AV bound to the endpoint. + +By default, the memory region will be associated with all authorization keys +in the AV. + +## key + +If the domain was opened with FI_DIRECTED_RECV, key can be used to limit the +memory region to a specific authorization key returned from +`fi_av_insert_auth_key`. + # NOTES Direct access to an application's memory by a remote peer requires that