man: Doc FI_AV_AUTH_KEY

FI_AV_AUTH_KEY is used to enable multiple auth keys per connectionless
endpoint.

Signed-off-by: Ian Ziemba <[email protected]>

man/fi_av.3.md

@@ -24,6 +24,12 @@ fi_av_lookup
 fi_av_straddr
 : Convert an address into a printable string.
 
+fi_av_insert_auth_key
+: Insert an authorization key into the address vector.
+
+fi_av_lookup_auth_key
+: Retrieve an authorization key stored in the address vector.
+
 # SYNOPSIS
 
 ```c
@@ -58,6 +64,12 @@ fi_addr_t fi_rx_addr(fi_addr_t fi_addr, int rx_index,
 
 const char * fi_av_straddr(struct fid_av *av, const void *addr,
       char *buf, size_t *len);
+
+int fi_av_insert_auth_key(struct fid_av *av, const void *auth_key,
+      size_t auth_key_size, fi_addr_t *fi_addr, uint64_t flags);
+
+int fi_av_lookup_auth_key(struct fid_av *av, fi_addr_t addr,
+      void *auth_key, size_t *auth_key_size);
 ```
 
 # ARGUMENTS
@@ -97,6 +109,14 @@ const char * fi_av_straddr(struct fid_av *av, const void *addr,
 *flags*
 : Additional flags to apply to the operation.
 
+*auth_key*
+: Buffer containing authorization key to be inserted into the address
+  vector.
+
+*auth_key_size*
+: On input, specifies size of auth_key buffer.  On output, stores number
+  of bytes written to auth_key buffer.
+
 # DESCRIPTION
 
 Address vectors are used to map higher-level addresses, which may be
@@ -328,6 +348,16 @@ that calls to fi_av_insert following a call to fi_av_remove always reference a
 valid buffer in the fi_addr parameter.  Otherwise it may be difficult to
 determine what the next assigned index will be.
 
+If the address vector is configured with authorization keys, the fi_addr
+parameter cannot be NULL and is used to define the authorization key associated
+with the address being inserted. Acceptable inputs are the fi_addr_t's generated
+from `fi_av_insert_auth_key`. Only `fi_addr[0]` can be used to pass in an
+authorization key fi_addr_t. Return returned fi_addr_t's will map to endpoint
+address against the specified authorization key. These fi_addr_t's can be
+used as the target for local data transfer operations. If the endpoint
+supports `FI_DIRECTED_RECV`, these fi_addr_t's can be used to restrict
+receive buffers to a specific endpoint address and authorization key.
+
 *flags*
 : The following flag may be passed to AV insertion calls: fi_av_insert,
   fi_av_insertsvc, or fi_av_insertsym.
@@ -414,6 +444,11 @@ accessed.  Inserted addresses are not required to be removed.
 fi_av_close will automatically cleanup any resources associated with
 addresses remaining in the AV when it is invoked.
 
+If the address being removed came from `fi_av_insert_auth_key`, the address
+will only be removed if all endpoints, which have been enabled against the
+corresponding authorization key, have been closed. If all endpoints are not
+closed, -FI_EBUSY will be returned.
+
 Flags are reserved for future use and must be 0.
 
 ## fi_av_lookup
@@ -450,6 +485,46 @@ size of the buffer needed to store the address.  This size may be
 larger than the input len.  If the provided buffer is too small, the
 results will be truncated.  fi_av_straddr returns a pointer to buf.
 
+## fi_av_insert_auth_key
+
+This function associates authorization keys with an address vector. This
+requires the domain to be opened with `FI_AV_AUTH_KEY`. `FI_AV_AUTH_KEY`
+enables endpoints and memory regions to be associated with authorization
+keys from the address vector. This behavior enables a single endpoint
+or memory region to be associated with multiple authorization keys.
+
+When endpoints or memory regions are enabled, they are configured with
+address vector authorization keys at that point in time. Later authorization
+key insertions will not propagate to already enabled endpoints and memory
+regions.
+
+The `auth_key` and `auth_key_size` parameters are used to input the
+authorization key into the address vector. The structure of the
+authorization key is provider specific.
+
+The output of `fi_av_insert_auth_key` is a fi_addr_t handle representing
+all endpoint addresses against this specific authorization key. For
+endpoints enabled with FI_DIRECTED_RECV, authorization key fi_addr_t's
+can be used to restrict incoming messages to only endpoint addresses
+within the authorization key.
+
+These authorization key fi_addr_t's can later be used an input for
+endpoint address insertion functions to generate an fi_addr_t for a
+specific endpoint address and authorization key.
+
+Flags are reserved for future use and must be 0.
+
+## fi_av_lookup_auth_key
+
+This functions returns the authorization key associated with a fi_addr_t.
+Acceptable fi_addr_t's input are the output of `fi_av_insert_auth_key` and
+AV address insertion functions. The returned authorization key is in a
+provider specific format. On input, the auth_key_size parameter should
+indicate the size of the auth_key buffer. If the actual authorization key
+is larger than what can fit into the buffer, it will be truncated.  On
+output, auth_key_size is set to the size of the buffer needed to store the
+authorization key, which may be larger than the input value.
+
 # NOTES
 
 An AV should only store a single instance of an address.

man/fi_cq.3.md

@@ -448,6 +448,11 @@ Notable completion error codes are given below.
   passed directly into an fi_av_insert call to add the source address
   to the address vector.
 
+  For API versions 1.20 and later, if the EP is configured with
+  FI_AV_AUTH_KEY, src_addr will be set to the fi_addr_t authorization key
+  handle corresponding to the incoming data transfer. Otherwise, the
+  value will be set to FI_ADDR_NOTAVAIL.
+
 ## fi_cq_signal
 
 The fi_cq_signal call will unblock any thread waiting in fi_cq_sread

man/fi_domain.3.md

@@ -743,6 +743,9 @@ The default authorization key to associate with endpoint and memory
 registrations created within the domain. This field is ignored unless the
 fabric is opened with API version 1.5 or greater.
 
+If domain auth_key_size is set to the value FI_AV_AUTH_KEY, domain auth_key
+is ignored.
+
 ## Default authorization key length (auth_key_size)
 
 The length in bytes of the default authorization key for the domain. If set to 0,
@@ -751,6 +754,11 @@ registrations created within the domain unless specified in the endpoint or
 memory registration attributes. This field is ignored unless the fabric is
 opened with API version 1.5 or greater.
 
+If the size is set to the value FI_AV_AUTH_KEY, all endpoints and memory
+regions will be configured to use authorization keys associated with the AV.
+Providers which support authorization keys and connectionless endpoint must
+support this option.
+
 ## Max Error Data Size (max_err_data)
 : The maximum amount of error data, in bytes, that may be returned as part of
   a completion or event queue error.  This value corresponds to the
@@ -776,7 +784,7 @@ for additional information.
 ## Max Authorization Keys per Endpoint (max_ep_auth_key_cnt)
 
 : The maximum number of authorization keys which can be supported per connectionless
-  endpoint.
+  endpoint. If this value is non-zero, providers must support FI_AV_AUTH_KEY.
 
 # RETURN VALUE
 

man/fi_endpoint.3.md

@@ -911,6 +911,8 @@ The length of the authorization key in bytes.  This field will be 0 if
 authorization keys are not available or used.  This field is ignored
 unless the fabric is opened with API version 1.5 or greater.
 
+If the domain is opened with FI_AV_AUTH_KEY, auth_key_size is ignored.
+
 ## auth_key - Authorization Key
 
 If supported by the fabric, an authorization key (a.k.a. job
@@ -923,6 +925,8 @@ cross traffic.  The domain authorization key will be used if auth_key_size
 is set to 0.  This field is ignored unless the fabric is opened with API
 version 1.5 or greater.
 
+If the domain is opened with FI_AV_AUTH_KEY, auth_key is ignored.
+
 # TRANSMIT CONTEXT ATTRIBUTES
 
 Attributes specific to the transmit capabilities of an endpoint are

man/fi_mr.3.md

@@ -373,12 +373,18 @@ must remain valid until the registration operation completes.  The
 context specified with the registration request is returned with the
 completion event.
 
+For domains opened with FI_AV_AUTH_KEY, fi_mr_reg is not supported
+and fi_mr_regattr must be used.
+
 ## fi_mr_regv
 
 The fi_mr_regv call adds support for a scatter-gather list to
 fi_mr_reg.  Multiple memory buffers are registered as a single memory
 region.  Otherwise, the operation is the same.
 
+For domains opened with FI_AV_AUTH_KEY, fi_mr_reg is not supported
+and fi_mr_regattr must be used.
+
 ## fi_mr_regattr
 
 The fi_mr_regattr call is a more generic, extensible registration call
@@ -525,6 +531,11 @@ struct fi_mr_attr {
     } device;
     void               *hmem_data;
 };
+
+struct fi_mr_auth_key {
+    struct fid_av *av;
+    fi_addr_t     key;
+};
 ```
 ## mr_iov
 
@@ -615,6 +626,9 @@ The size of key referenced by the auth_key field in bytes, or 0 if no authorizat
 key is given.  This field is ignored unless the fabric is opened with API
 version 1.5 or greater.
 
+If the domain is opened with FI_AV_AUTH_KEY, auth_key_size must equal
+`sizeof(struct fi_mr_auth_key)`.
+
 ## auth_key
 
 Indicates the key to associate with this memory registration.  Authorization
@@ -624,6 +638,9 @@ region.  The domain authorization key will be used if the auth_key_size
 provided is 0.  This field is ignored unless the fabric is opened with API
 version 1.5 or greater.
 
+If the domain is opened with FI_AV_AUTH_KEY, auth_key must point to a
+user-defined `struct fi_mr_auth_key`.
+
 ## iface
 Indicates the software interfaces used by the application to allocate and
 manage the memory region. This field is ignored unless the application has
@@ -681,6 +698,22 @@ FI_HMEM_ZE interfaces.  The driver and device index values represent
 their 0-based positions in arrays returned from zeDriverGet and zeDeviceGet,
 respectively.
 
+## av
+
+For memory registration being allocated against a domain configured with
+FI_AV_AUTH_KEY, av is used to define the fid_av which contains the authorization
+keys to be associated with the memory region. If the domain is also opened with
+FI_MR_ENDPOINT, the specified AV must be the same AV bound to the endpoint.
+
+By default, the memory region will be associated with all authorization keys
+in the AV.
+
+## key
+
+If the domain was opened with FI_DIRECTED_RECV, key can be used to limit the
+memory region to a specific authorization key returned from
+`fi_av_insert_auth_key`.
+
 # NOTES
 
 Direct access to an application's memory by a remote peer requires that