-
Notifications
You must be signed in to change notification settings - Fork 22
Programming with libcxl
Currently PSLSE supports the following functions in the libcxl library:
struct cxl_afu_h * cxl_afu_open_dev (char *path);
void cxl_afu_free (struct cxl_afu_h *afu);
int cxl_afu_attach (struct cxl_afu_h *afu, __u64 wed);
bool cxl_pending_event (struct cxl_afu_h *afu);
int cxl_read_event (struct cxl_afu_h *afu, struct cxl_event *event);
int cxl_mmio_map (struct cxl_afu_h *afu, __u32 flags);
int cxl_mmio_unmap (struct cxl_afu_h *afu);
int cxl_mmio_write64 (struct cxl_afu_h *afu, uint64_t offset, uint64_t data);
int cxl_mmio_read64 (struct cxl_afu_h *afu, uint64_t offset, uint64_t *data);
int cxl_mmio_write32 (struct cxl_afu_h *afu, uint64_t offset, uint32_t data);
int cxl_mmio_read32 (struct cxl_afu_h *afu, uint64_t offset, uint32_t *data);
These functions can be split into three categories: access, interrupts and mmio.
struct cxl_afu_h * cxl_afu_open_dev (char *path);
void cxl_afu_free (struct cxl_afu_h *afu);
int cxl_afu_attach (struct cxl_afu_h *afu, __u64 wed);
The first function that will be used to access an AFU is the cxl_afu_open_dev() function. The path to the desired AFU device is passed to this function. This function is similar to fopen(), expect instead of opening access to a file you are opening access to an AFU. Like fopen() which returns a handle, cxl_afu_open_dev() returns a pointer that is used as a handle for all other cxl functions to bind to that specific AFU.
#include "libcxl.h"
struct cxl_afu_h *afu;
// Open AFU device
afu = cxl_afu_open_dev ("/dev/cxl/afu0.0d");
if (!afu) {
perror ("cxl_afu_open_dev");
return -1;
}
The last function that will be used to stop accessing an AFU is the cxl_afu_free() function. Only the handle received from cxl_afu_open_dev() is passed. This function is similiar to fclose() in that it closes out access to the AFU device.
#include "libcxl.h"
struct cxl_afu_h *afu;
// Open AFU device
afu = cxl_afu_open_dev ("/dev/cxl/afu0.0d");
if (!afu) {
perror ("cxl_afu_open_dev");
return -1;
}
...
// Free AFU device
cxl_afu_free (afu);
Finally there is the cxl_afu_attach() function. Although cxl_afu_open_dev() opens the device for access, the AFU isn't usable until the attach function is used to start the AFU. When cxl_afu_attach() is called the 64-bit WED value is provided to the AFU by the PSL.
#include <stdint.h>
#include <stdlib.h>
#include "libcxl.h"
#define CACHELINE_BYTES 128
struct cxl_afu_h *afu;
char *wed;
int ret;
// Open AFU device
afu = cxl_afu_open_dev ("/dev/cxl/afu0.0d");
if (!afu) {
perror ("cxl_afu_open_dev");
return -1;
}
// Allocate exactly 1 aligned cacheline of memory for WED
if (posix_memalign ((void **) &wed, CACHELINE_BYTES, CACHELINE_BYTES)) {
perror ("posix_memalign");
return -1;
}
// Attach AFU passing WED address
if (cxl_afu_attach (afu, (__u64) wed) < 0) {
perror ("cxl_afu_attach");
return -1;
}
/* AFU does stuff here, so wait until it's done */
// Free AFU device
cxl_afu_free (afu);
There are two functions used to catch interrupts issued by the AFU logic. The cxl_pending_event() function simply returns true is an interrupt has been received and false otherwise. The application may choose to poll this status before calling cxl_read_event(). The cxl_read_event() is a blocking function and will not return if no event is pending. When cxl_pending_event() returns the application can take the appropriate action for servicing the interrupt condition in the AFU. At this point no further action is needed regarding that interrupt from a kernel or libcxl perspective. Further calls to cxl_read_event() will act on events that occur after that first event. However, when cxl_read_event() returns the application should call cxl_pending_event() to see if there are indeed other events to be serviced.
Blocking example:
struct cxl_event intr;
// Blocking call for expected interrupt
cxl_read_event (afu, &intr);
handle_event (intr); // Application's interrupt handler
while (cxl_pending_event (afu)) {
cxl_read_event (afu, &intr);
handle_event (intr); // Application's interrupt handler
}
Non-blocking example:
struct cxl_event intr;
// Non-blocking call for expected interrupt
while (cxl_pending_event (afu)) {
cxl_read_event (afu, &intr);
handle_event (intr); // Application's interrupt handler
}
The AFU Problem Space Area (PSA) can be access via MMIO transactions. The libcxl library provides abstraction functions for this access that work both in the PSLSE simulation environment as well as on hardware. In order to access the MMIO space the cxl_mmio_map() function must be called first. This is similar to fopen() except that it allows access to AFU PSA and not a file. The cxl_afu_attach() function must be called before cxl_mmio_map(). The call to cxl_mmio_map() requires a flags field to be passed using one of the flags defined in libcxl.h. These flags are used to define the endianess of the PSA in the AFU.
#define CXL_MMIO_BIG_ENDIAN 0x1
#define CXL_MMIO_LITTLE_ENDIAN 0x2
#define CXL_MMIO_HOST_ENDIAN 0x3
These flags allow libcxl to handle mixed endianess between the AFU, the simulation environment and the actual hardware seemlessly. The application programmer doesn't need to think about endianess for accessing these registers beyond this function call.
#include "libcxl.h"
struct cxl_afu_h *afu;
// Open AFU device
afu = cxl_afu_open_dev ("/dev/cxl/afu0.0d");
if (!afu) {
perror ("cxl_afu_open_dev");
return -1;
}
// Map MMIO region for AFU PSA
if (cxl_mmio_map (afu, CXL_MMIO_FLAGS_AFU_BIG_ENDIAN) < 0) {
perror ("cxl_mmio_map");
return -1;
}
Access to AFU PSA is limited to aligned 32-bit or 64-bit accesses. The libcxl abstraction functions will check these rules are obeyed. The offset value is a byte offset and must be a multiple of 4 in the case of 32-bit accesses and a multiple of 8 in the case of 64-bit accesses.
#include <stdint.h>
#include "libcxl.h"
struct cxl_afu_h *afu;
uint64_t data64;
uint32_t data32;
// Open AFU device
afu = cxl_afu_open_dev ("/dev/cxl/afu0.0d");
if (!afu) {
perror ("cxl_afu_open_dev");
return -1;
}
// Map MMIO region for AFU PSA
if (cxl_mmio_map (afu, CXL_MMIO_FLAGS_AFU_BIG_ENDIAN) < 0) {
perror ("cxl_mmio_map");
return -1;
}
// Do some reads and writes
if (cxl_mmio_read64 (afu, 0, &data64) < 0) {
perror ("cxl_mmio_read64");
return -1;
}
if (cxl_mmio_write64 (afu, 8, data64) < 0) {
perror ("cxl_mmio_write64");
return -1;
}
if (cxl_mmio_read32 (afu, 20, &data32) < 0) {
perror ("cxl_mmio_read32");
return -1;
}
if (cxl_mmio_write32 (afu, 28, data32) < 0) {
perror ("cxl_mmio_write32");
return -1;
}
Finally the cxl_mmio_unmap() function is used to cleanly unmap the MMIO region when the application is completely done with it. This is similar to using fclose() to finish using a file.
// Unmap MMIO region for AFU PSA
cxl_mmio_unmap (afu);