-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
56 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,6 +8,60 @@ Implementation of a cool communication layer. | |
- \@JiakunYan ([email protected]) | ||
- \@snirmarc | ||
|
||
## Overview | ||
|
||
The Lightweight Communication Interface (LCI) is designed to be an efficient communication library | ||
for multithreaded, irregular communications. It is a research tool to explore design choices | ||
for such libraries. It has the following major features: | ||
- **Multithreaded performance as the first priority**: No big blocking locks like those in MPI! | ||
We carefully design the internal data structures to minimize interference between threads. | ||
We use atomic operations and fine-grained try locks extensively instead of coarse-grained blocking locks. | ||
The posting sends/receives, polling completions, and making progress on the progress engine | ||
(`LCI_progress`) use different locks or no locks at all! Threads would not interfere with each other | ||
unless necessary. | ||
|
||
- **Versatile communication interface**: LCI provides users with various options including: | ||
- Communication primitives: two-sided send/recv, one-sided put/get. | ||
- Completion mechanisms: synchronizers (similar to MPI requests/futures), completion queues, function handlers. | ||
- Protocols: small/medium/long messages mapping to eager/rendezvous protocol. | ||
- Communication buffers: for both source/target buffers, we can use | ||
user-provided/runtime-allocated buffers. | ||
- Registration: for long messages, users can explicitly register the buffer or leave it to runtime | ||
(just use `LCI_SEGMENT_ALL`). | ||
|
||
The options are orthogonal and almost all combinations are valid! | ||
For example, the example code [putla_queue](examples/putla_queue.c) uses | ||
one-sided put + user-provided source buffer + runtime-allocated target buffer + | ||
rendezvous protocol + completion queue on source/targer side + explicit registration. | ||
|
||
- **Explicit control of communication behaviors and resources**: versatile communication interface has already | ||
given users a lot of control. Besides, users can control various low-level features through | ||
API/environmental variables/cmake variables such as | ||
- Replication of communication devices. | ||
- The semantics of send/receive tag matching. | ||
- All communication primitives are non-blocking and users can decide when to retry in case of | ||
temporarily unavailable resources. | ||
- LCI also gives users an explicit function (`LCI_progress`) to make progress on the communication engine. | ||
- Different implementation and size of completion queues/matching tables. | ||
|
||
Users can tailor the LCI configuration to reduce software overheads, or just use default settings if | ||
LCI is not a performance bottleneck. | ||
|
||
Currently, LCI is implemented as a mix of C and C++ libraries. Lightweight Communication Tools (LCT) | ||
is a C++ library providing basic tools that can be used across libraries. Lightweight Communication | ||
Interface (LCI) is a C library implementing communication-related features. | ||
|
||
Currently, the functionalities in the LCT library include: | ||
- timing. | ||
- string searching and manipulation. | ||
- query thread ID and number. | ||
- logging. | ||
- performance counters. | ||
- different implementation of queues. | ||
- PMI (Process Management Interface) wrappers. | ||
|
||
The actual API and (some) documentation are located in [lct.h](lct/api/lct.h) and [lci.h](lci/api/lci.h). | ||
|
||
## Installation | ||
``` | ||
cmake . | ||
|
@@ -27,6 +81,8 @@ make install | |
- `ofi`: libfabrics, for all other networks (slingshot-11, ethernet, shared memory). | ||
- `LCI_FORCE_SERVER=ON/OFF`: Default value is `OFF`. If it is set to `ON`, | ||
`LCI_SERVER` will not be treated as a hint but a requirement. | ||
- `LCI_WITH_LCT_ONLY=ON/OFF`: Whether to only build LCT (The Lightweight COmmunication Tools). | ||
Default is `OFF` (build both LCT and LCI). | ||
|
||
## Run LCI applications | ||
|
||
|