diff --git a/README.md b/README.md index 15b41e2..5f52097 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,13 @@ # RocksQ -An inproc RocksDB-based queue with Python bindings. It is implemented in Rust. +An inproc RocksDB-based queues with Python bindings. -Features: +The library is implemented in Rust and PyO3, which allows to release GIL when necessary. The library does not require +external dependencies to be installed in the environment. + +## Persistent queue + +A persistent queue with following features: - max capacity limit in number of elements; - size calculation based on filesystem space usage; @@ -17,11 +22,46 @@ What is not supported: - pub/sub is not supported intentionally (implement it on top of RocksQ if necessary); - TTL is not supported intentionally (implement it on top of RocksQ if necessary). -## Implementation details +### Implementation details + +It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys +are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs. + +## MPMC queue + +A persistent queue with following features: -It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs. +- TTL in seconds; +- multiple consumers marked with labels; +- size calculation based on filesystem space usage; +- length calculation based on number of elements; +- supports only bytes-like objects; +- can operate in a multithreaded environment efficiently (add and next methods can release GIL if necessary); +- keeps the state between restarts; +- two implementations: blocking and nonblocking; -It is implemented in Rust and PyO3, thus allows to release GIL when necessary. The library does not require external dependencies installed in the environment. +### Implementation details + +It works on RocksDB and uses three column families: + +- data + + Stores queue elements. The keys are 64-bit integers, the values are byte arrays. The keys are generated by + incrementing a counter. + +- system + + Stores a system information like start and write counters, a timestamp of the last write. + +- reader + + Stores an information about consumers like read counters, expiration of elements after last reading. The keys are + string labels of consumers, the values are binary serialized objects. + +TTL is implemented via [RocksDB TTL feature](https://github.com/facebook/rocksdb/wiki/Time-to-Live). TTL is not strict. +It means that the element will remain in the queue for TTL seconds after insertion and the queue will make efforts to +remove the element after TTL seconds but it is not guaranteed to be done immediately. Thus, consumers can retrieve +expired but not removed elements. ## Supported Platforms and Python Versions @@ -29,7 +69,7 @@ It is implemented in Rust and PyO3, thus allows to release GIL when necessary. T **Linux**: ManyLinux Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually. -**MacOS**: Currently, I do not have MacOS environment to debug the build process in MacOS, all volounteers are welcome. +**MacOS**: Currently, I do not have MacOS environment to debug the build process in MacOS, all volunteers are welcome. ## Installation @@ -45,5 +85,5 @@ API docs are located at: [https://insight-platform.github.io/RocksQ/](https://in ## Performance -The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the throughput of the filesystem. - +The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the +throughput of the filesystem. diff --git a/queue_py/src/blocking.rs b/queue_py/src/blocking.rs index 7510afb..3564c22 100644 --- a/queue_py/src/blocking.rs +++ b/queue_py/src/blocking.rs @@ -189,10 +189,10 @@ impl PersistentQueueWithCapacity { /// path : str /// The path to the queue. /// ttl : int -/// The amount of seconds after which the element in the queue will be removed. Ttl is non-strict -/// meaning that it is guaranteed that the element inserted will remain in the queue for at least -/// ttl amount of time and the queue will make efforts to remove the element as soon as possible -/// after ttl seconds of its insertion. +/// The amount of seconds after which the element in the queue will be removed. TTL is not strict. +/// It means that the element will remain in the queue for TTL seconds after insertion and the +/// queue will make efforts to remove the element after TTL seconds but it is not guaranteed to be +/// done immediately. Thus, consumers can retrieve expired but not removed elements. /// /// Raises /// ------ diff --git a/queue_py/src/nonblocking.rs b/queue_py/src/nonblocking.rs index 5d1cc2d..1591a3a 100644 --- a/queue_py/src/nonblocking.rs +++ b/queue_py/src/nonblocking.rs @@ -567,10 +567,10 @@ impl MpmcResponse { /// path : str /// The path to the queue. /// ttl : int -/// The amount of seconds after which the element in the queue will be removed. Ttl is non-strict -/// meaning that it is guaranteed that the element inserted will remain in the queue for at least -/// ttl amount of time and the queue will make efforts to remove the element as soon as possible -/// after ttl seconds of its insertion. +/// The amount of seconds after which the element in the queue will be removed. TTL is not strict. +/// It means that the element will remain in the queue for TTL seconds after insertion and the +/// queue will make efforts to remove the element after TTL seconds but it is not guaranteed to be +/// done immediately. Thus, consumers can retrieve expired but not removed elements. /// max_inflight_ops : int /// The maximum number of inflight operations. If the number of inflight operations reached its limit, /// further ops are blocked until the capacity is available. Default to ``1_000``.