-
|
Hi, i'm wondering if there's an existing interface to track buffer memory usage within arrow-rs. In C++ there's an arrow::MemoryPool, e.g., in page decompression it's used for decompressor buffer while in rust the buffer is from Vec. Tracking memory usage can be helpful for cases where memory control is desired. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
We provide methods to get the allocated size of buffers in arrays and these can be hooked into a memory tracking system. DataFusion has examples of this. We don't provide anything first-party in part because Rust doesn't have stable support for custom allocators which would be the ideal way to provide this functionality. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for the rely!
would this be too late? i.e., it seems better to prevent insane huge allocation by checking the size to be allocated before real allocation. |
Beta Was this translation helpful? Give feedback.
-
|
We support tracking memory usage, fallible allocation could be achieved by overriding the global allocator. |
Beta Was this translation helpful? Give feedback.
We provide methods to get the allocated size of buffers in arrays and these can be hooked into a memory tracking system. DataFusion has examples of this. We don't provide anything first-party in part because Rust doesn't have stable support for custom allocators which would be the ideal way to provide this functionality.