Seeking guidance on keeping the number of file descriptors low.

[StephanvanSchaik]

In my application, I am currently using Fjall to store a number of data sets together with axum to provide a REST API around those. The application is structured such that it opens all the partitions on start up, and then keeps them open indefinitely as part of a State struct used by the axum application. However, over time I have found that this approach accumulates a lot of open file descriptors exhausting the default soft limit of 1024 file descriptors. When this happens, and I run lsof, most of the open file descriptors are being held open by fjall. Of course, it is possible to change the soft limit, but I am wondering if there is maybe another approach to keep the number of open file descriptors low, since fjall can easily end up using hundreds of them.

Would it make sense to restructure the application to only keep partitions opened for a limited time for as long as they are needed and then simply drop the partition handle (with the necessary synchronization to ensure the partition is only opened at most once) to reduce the pressure on the number of open file descriptors?

[marvin-j97]

This is caused by caching of file descriptors for read operations in the DescriptorTable (akin to the TableCache in RocksDB). The default depends on the OS:

fjall/src/config.rs

Lines 57 to 66 in d00876b

	fn get_open_file_limit() -> usize {
	#[cfg(not(any(target_os = "windows", target_os = "macos")))]
	return 900;
	#[cfg(target_os = "windows")]
	return 400;
	#[cfg(target_os = "macos")]
	return 150;
	}

This does not scale with partitions. Partitions themselves do not open a file descriptor permanently. Instead these are file descriptors for the segment files in the LSM-tree (each partition being one LSM-tree). That also means that dropping a PartitionHandle does nothing. It's just a handle. It's still kept in the keyspace.

You should be able to reduce the limit at the cost of potentially more fopen calls.

Are your keys monotonic (something like ULID or timestamps)?

[StephanvanSchaik]

Thanks for the pointer to Config::max_open_files(). That does look helpful.

I try to use an atomic counter for the keys where possible, but I have a few cases where I need to be able to do a key lookup of arbitrary strings or some such.