Can Lucene support selective preloading of just slices/files within a CFS file?

[gautamworah96]

Description

At Amazon Product Search, we warm up our service by preloading vector files into RAM and use the mmapDir.setPreload API for this.
However, when vector files get compacted into a .cfs file, they do not get paged into RAM due to setPreload being called on the basis of .vec, .veq extensions.

This would also help for lexical search files that get compacted into .cfs files and which need preloading for service warm up purposes..

[mikemccand]

One simple workaround is to disable compound file format (.cfs files).

[uschindler]

In general, I would recommend to allow that, too.

We are currently a bit planning how to make CFS files and madvise work together in a better way and this already improved in Lucene 10.

IMHO, the preloading is no longer a good idea when we are using madvise correctly (WILL_NEED). Preloading should be replaced by that (the JVM does the same behind scenes). For this we already have support in CFS files.