refreshAfterWrite jitter

[betrcode]

We have a problem where a lot of our cache entries all get refreshed at the same time, causing problems for our service and downstream service. An idea is to be able to configure a jitter time interval, for example (0ms to 2000ms) so that the refreshes get spread out in time. Wdyt?

From the Java docs:

Specifies that active entries are eligible for automatic refresh once a fixed duration has elapsed after the entry's creation, or the most recent replacement of its value. The semantics of refreshes are specified in LoadingCache.refresh, and are performed by calling CacheLoader.reload.
Automatic refreshes are performed when the first stale request for an entry occurs. The request triggering refresh will make an asynchronous call to CacheLoader.reload and immediately return the old value.

[ben-manes]

In refreshAfterWrite, after the duration has passed the next request to that entry triggers an asynchronous reload. When that occurs is non-deterministic, and often it is combined with expiration so inactive entries fade away. Since this is not a scheduled periodic reload, I don't think the cache should try to perform some type of jitter because it is all reactionary rather than a deterministic action.

If you want to introduce jitter then consider adding an artificial delay to the reload future. This could be done by having the CacheLoader.asyncReload method return a future using CompletableFuture.delayedExecutor. Then you can add a jittered delay and, since the stale value is returned during reload, it won't impact your latencies.

@Override public CompletableFuture<? extends V> asyncReload(K key, V oldValue, Executor executor) {
  return super.asyncReload(key, oldValue, CompletableFuture.delayedExecutor(
      ThreadLocalRandom.current().nextLong(0, 2_000), TimeUnit.MILLISECONDS, executor));
}

You might also find batching the reloads to be assist in performance. See coalescing-bulkloader-reactor for an example using a reactive streams library, which makes this fairly trivial. This can buffer over a time/space interval to make more efficient requests, with a max parallelism to throttle the number of queries being made at once.

does any of that help?

[betrcode]

Excellent! Will try one or both of these. Thank you so much for answering so clearly and quickly. 👍