Thanks for this. I wonder if the KV cache should be marked ephemeral when the model is created? Otherwise you can never keep the cache around long term (say you want to store threads of different conversations to go back to)

Thanks for this. I wonder if the KV cache should be marked ephemeral when the model is created? Otherwise you can never keep the cache around long term (say you want to store threads of different conversations to go back to)

Perhaps it's possible to achieve the best of both worlds: short-lived sessions that can be deleted and long-lived sessions that can be resumed by explicitly modeling the concept of a session.

E.g. something like that

/** 
 * Represents a session with a unique ID and persistence setting.
 */
public record Session(UUID sessionId, boolean persistent) {

    /**
     * Creates a persistent session with the provided session ID.
     * 
     * <p>
     * This session can be resumed even after the program exits.
     * </p>
     */
    public Session(UUID sessionId) {
        this(sessionId, true);
    }

    /**
     * Creates an ephemeral session with a new random session ID.
     * 
     * <p>
     * All resources are freed when the session is closed.
     * This session cannot be resumed later.
     * </p>
     */
    public Session() {
        this(UUID.randomUUID(), false);
    }
}

....

AbstractModel model = ModelSupport.loadModel(localModelPath, workingMemory, workingQuantization);

// Creates an ephemeral session
Session session = new Session();

Generator.Response response = model.generate(session, ctx, 0.1f, 1024, (s, f) -> {
    // Handle generation callback
});

/*
 * Closes the the given session
 * - Persistent sessions: No deletion of the temporary files
 * - Ephemeral sessions: Deletes temporary files and marks them for deletion on exit
 */
model.close(session);

Since Jlama provides LangChain4j integration, their expectations regarding this integration should also be considered. Based on my understanding of the LangChain4j chat memory documentation, there is no default persistence. However, I must admit that I haven’t explored the LangChain4j integration and its usage in depth yet.

Based on my understanding of the LangChain4j chat memory documentation, there is no default persistence.

Correct, in this case it would always be ephemeral. But for Jlama I want to handle stored sessions. I can take a crack at fixing this based on your PR!