Tokens Chunking to respect Language Word Boundaries

[loretoparisi]

Hello, for my LLM input I need to split input prompt tokens into chunks, where each chunk represent a context window (or session):

let tokensToChunks =  function (arr, chunks) {
      return arr.map(function (e, i) {
        return i % chunks === 0 ? arr.slice(i, i + chunks) : null;
      })
        .filter(function (e) { return e; });
    };
// inputTokens is Uint32Array
let sessions = tokensToChunks(Array.from(inputTokens), max_tokens)

in order to respect the max_tokens at each inference session. This works ok in most cases, but occasionally subwords may fall into different chunks e.g.

let codes = encode("Yorkshire")
//  [ 100077, 15255 ] 
session1 = session1.append( codes[0] )
session2 = session2.append(  codes[1])

thus causing the context window to have improper or missing semantic meaning.
Assumed we work at token level (hence no words, but codes), any good practice to handle in-context chunking to avoid this semantic displacement between different contexts (chunks)?