Tokenizer running on UI thread blocks the UI

[Anajrim01]

I've noticed that the tokenizer currently runs on the UI thread, causing the UI to block and become unresponsive during tokenization. This can significantly impact the user experience, especially when dealing with large texts or data sets. It would be beneficial to move this processing to a background thread using WorkerThreads or a similar asynchronous solution to keep the UI responsive.

Steps to Reproduce:

1. Open a chat.
2. Trigger the tokenizer process with large text (copy and paste works best).
3. Observe the UI responsiveness during tokenization.

Expected Behavior:

The tokenizer should run in the background, allowing the UI to remain responsive. The UI thread should not be blocked by lengthy tokenization processes.

Actual Behavior:

Currently, the tokenizer runs on the UI thread, causing the UI to freeze and become unresponsive until the tokenization is complete.

Proposed Solution:

Refactor the tokenizer to run in a background thread using Web Workers or another async mechanism. This will prevent the UI thread from being blocked and enhance the overall user experience.

Here’s a high-level approach:

1. Create a Worker that will handle the tokenization process.
2. Move the tokenization logic to the Worker.
3. Communicate the results from the Worker back to the main thread once tokenization is complete.
4. Update the UI with the tokenization results without blocking the main thread.

References:

• Web Workers API - MDN
• Using Web Workers - MDN

I'm happy to submit a pull request (PR) with a fix for this issue if no one else is interested or has another solution.