Examples

[mverrilli]

Hi @ggerganov,

I have been working on an implementation using ggml as a lib following some of the example code. In the process, I've been trying to make things a little more modular in the examples area and adding some functionality. I can post a repo for this after I get things cleaned up, but it could probably be adapted to other models and used as an example. However, some of it might not really be the direction you want to go.

Here is a list of some things I've added:

1. Using ICU for better Unicode support
2. nlohmann json
3. Wrapped ggml api calls in a set of classes under a ggml namespace
4. New classes for Models, Vocabs, Tokenizers, etc.
5. BPE tokenizer
6. Modified sample a bit to allow for some more choice in top_k, top_p, temp
7. Added a repetition penalty with distance consideration (this probably needs some additional vetting, for my test model it has not worked out well, but I feel like I'm doing it right).
8. Eval thread w/ thread-safe result queue

[ggerganov]

Sounds interesting - would love to take a look at the repo.

Currently, there is a lot of stuff that can be abstracted and simplified in the examples, but I have been intentionally avoiding it.
Obviously, the result is a ton of "copy-paste" :)

The main reason to avoid it is because I don't want to commit to a specific high-level software architecture yet.
I want to first gather some experience with typical use cases, see what are the needed features and once I have a better understanding, then will try to make something more generic.

I might as well just wait and see what other people like you propose in this regard