Samantha Interface Assistant: An Open Science Project

[controlecidadao]

✨ Experimental Environment Designed to Democratize the Use of Open Source Large Language Models (LLM)

Samantha Project: https://github.com/controlecidadao/samantha_ia

👇 Check out an excerpt from its README file. Suggestions will be very welcome!

💻 Samantha is just a simple interface assistant for open source artificial intelligence models, developed under Open Science principles (open methodology, open source, open data, open access, open peer review and open educational resources) and MIT License for use on common Windows computers (without GPU). The program runs the LLM locally, free of charge and unlimitedly, without the need for an internet connection, except to download GGUF models or when required by the execution of the code created by the models (g.e. to download datasets for data analysis). Its objective is to democratize knowledge about the use of AI and demonstrate that, using the appropriate technique, even small models are capable of producing responses similar to those of larger ones. Her mission is to help explore the boundaries of (realy) open AI models.

♾️ The system allows the sequential loading of a list of prompts (prompt chaining) and models (model chaining), one model at a time to save memory, as well as the adjustment of their hyperparameters, allowing the response generated by the previous model to be feedbacked and analyzed by the subsequent model to generate the next response (Feedback Loop feature), in an unlimited number of interaction cycles between LLMs without human intervention. Models can interact with the answer provided by the immediately preceding model, so each new response replaces the previous one. You can also use just one model and have it interact with its previous response over an unlimited number of text generation cycles. Use your imagination to combine models, prompts and features!

🎬 Usage Example:

Intelligence Challenge: Gemma 2 vs Phi 3.5 with Llama 3.1 as Judge

📌 Samantha's Key Features

✅ Open Source Foundation: Built upon Llama.cpp / llama-cpp-python and Gradio , under MIT license, Samantha runs on standard computers, even without a dedicated Graphics Processing Unit (GPU).

✅ Offline Capability: Samantha operates independently of the internet, requiring connectivity only for the initial download of model files or when required by the execution of the code created by the models. This ensures privacy and security for your data processing needs. Your sensitive data is not shared via the internet with companies through confidentiality agreements.

✅ Unlimited and Free Use: Samantha's open source nature allows for unrestricted use without any costs or limitations, making it accessible to anyone, anywhere, anytime.

✅ Extensive Model Selection: With access to thousands of foundation and fine-tuned open source models, users can experiment with various AI capabilities, each tailored to different tasks and applications, allowing to chain the sequence of models that best meet your needs.

✅ Copy and paste LLMs: To try out a sequence of gguf models, just copy their download links from any Hugging Face repository and paste inside Samantha to run them right away in sequence.

✅ Customizable Parameters: Users have control over model hyperparameters such as context window length (n_ctx, max_tokens), token sampling (temperature, tfs_z, top-k, top-p, min_p, typical_p), penalties (presence_penalty, frequency_penalty, repeat_penalty) and stop words (stop), allowing for responses that suit specific requirements, with deterministic or stochastic behavior.

✅ Random hyperparameter adjustments: You can test random combinations of hyperparameter settings and observe their impact on the responses generated by the model.

✅ Interactive Experience: Samantha's chaining functionality enables users to generate endless texts by chaining prompts and models, facilitating complex interactions between different LLMs without human intervention.

✅ Feedback Loop: This feature allows you to capture the response generated by the model and feed it back into the next cycle of the conversation.

✅ Prompt List: You can add any number of prompts (separated by $$$\n or \n) to control the sequence of instructions to be executed by the models. It is possible to import a TXT file with a predefined sequence of prompts.

✅ Model List: You can select any number of models and in any order to control which model responds to the next prompt.

✅ Cumulative Response: You can concatenate each new response by adding it to the previous response to be considered when generating the next response by the model. It is important to highlight that the set of concatenated responses must fit in the model's context window.

✅ Learning Insights: A feature called Learning Mode lets users observe the model's decision-making process, providing insights into how it selects output tokens based on their probability scores (logistic units or just logits) and hyperparameter settings. A list of the least likely selected tokens is also generated.

✅ Voice Interaction: Samantha supports simple voice commands with offline speech-to-text Vosk (English and Portuguese) and text-to-speech with SAPI5 voices, making it accessible and user-friendly.

✅ Audio feedback: The interface provides audible alerts to the user, signaling the beginning and end of the text generation phase by the model.

✅ Document Handling: The system can load small PDF and TXT files. Chaining user prompts, system prompt and model's URL list can be inputted via a TXT file for convenience.

✅ Versatile Text Input: Fields for prompt insertion allow users to interact with the system effectively, including system prompt, previous model response and user prompt to guide the model's response.

✅ Code Integration: Automatic extraction of Python code blocks from model's response, along with pre-installed JupyterLab integrated development environment (IDE) in an isolated virtual environment, enables users to run generated code swiftly for immediate results.

✅ Edit, Copy and Run Python Code: The system allows the user to edit the code generated by the model and run it by selecting, copying with CTRL + C and clicking the Run code button. You can also copy a Python code from anywhere (e.g. from a webpage) and run it just by pressing Copy Python code and Run code buttons (as long as it uses the installed Python libraries).

✅ Code Blocks Editing: Users can select and run Python code blocks generated by the model that uses the libraries installed in the jupyterlab virtual environment by entering the #IDE comment in the output code, selecting and copying with CTRL + C, and finally clicking the Run code button;

✅ HTML output: Display Python interpreter output in an HTML pop-up window when text printed in the terminal is other than '' (empty string). This feature allows, for example, to execute a script unlimitedly and only display the result when a certain condition is met;

✅ Automatic Code Execution: Samantha features the option to automatically run the Python code generated by the models sequentially.

✅ Stop Condition: Stops Samantha if the automatic execution of the Python code generated by the model prints in the terminal a value other than '' (empty string) and that does not contain error message. You can also force exit a running loop by creating a function that returns only the string STOP_SAMANTHA when a certain condition is met.

✅ Incremental Coding: Using deterministic settings, create Python code incrementally, making sure each part works before moving on to the next.

✅ Complete access and control: Through the ecosystem of Python libraries and the codes generated by the models, it is possible to access computer files, allowing you to read, create, change and delete local files, as well as access the internet, if available, to upload and download information and files.

✅ Data Analysis Tools: A suite of data analysis tools like Pandas, Numpy, SciPy, Scikit-Learn, Matplotlib, Seaborn, Vega-Altair, Plotly, Bokeh, Dash, Ydata-Profiling, Sweetviz, D-Tale, DataPrep, NetworkX, Pyvis, Selenium, PyMuPDF and Beautiful Soup are available within JupyterLab for comprehensive analysis and visualization (for a complete list of all Python available libraries, use a prompt like "create a Python code that prints all modules installed using pkgutil library. Separate each module with
tag." and press Run code button. The result will be displayed in a browser popup). Integration with DB Browser is also available.

✅ Performance Optimized: To ensure smooth performance on CPUs, Samantha maintains a limited chat history to just the previous response, reducing the model's context window size to save memory and computational resources.

👟 Testing a Model in 5 Steps:

Copy & Paste LLM: Samantha needs just a .gguf model file (Hugging Face URL) to generate text. Follow these steps to perform a simple model test:

1) Open Windows Task Management by pressing CTRL + SHIFT + ESC and check available memory. Close some programs if necessary to free memory.

2) Visit Hugging Face repository and click on the card to open the corresponding page. Locate the Files and versions tab and choose a .gguf model that fits in your available memory.

3) Right click over the model download link icon and copy its URL.

4) Paste the model URL into Samantha's Download models for testing field.

5) Insert a prompt into User prompt field and press Enter. Keep the $$$ sign at the end of your prompt. The model will be downloaded and the response will be generated using the default deterministic settings. You can track this process via Windows Task Management.

🙏 On the Shoulders of Giants

Special thanks to Georgi Gerganov and the whole team working on llama.cpp for making all of this possible, as well as to Andrei Bleten by his amazing Python bidings for the Gerganov C++ library (llama-cpp-python).