Upload 3.0.0 files from dev branch

.gitignore

@@ -1,4 +1,6 @@
 __pycache__
 *.bin
 /models/*
-start.bat
+/.cache/*
+/presets/user.json
+start.bat

CITATION.cff

@@ -0,0 +1,7 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+  - given-names: ehristoforu
+title: "TensorLM-webui"
+date-released: 2024-04-03
+url: "https://github.com/ehristoforu/TensorLM-webui"

CODEOWNERS

@@ -0,0 +1 @@
+*       @ehristoforu

README.md

@@ -1,22 +1,97 @@
-![](https://img.shields.io/github/license/ehristoforu/TensorLM-webui.svg) ![](https://img.shields.io/github/downloads/ehristoforu/TensorLM-webui/total.svg) ![](https://img.shields.io/github/stars/ehristoforu/TensorLM-webui.svg)
-
 # TensorLM - webui for LLM models
 
 ![preview](assets/preview.png)
 
-![settings-preview](assets/settings-preview.png)
+*This is Fooocus from the world of Stable Difusion in the world of Text Generation, the same ease of use and the same convenience.*
 
 This is simple and modern [Gradio](https://gradio.app) webui for LLM models GGML format (.bin) based on LLaMA.
 
+---
+Navigation: \
+[**Installing**](#installing) \
+[**Presets**](#presets) \
+[**Model downloading**](#model-downloading) \
+[**Args**](#args)
+---
+
 ## Fast use
 
 You can use this webui in cloud service Colab: <a target="_blank" href="https://colab.research.google.com/github/ehristoforu/TensorLM-webui/blob/dev/assets/TensorLM_colab.ipynb">
   <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
 </a>
 
-## How to install
+## Features
+
+- Simple to use
+- Comfy to work
+- Not demanding on resources
+- Beautiful and pleasant interface
+
+## Installing
+
+### In Windows
+
+[**>>> Portable one-click packege <<<**](https://github.com/ehristoforu/TensorLM-webui/releases/download/latest/latest_TensorLM_portable_win64.7z)
+
+Step-by-step installation:
+  1. Install [Python 3.10.6](https://www.python.org/ftp/python/3.10.6/python-3.10.6-amd64.exe) and [Git](https://github.com/git-for-windows/git/releases/download/v2.44.0.windows.1/Git-2.44.0-64-bit.exe)
+  2. Run ```git clone https://github.com/ehristoforu/TensorLM-webui.git```
+  3. Run ```cd TensorLM-webui```
+  4. Run ```update_mode.bat``` && enter 1 and 2
+  5. Run ```start.bat```
+
+### In MacOS
+
+Step-by-step installation:
+  1. Install [Python 3.10.6](https://www.python.org/ftp/python/3.10.6/python-3.10.6-macos11.pkg) and [Git](https://git-scm.com/download/mac)
+  2. Run ```git clone https://github.com/ehristoforu/TensorLM-webui.git```
+  3. Run ```cd TensorLM-webui```
+  4. Run ```python pip install -r requirements.txt```
+  5. Run ```python webui.py```
+
+### In Linux
+
+Step-by-step installation:
+  1. Install [Python 3.10.6](https://www.python.org/downloads/release/python-3106/) and [Git](https://git-scm.com/download/linux)
+  2. Run ```git clone https://github.com/ehristoforu/TensorLM-webui.git```
+  3. Run ```cd TensorLM-webui```
+  4. Run ```python pip install -r requirements.txt```
+  5. Run ```python webui.py```
+
+## Presets
+
+In this app there is 23 default presets. \
+*Thanks, [@mustvlad](https://github.com/mustvlad/ChatGPT-System-Prompts) for system prompts!*
+
+You can create your custom presets, instruction in ```presets``` folder (it is .md-file).
+
+## Model downloading
+
+With this interface you don't need to scour the Internet looking for a compatible model; in the "Tabs" checkbox and in the "ModelGet" tab you can choose which model to download from our verified repository on HuggingFace.
+
+## Args
+
+*To use args*:
+  - In Windows: edit start.bat with Notepad and edit line with ```python webui.py``` to ```python webui.py [Your args]```, for ex. ```python webui.py --inbrowser```
+  - In MacOS & Linux: run ```python webui.py``` with args - ```python webui.py {Your args}```, for ex. ```python webui.py --inbrowser```
+
+### Args list
+
+`--inbrowser --share --lowvram --debug --quiet`
+
+## Forks
 
-It is very simple and easy. Go to releases on Github project page and download latest portable version.
+While there are no forks 😔, perhaps you will be the first who can significantly improve this application!
 
-Next, open README.md in release and follow the instructions.
+## ![](https://img.shields.io/github/license/ehristoforu/TensorLM-webui.svg)
 
+## Citation
+```bibtex
+    @software{ehristoforu_TensorLM-webui_2024,
+        author = {ehristoforu},
+        month = apr,
+        title = {{TensorLM-webui}},
+        url = {https://github.com/ehristoforu/TensorLM-webui},
+        year = {2024}
+    }
+```

assets/TensorLM_colab.ipynb

@@ -0,0 +1,68 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "T4",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/ehristoforu/TensorLM-webui/blob/dev/assets/TensorLM_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# TensorLM - webui for LLM models\n",
+        "\n",
+        "![preview.png](https://raw.githubusercontent.com/ehristoforu/TensorLM-webui/main/assets/preview.png?token=GHSAT0AAAAAACMN7XRD5UHXZGGJF77PGJDKZN7SQLA)\n",
+        "\n",
+        "This is simple and modern [Gradio](https://gradio.app) webui for LLM models GGML format (.bin) based on LLaMA."
+      ],
+      "metadata": {
+        "id": "1OkQD7FRWDAg"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "66p_ooUpUJ0o"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Install & Run UI\n",
+        "\n",
+        "!git clone https://github.com/ehristoforu/TensorLM-webui.git -b colab\n",
+        "\n",
+        "%cd /content/\n",
+        "\n",
+        "!pip install -q -r TensorLM-webui/requirements.txt\n",
+        "\n",
+        "!wget -O TensorLM-webui/models/llama-2-7b-chat.ggmlv3.q2_K.bin https://huggingface.co/ehristoforu/LLMs/resolve/main/llama-2-7b-chat.ggmlv3.q2_K.bin\n",
+        "\n",
+        "%cd TensorLM-webui\n",
+        "\n",
+        "!python webui.py"
+      ]
+    }
+  ]
+}

configure.txt

@@ -0,0 +1,13 @@
+tlm_version = 3.0.0pre3
+
+theme = theme-repo/STONE_Theme # | theme-repo/STONE_Theme | theme-repo/MONO_Theme | theme-repo/GLASS_Theme 
+
+echo = True
+
+chat_style = panel # | panel | bubble
+
+footer = True
+
+show_api = True # | True | False
+
+share_server_protocol = https # | https | http

modules/arg_parser.py

@@ -0,0 +1,36 @@
+import os
+import gradio as gr
+import copy
+import llama_cpp
+from llama_cpp import Llama
+import random
+from huggingface_hub import hf_hub_download  
+import time
+import argparse
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument("--inbrowser", action="store_true", help="Run in browser")
+parser.add_argument("--share", action="store_true", help="Run in share mode with public url")
+parser.add_argument("--debug", action="store_true", help="Run in debug mode for dev")
+parser.add_argument("--quiet", action="store_true", help="Run in quiet mode without many console trash logs")
+parser.add_argument("--lowvram", action="store_true", help="Run in low vram mode")
+
+
+args = parser.parse_args()
+
+
+global inbrowser_arg
+inbrowser_arg = args.inbrowser
+
+global share_arg
+share_arg = args.share
+
+global debug_arg
+debug_arg = args.debug
+
+global quiet_arg
+quiet_arg = args.quiet
+
+global lowvram_arg
+lowvram_arg = args.lowvram

modules/download_model.py

@@ -0,0 +1,18 @@
+import os
+import gradio as gr
+import copy
+import llama_cpp
+from llama_cpp import Llama
+import random
+from huggingface_hub import hf_hub_download  
+import time
+
+def download_model(repo_id, filename):
+    hf_hub_download(
+        repo_id=repo_id,
+        filename=filename,
+        local_dir="models",
+        force_download=True, resume_download=False,
+        cache_dir=".cache",
+    )
+    return f"Downloaded!"

modules/echo.py

@@ -0,0 +1,34 @@
+from art import *
+import os
+import time
+import psutil
+
+from modules.load_configure import *
+
+print("")
+print(text2art('''TensorLM''', font="small"))
+print("Our license: https://www.gnu.org/licenses/gpl-3.0.txt")
+time.sleep(2.5)
+print(" ")
+
+print(f"Version: {tlm_version}")
+print("")
+print(f"Share server protocol: {share_server_protocol}")
+print(f"Show API: {show_api}")
+print(f"Theme: {theme}")
+print(f"Chat style: {chat_style}")
+print("")
+print('System memory:', psutil.virtual_memory())
+print('System swap memory:', psutil.swap_memory())
+print("")
+for _ in range(3):
+    info = psutil.cpu_percent(interval=1)
+    print('CPU percent (interval=1, percpu=False):', info)
+
+print()
+for _ in range(3):
+    info = psutil.cpu_percent(interval=1, percpu=True)
+    print('CPU percent (interval=1, percpu=True):', info)
+
+print()
+print('Logical CPUs:', psutil.cpu_count())

modules/inference.py

@@ -0,0 +1,66 @@
+import os
+import gradio as gr
+import copy
+import llama_cpp
+from llama_cpp import Llama
+import random
+from huggingface_hub import hf_hub_download  
+import time
+
+from modules.load_presets import load_presets_value
+from modules.load_model import *
+
+def generate_text(message, history, system_prompt, preset, temperature, max_tokens, top_p, top_k, repeat_penalty, model, n_ctx, n_gpu_layers, n_threads, verbose, f16_kv, logits_all, vocab_only, use_mmap, use_mlock, n_batch, last_n_tokens_size, low_vram, rope_freq_base, rope_freq_scale):
+    dir = os.getcwd()
+    global llm
+    llm = Llama(
+        model_path=f"{dir}\models\{model}",
+        n_ctx=n_ctx,
+        n_gpu_layers=n_gpu_layers,
+        n_threads=n_threads,
+        verbose=verbose,
+        f16_kv=f16_kv,
+        logits_all=logits_all,
+        vocab_only=vocab_only,
+        use_mmap=use_mmap,
+        use_mlock=use_mlock,
+        n_batch=n_batch,
+        last_n_tokens_size=last_n_tokens_size,
+        low_vram=low_vram,
+        rope_freq_base=rope_freq_base,
+        rope_freq_scale=rope_freq_scale,
+
+
+
+    )
+    global_sys_prompt = load_presets_value(preset) + " " + system_prompt
+    temp = ""
+    input_prompt = f"[INST] <<SYS>>\n{global_sys_prompt}.\n<</SYS>>\n\n "
+    for interaction in history:
+        input_prompt = input_prompt + str(interaction[0]) + " [/INST] " + str(interaction[1]) + " </s><s> [INST] "
+
+    input_prompt = input_prompt + str(message) + " [/INST] "
+
+    output = llm(
+        input_prompt,
+        temperature=temperature,
+        top_p=top_p,
+        top_k=top_k, 
+        repeat_penalty=repeat_penalty,
+        max_tokens=max_tokens,
+        stop=[
+            "<|prompter|>",
+            "<|endoftext|>",
+            "<|endoftext|> \n",
+            "ASSISTANT:",
+            "USER:",
+            "SYSTEM:",
+        ],
+        stream=True,
+    )
+    for out in output:
+        stream = copy.deepcopy(out)
+        temp += stream["choices"][0]["text"]
+        yield temp
+
+    history = ["init", input_prompt]