Telegram bot chat wrapper

[innightwolfsleep]

Made an telegram chat bot uses llama.cpp + llama-cpp-python

It is a lightweight version of innightwolfsleep/text-generation-webui-telegram_bot

[gjmulder]

Sweet! Thanks @innightwolfsleep

I've forked rearranged stuff to make it easier to package the bot as a Docker container that includes NVidia GPU acceleration. It is a WIP, and I'm looking for feedback on where to take this next: llama-cpp-telegram_bot with GPU powered Docker

Functional changes

1. Added the BOT_API as an env variable instead of a file for security reasons
2. Hardcoded n_gpu_layers = 1000, to force maximal performance from the GPU (requires enough VRAM to hold the model, eg. 10GB for 13B).
3. Added a "Olivia Pope" and "Dr Seuss" personae

Docker build

$ git clone https://github.com/gjmulder/llama-cpp-telegram_bot
$ cd llama-cpp-telegram_bot
llama-cpp-telegram_bot$ git checkout olivia-dev
$ cd docker
docker$ docker build --no-cache --force-rm -t lct:olivia .

Sending build context to Docker daemon  4.096kB
Step 1/15 : FROM nvidia/cuda:12.1.1-devel-ubuntu20.04
 ---> 7e82987c23b0
Step 2/15 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Running in 1433c7f66562
Removing intermediate container 1433c7f66562
 ---> 4df9395103fa
Step 3/15 : ENV TZ=Etc/GMT
 ---> Running in 9ab9c23ff8c8
Removing intermediate container 9ab9c23ff8c8
 ---> 21a42f88ed24
Step 4/15 : RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip git cmake
 ---> Running in 325cf3653c0a
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [1010 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2228 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1046 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2693 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [28.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2366 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3167 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1342 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [31.2 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Fetched 27.4 MB in 34s (814 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  cmake-data git-man libarchive13 libbrotli1 libcurl3-gnutls libcurl4
  liberror-perl libexpat1 libgssapi-krb5-2 libicu66 libjsoncpp1 libk5crypto3
  libkeyutils1 libkrb5-3 libkrb5support0 libmpdec2 libnghttp2-14 libpsl5
  libpython3-stdlib libpython3.8-minimal libpython3.8-stdlib librhash0
  librtmp1 libssh-4 libuv1 libxml2 mime-support python-pip-whl
  python3-distutils python3-lib2to3 python3-minimal python3-pkg-resources
  python3-setuptools python3-wheel python3.8 python3.8-minimal tzdata
Suggested packages:
  cmake-doc ninja-build gettext-base git-daemon-run | git-daemon-sysvinit
  git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
  lrzip krb5-doc krb5-user python3-doc python3-tk python3-venv
  python-setuptools-doc python3.8-venv python3.8-doc binfmt-support
Recommended packages:
  less ssh-client krb5-locales publicsuffix file python3-dev
The following NEW packages will be installed:
  cmake cmake-data git git-man libarchive13 libbrotli1 libcurl3-gnutls
  libcurl4 liberror-perl libexpat1 libgssapi-krb5-2 libicu66 libjsoncpp1
  libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libmpdec2 libnghttp2-14
  libpsl5 libpython3-stdlib libpython3.8-minimal libpython3.8-stdlib librhash0
  librtmp1 libssh-4 libuv1 libxml2 mime-support python-pip-whl python3
  python3-distutils python3-lib2to3 python3-minimal python3-pip
  python3-pkg-resources python3-setuptools python3-wheel python3.8
  python3.8-minimal tzdata
0 upgraded, 41 newly installed, 0 to remove and 2 not upgraded.
Need to get 30.2 MB of archives.
After this operation, 141 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04.7 [717 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libexpat1 amd64 2.2.9-1ubuntu0.6 [74.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04.7 [1903 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-minimal amd64 3.8.2-0ubuntu2 [23.6 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 mime-support all 3.64ubuntu1 [30.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libmpdec2 amd64 2.4.2-3 [81.1 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04.7 [1675 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04.7 [387 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libpython3-stdlib amd64 3.8.2-0ubuntu2 [7068 B]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3 amd64 3.8.2-0ubuntu2 [47.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 tzdata all 2023c-0ubuntu0.20.04.1 [306 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libicu66 amd64 66.1-2ubuntu2.1 [8515 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2 amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [640 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1 [130 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libkrb5support0 amd64 1.17-6ubuntu4.3 [31.1 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libk5crypto3 amd64 1.17-6ubuntu4.3 [80.2 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libkeyutils1 amd64 1.6-6ubuntu1.1 [10.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libkrb5-3 amd64 1.17-6ubuntu4.3 [330 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgssapi-krb5-2 amd64 1.17-6ubuntu4.3 [121 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 libpsl5 amd64 0.21.0-1ubuntu1 [51.5 kB]
Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuv1 amd64 1.34.2-1ubuntu1.3 [80.8 kB]
Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 cmake-data all 3.16.3-1ubuntu1.20.04.1 [1613 kB]
Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libarchive13 amd64 3.4.0-2ubuntu1.2 [327 kB]
Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbrotli1 amd64 1.0.7-6ubuntu0.1 [267 kB]
Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 libnghttp2-14 amd64 1.40.0-1build1 [78.7 kB]
Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2build1 [54.9 kB]
Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssh-4 amd64 0.9.3-2ubuntu2.2 [170 kB]
Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.18 [236 kB]
Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 libjsoncpp1 amd64 1.7.4-3.1ubuntu2 [75.6 kB]
Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 librhash0 amd64 1.3.9-1 [113 kB]
Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 cmake amd64 3.16.3-1ubuntu1.20.04.1 [3668 kB]
Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl3-gnutls amd64 7.68.0-1ubuntu2.18 [233 kB]
Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 liberror-perl all 0.17029-1 [26.5 kB]
Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git-man all 1:2.25.1-1ubuntu3.11 [887 kB]
Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.11 [4605 kB]
Get:36 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.8 [1805 kB]
Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lib2to3 all 3.8.10-0ubuntu1~20.04 [76.3 kB]
Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-distutils all 3.8.10-0ubuntu1~20.04 [141 kB]
Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]
Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-wheel all 0.34.2-1ubuntu0.1 [23.9 kB]
Get:41 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.8 [231 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 30.2 MB in 3s (11.4 MB/s)
Selecting previously unselected package libpython3.8-minimal:amd64.
(Reading database ... 12855 files and directories currently installed.)
Preparing to unpack .../libpython3.8-minimal_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../libexpat1_2.2.9-1ubuntu0.6_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.9-1ubuntu0.6) ...
Selecting previously unselected package python3.8-minimal.
Preparing to unpack .../python3.8-minimal_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04.7) ...
Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up libexpat1:amd64 (2.2.9-1ubuntu0.6) ...
Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package python3-minimal.
(Reading database ... 13146 files and directories currently installed.)
Preparing to unpack .../0-python3-minimal_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3-minimal (3.8.2-0ubuntu2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../1-mime-support_3.64ubuntu1_all.deb ...
Unpacking mime-support (3.64ubuntu1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../2-libmpdec2_2.4.2-3_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-3) ...
Selecting previously unselected package libpython3.8-stdlib:amd64.
Preparing to unpack .../3-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package python3.8.
Preparing to unpack .../4-python3.8_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking python3.8 (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package libpython3-stdlib:amd64.
Preparing to unpack .../5-libpython3-stdlib_3.8.2-0ubuntu2_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.8.2-0ubuntu2) ...
Setting up python3-minimal (3.8.2-0ubuntu2) ...
Selecting previously unselected package python3.
(Reading database ... 13548 files and directories currently installed.)
Preparing to unpack .../00-python3_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3 (3.8.2-0ubuntu2) ...
Selecting previously unselected package tzdata.
Preparing to unpack .../01-tzdata_2023c-0ubuntu0.20.04.1_all.deb ...
Unpacking tzdata (2023c-0ubuntu0.20.04.1) ...
Selecting previously unselected package libicu66:amd64.
Preparing to unpack .../02-libicu66_66.1-2ubuntu2.1_amd64.deb ...
Unpacking libicu66:amd64 (66.1-2ubuntu2.1) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../03-libxml2_2.9.10+dfsg-5ubuntu0.20.04.6_amd64.deb ...
Unpacking libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../04-python3-pkg-resources_45.2.0-1ubuntu0.1_all.deb ...
Unpacking python3-pkg-resources (45.2.0-1ubuntu0.1) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-6ubuntu4.3_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-6ubuntu4.3) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-6ubuntu4.3_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-6ubuntu4.3) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../07-libkeyutils1_1.6-6ubuntu1.1_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6ubuntu1.1) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../08-libkrb5-3_1.17-6ubuntu4.3_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-6ubuntu4.3) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../09-libgssapi-krb5-2_1.17-6ubuntu4.3_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-6ubuntu4.3) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../10-libpsl5_0.21.0-1ubuntu1_amd64.deb ...
Unpacking libpsl5:amd64 (0.21.0-1ubuntu1) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../11-libuv1_1.34.2-1ubuntu1.3_amd64.deb ...
Unpacking libuv1:amd64 (1.34.2-1ubuntu1.3) ...
Selecting previously unselected package cmake-data.
Preparing to unpack .../12-cmake-data_3.16.3-1ubuntu1.20.04.1_all.deb ...
Unpacking cmake-data (3.16.3-1ubuntu1.20.04.1) ...
Selecting previously unselected package libarchive13:amd64.
Preparing to unpack .../13-libarchive13_3.4.0-2ubuntu1.2_amd64.deb ...
Unpacking libarchive13:amd64 (3.4.0-2ubuntu1.2) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../14-libbrotli1_1.0.7-6ubuntu0.1_amd64.deb ...
Unpacking libbrotli1:amd64 (1.0.7-6ubuntu0.1) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../15-libnghttp2-14_1.40.0-1build1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.40.0-1build1) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../16-librtmp1_2.4+20151223.gitfa8646d.1-2build1_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build1) ...
Selecting previously unselected package libssh-4:amd64.
Preparing to unpack .../17-libssh-4_0.9.3-2ubuntu2.2_amd64.deb ...
Unpacking libssh-4:amd64 (0.9.3-2ubuntu2.2) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../18-libcurl4_7.68.0-1ubuntu2.18_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.18) ...
Selecting previously unselected package libjsoncpp1:amd64.
Preparing to unpack .../19-libjsoncpp1_1.7.4-3.1ubuntu2_amd64.deb ...
Unpacking libjsoncpp1:amd64 (1.7.4-3.1ubuntu2) ...
Selecting previously unselected package librhash0:amd64.
Preparing to unpack .../20-librhash0_1.3.9-1_amd64.deb ...
Unpacking librhash0:amd64 (1.3.9-1) ...
Selecting previously unselected package cmake.
Preparing to unpack .../21-cmake_3.16.3-1ubuntu1.20.04.1_amd64.deb ...
Unpacking cmake (3.16.3-1ubuntu1.20.04.1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../22-libcurl3-gnutls_7.68.0-1ubuntu2.18_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.18) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../23-liberror-perl_0.17029-1_all.deb ...
Unpacking liberror-perl (0.17029-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../24-git-man_1%3a2.25.1-1ubuntu3.11_all.deb ...
Unpacking git-man (1:2.25.1-1ubuntu3.11) ...
Selecting previously unselected package git.
Preparing to unpack .../25-git_1%3a2.25.1-1ubuntu3.11_amd64.deb ...
Unpacking git (1:2.25.1-1ubuntu3.11) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../26-python-pip-whl_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python-pip-whl (20.0.2-5ubuntu1.8) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../27-python3-lib2to3_3.8.10-0ubuntu1~20.04_all.deb ...
Unpacking python3-lib2to3 (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../28-python3-distutils_3.8.10-0ubuntu1~20.04_all.deb ...
Unpacking python3-distutils (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../29-python3-setuptools_45.2.0-1ubuntu0.1_all.deb ...
Unpacking python3-setuptools (45.2.0-1ubuntu0.1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../30-python3-wheel_0.34.2-1ubuntu0.1_all.deb ...
Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../31-python3-pip_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python3-pip (20.0.2-5ubuntu1.8) ...
Setting up libkeyutils1:amd64 (1.6-6ubuntu1.1) ...
Setting up libpsl5:amd64 (0.21.0-1ubuntu1) ...
Setting up mime-support (3.64ubuntu1) ...
Setting up libbrotli1:amd64 (1.0.7-6ubuntu0.1) ...
Setting up libnghttp2-14:amd64 (1.40.0-1build1) ...
Setting up libkrb5support0:amd64 (1.17-6ubuntu4.3) ...
Setting up tzdata (2023c-0ubuntu0.20.04.1) ...

Current default time zone: 'Etc/UTC'
Local time is now:      Wed May 17 16:57:04 UTC 2023.
Universal Time is now:  Wed May 17 16:57:04 UTC 2023.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Setting up liberror-perl (0.17029-1) ...
Setting up libuv1:amd64 (1.34.2-1ubuntu1.3) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build1) ...
Setting up libk5crypto3:amd64 (1.17-6ubuntu4.3) ...
Setting up librhash0:amd64 (1.3.9-1) ...
Setting up git-man (1:2.25.1-1ubuntu3.11) ...
Setting up python-pip-whl (20.0.2-5ubuntu1.8) ...
Setting up cmake-data (3.16.3-1ubuntu1.20.04.1) ...
Setting up libkrb5-3:amd64 (1.17-6ubuntu4.3) ...
Setting up libmpdec2:amd64 (2.4.2-3) ...
Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up python3.8 (3.8.10-0ubuntu1~20.04.7) ...
Setting up libjsoncpp1:amd64 (1.7.4-3.1ubuntu2) ...
Setting up libpython3-stdlib:amd64 (3.8.2-0ubuntu2) ...
Setting up libicu66:amd64 (66.1-2ubuntu2.1) ...
Setting up python3 (3.8.2-0ubuntu2) ...
Setting up python3-wheel (0.34.2-1ubuntu0.1) ...
Setting up libgssapi-krb5-2:amd64 (1.17-6ubuntu4.3) ...
Setting up libssh-4:amd64 (0.9.3-2ubuntu2.2) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.18) ...
Setting up python3-lib2to3 (3.8.10-0ubuntu1~20.04) ...
Setting up libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ...
Setting up python3-pkg-resources (45.2.0-1ubuntu0.1) ...
Setting up python3-distutils (3.8.10-0ubuntu1~20.04) ...
Setting up python3-setuptools (45.2.0-1ubuntu0.1) ...
Setting up libarchive13:amd64 (3.4.0-2ubuntu1.2) ...
Setting up libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.18) ...
Setting up git (1:2.25.1-1ubuntu3.11) ...
Setting up python3-pip (20.0.2-5ubuntu1.8) ...
Setting up cmake (3.16.3-1ubuntu1.20.04.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Removing intermediate container 325cf3653c0a
 ---> d6dd671b3e5f
Step 5/15 : WORKDIR /usr/src
 ---> Running in 7074962f8e25
Removing intermediate container 7074962f8e25
 ---> 7ce98461701e
Step 6/15 : RUN git clone https://github.com/abetlen/llama-cpp-python.git
 ---> Running in d6643ee05282
Cloning into 'llama-cpp-python'...
Removing intermediate container d6643ee05282
 ---> a7fe0e78ec48
Step 7/15 : WORKDIR /usr/src/llama-cpp-python
 ---> Running in b8a6d3de4dc9
Removing intermediate container b8a6d3de4dc9
 ---> 96b8ff67b8d6
Step 8/15 : RUN sed -i 's|[email protected]:ggerganov/llama.cpp.git|https://github.com/ggerganov/llama.cpp.git|' .gitmodules
 ---> Running in 4c497e5882be
Removing intermediate container 4c497e5882be
 ---> bcfae11bcece
Step 9/15 : RUN git submodule update --init --recursive
 ---> Running in 09cc76246383
Submodule 'vendor/llama.cpp' (https://github.com/ggerganov/llama.cpp.git) registered for path 'vendor/llama.cpp'
Cloning into '/usr/src/llama-cpp-python/vendor/llama.cpp'...
Submodule path 'vendor/llama.cpp': checked out 'b5c9295eef2b56e307393b35b3a923e3518d226e'
Removing intermediate container 09cc76246383
 ---> 5ecd60f4372f
Step 10/15 : RUN pip install scikit-build fastapi sse_starlette uvicorn && LLAMA_CUBLAS=1 python3 setup.py install --verbose
 ---> Running in 4cf0658bc2da
Collecting scikit-build
  Downloading scikit_build-0.17.5-py3-none-any.whl (82 kB)
Collecting fastapi
  Downloading fastapi-0.95.2-py3-none-any.whl (56 kB)
Collecting sse_starlette
  Downloading sse_starlette-1.6.0-py3-none-any.whl (9.5 kB)
Collecting uvicorn
  Downloading uvicorn-0.22.0-py3-none-any.whl (58 kB)
Collecting packaging
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
Requirement already satisfied: wheel>=0.32.0 in /usr/lib/python3/dist-packages (from scikit-build) (0.34.2)
Collecting tomli; python_version < "3.11"
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Requirement already satisfied: setuptools>=42.0.0 in /usr/lib/python3/dist-packages (from scikit-build) (45.2.0)
Collecting distro
  Downloading distro-1.8.0-py3-none-any.whl (20 kB)
Collecting starlette<0.28.0,>=0.27.0
  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)
Collecting pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2
  Downloading pydantic-1.10.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting click>=7.0
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
Collecting h11>=0.8
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
Collecting typing-extensions>=3.10.0; python_version < "3.10"
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting anyio<5,>=3.4.0
  Downloading anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting sniffio>=1.1
  Downloading sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting idna>=2.8
  Downloading idna-3.4-py3-none-any.whl (61 kB)
Installing collected packages: packaging, tomli, distro, scikit-build, typing-extensions, sniffio, idna, anyio, starlette, pydantic, fastapi, sse-starlette, click, h11, uvicorn
Successfully installed anyio-3.6.2 click-8.1.3 distro-1.8.0 fastapi-0.95.2 h11-0.14.0 idna-3.4 packaging-23.1 pydantic-1.10.7 scikit-build-0.17.5 sniffio-1.3.0 sse-starlette-1.6.0 starlette-0.27.0 tomli-2.0.1 typing-extensions-4.5.0 uvicorn-0.22.0


--------------------------------------------------------------------------------
-- Trying 'Ninja' generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- Configuring incomplete, errors occurred!
See also "/usr/src/llama-cpp-python/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Ninja' generator - failure
--------------------------------------------------------------------------------



--------------------------------------------------------------------------------
-- Trying 'Unix Makefiles' generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/src/llama-cpp-python/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Unix Makefiles' generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /usr/src/llama-cpp-python/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    /usr/bin/cmake /usr/src/llama-cpp-python -G 'Unix Makefiles' --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/usr/src/llama-cpp-python/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_VERSION_STRING:STRING=3.8.10 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/local/lib/python3.8/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DCMAKE_BUILD_TYPE:STRING=Release

Not searching for unused variables given on the command line.
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning (dev) in CMakeLists.txt:
  A logical block opening on the line

    /usr/src/llama-cpp-python/CMakeLists.txt:9 (if)

  closes on the line

    /usr/src/llama-cpp-python/CMakeLists.txt:31 (endif)

  with mis-matching arguments.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
-- Build files have been written to: /usr/src/llama-cpp-python/_skbuild/linux-x86_64-3.8/cmake-build
Scanning dependencies of target run
[100%] Generating ../../../vendor/llama.cpp/libllama.so
I llama.cpp build info: 
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I LDFLAGS:   -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
I CC:       cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
I CXX:      g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

llama.cpp: In function ‘size_t llama_set_state_data(llama_context*, const uint8_t*)’:
llama.cpp:2686:36: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2686 |             kin3d->data = (void *) inp;
      |                                    ^~~
llama.cpp:2690:36: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2690 |             vin3d->data = (void *) inp;
      |                                    ^~~
[100%] Built target run
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/src/llama-cpp-python/_skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/libllama.so

copying llama_cpp/__init__.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/__init__.py
copying llama_cpp/llama_cpp.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama_cpp.py
copying llama_cpp/llama_types.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama_types.py
copying llama_cpp/llama.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama.py
creating directory _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server
copying llama_cpp/server/__init__.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/__init__.py
copying llama_cpp/server/app.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/app.py
copying llama_cpp/server/__main__.py -> _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/__main__.py

running install
running bdist_egg
running egg_info
creating llama_cpp_python.egg-info
writing llama_cpp_python.egg-info/PKG-INFO
writing dependency_links to llama_cpp_python.egg-info/dependency_links.txt
writing requirements to llama_cpp_python.egg-info/requires.txt
writing top-level names to llama_cpp_python.egg-info/top_level.txt
writing manifest file 'llama_cpp_python.egg-info/SOURCES.txt'
reading manifest file 'llama_cpp_python.egg-info/SOURCES.txt'
writing manifest file 'llama_cpp_python.egg-info/SOURCES.txt'
installing library code to _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg
running install_lib
running build_py
creating _skbuild/linux-x86_64-3.8/setuptools/lib
creating _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama_cpp.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama_types.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/llama.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
creating _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/app.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/server/__main__.py -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/cmake-install/llama_cpp/libllama.so -> _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp
copied 7 files
running build_ext
creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64
creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg
creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/llama_cpp.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server/__init__.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server/app.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/server/__main__.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/llama_types.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/libllama.so -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
copying _skbuild/linux-x86_64-3.8/setuptools/lib/llama_cpp/llama.py -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp
copied 8 files
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/__init__.py to __init__.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/llama_cpp.py to llama_cpp.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server/__init__.py to __init__.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server/app.py to app.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/server/__main__.py to __main__.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/llama_types.py to llama_types.cpython-38.pyc
byte-compiling _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/llama_cpp/llama.py to llama.cpython-38.pyc
creating _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/PKG-INFO -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/SOURCES.txt -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/dependency_links.txt -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/not-zip-safe -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/requires.txt -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
copying llama_cpp_python.egg-info/top_level.txt -> _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO
writing _skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
creating dist
creating 'dist/llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg' and adding '_skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg' to it
removing '_skbuild/linux-x86_64-3.8/setuptools/bdist.linux-x86_64/egg' (and everything under it)
Processing llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg
creating /usr/local/lib/python3.8/dist-packages/llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg
Extracting llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg to /usr/local/lib/python3.8/dist-packages
Adding llama-cpp-python 0.1.50 to easy-install.pth file

Installed /usr/local/lib/python3.8/dist-packages/llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg
Processing dependencies for llama-cpp-python==0.1.50
Searching for typing-extensions==4.5.0
Best match: typing-extensions 4.5.0
Adding typing-extensions 4.5.0 to easy-install.pth file

Using /usr/local/lib/python3.8/dist-packages
Finished processing dependencies for llama-cpp-python==0.1.50
Removing intermediate container 4cf0658bc2da
 ---> 3ef5eecd372b
Step 11/15 : WORKDIR /usr/src/
 ---> Running in 5d6754dd8315
Removing intermediate container 5d6754dd8315
 ---> 84d02da451d7
Step 12/15 : RUN git clone https://github.com/gjmulder/llama-cpp-telegram_bot.git
 ---> Running in ded6d72859a5
Cloning into 'llama-cpp-telegram_bot'...
Removing intermediate container ded6d72859a5
 ---> 760b1660e1e2
Step 13/15 : WORKDIR /usr/src/llama-cpp-telegram_bot
 ---> Running in be0c31d26353
Removing intermediate container be0c31d26353
 ---> 09a0a8a0080a
Step 14/15 : RUN git checkout olivia-dev && pip install -r requirements.txt
 ---> Running in f91506aee151
Switched to a new branch 'olivia-dev'
Branch 'olivia-dev' set up to track remote branch 'olivia-dev' from 'origin'.
Collecting python-telegram-bot==13.15
  Downloading python_telegram_bot-13.15-py3-none-any.whl (519 kB)
Collecting pyyaml
  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting deep-translator==1.9.2
  Downloading deep_translator-1.9.2-py3-none-any.whl (30 kB)
Requirement already satisfied: llama-cpp-python in /usr/local/lib/python3.8/dist-packages/llama_cpp_python-0.1.50-py3.8-linux-x86_64.egg (from -r requirements.txt (line 4)) (0.1.50)
Collecting pytz>=2018.6
  Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting APScheduler==3.6.3
  Downloading APScheduler-3.6.3-py2.py3-none-any.whl (58 kB)
Collecting tornado==6.1
  Downloading tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl (427 kB)
Collecting certifi
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
Collecting cachetools==4.2.2
  Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting beautifulsoup4<5.0.0,>=4.9.1
  Downloading beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)
Collecting requests<3.0.0,>=2.23.0
  Downloading requests-2.30.0-py3-none-any.whl (62 kB)
Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.8/dist-packages (from llama-cpp-python->-r requirements.txt (line 4)) (4.5.0)
Requirement already satisfied: setuptools>=0.7 in /usr/lib/python3/dist-packages (from APScheduler==3.6.3->python-telegram-bot==13.15->-r requirements.txt (line 1)) (45.2.0)
Collecting tzlocal>=1.2
  Downloading tzlocal-5.0.1-py3-none-any.whl (20 kB)
Collecting six>=1.4.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.4.1-py3-none-any.whl (36 kB)
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.23.0->deep-translator==1.9.2->-r requirements.txt (line 3)) (3.4)
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.0.2-py3-none-any.whl (123 kB)
Collecting backports.zoneinfo; python_version < "3.9"
  Downloading backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB)
Installing collected packages: pytz, backports.zoneinfo, tzlocal, six, APScheduler, tornado, certifi, cachetools, python-telegram-bot, pyyaml, soupsieve, beautifulsoup4, charset-normalizer, urllib3, requests, deep-translator
Successfully installed APScheduler-3.6.3 backports.zoneinfo-0.2.1 beautifulsoup4-4.12.2 cachetools-4.2.2 certifi-2023.5.7 charset-normalizer-3.1.0 deep-translator-1.9.2 python-telegram-bot-13.15 pytz-2023.3 pyyaml-6.0 requests-2.30.0 six-1.16.0 soupsieve-2.4.1 tornado-6.1 tzlocal-5.0.1 urllib3-2.0.2
Removing intermediate container f91506aee151
 ---> c28c74aeeddc
Step 15/15 : CMD bash -c "ulimit -l unlimited && python3 main.py"
 ---> Running in e68f9fa77112
Removing intermediate container e68f9fa77112
 ---> a0a08b56914c
Successfully built a0a08b56914c
Successfully tagged lct:olivia

Review ./docker/run.sh, install your model as model.bin, fix volume mount path, and export your BOT_API token for your Telegram Bot

docker$ ls -l ../var/models
total 9534872
-rw-rw-r-- 1 mulderg mulderg 9763701888 May 17 16:06 model.bin
-rw-rw-r-- 1 mulderg mulderg          0 May 17 12:33 place_models_here.txt

docker$ ./run.sh lct:olivia

==========
== CUDA ==
==========

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

llama.cpp: loading model from /usr/src/llama-cpp-telegram_bot/models/model.bin
llama_model_load_internal: format     = ggjt v2 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 8196
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =  90.75 KB
llama_model_load_internal: mem required  = 11359.05 MB (+ 1608.00 MB per state)
llama_model_load_internal: [cublas] offloading 40 layers to GPU
llama_model_load_internal: [cublas] offloading output layer to GPU
llama_model_load_internal: [cublas] total VRAM used: 9192 MB
llama_init_from_file: kv self size  = 6403.12 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 

llama_print_timings:        load time =  2839.70 ms
llama_print_timings:      sample time =     7.81 ms /    11 runs   (    0.71 ms per token)
llama_print_timings: prompt eval time =   679.64 ms /    33 tokens (   20.60 ms per token)
llama_print_timings:        eval time =  1842.84 ms /    10 runs   (  184.28 ms per token)
llama_print_timings:       total time =  3057.65 ms

llama_print_timings:        load time =  2839.70 ms
llama_print_timings:      sample time =     9.84 ms /    14 runs   (    0.70 ms per token)
llama_print_timings: prompt eval time =   605.38 ms /    15 tokens (   40.36 ms per token)
llama_print_timings:        eval time =  2401.41 ms /    13 runs   (  184.72 ms per token)
llama_print_timings:       total time =  3684.56 ms

llama_print_timings:        load time =  2839.70 ms
llama_print_timings:      sample time =    20.01 ms /    28 runs   (    0.71 ms per token)
llama_print_timings: prompt eval time =   606.21 ms /    18 tokens (   33.68 ms per token)
llama_print_timings:        eval time =  5098.20 ms /    27 runs   (  188.82 ms per token)
llama_print_timings:       total time =  7079.92 ms

llama_print_timings:        load time =  2839.70 ms
llama_print_timings:      sample time =    14.96 ms /    21 runs   (    0.71 ms per token)
llama_print_timings: prompt eval time =   675.24 ms /    32 tokens (   21.10 ms per token)
llama_print_timings:        eval time =  3765.59 ms /    20 runs   (  188.28 ms per token)
llama_print_timings:       total time =  5467.93 ms

llama_print_timings:        load time =  2839.70 ms
llama_print_timings:      sample time =     9.67 ms /    14 runs   (    0.69 ms per token)
llama_print_timings: prompt eval time =   661.97 ms /    25 tokens (   26.48 ms per token)
llama_print_timings:        eval time =  2457.45 ms /    13 runs   (  189.03 ms per token)
llama_print_timings:       total time =  3811.44 ms

Telegram output running on a GTX 1080Ti

Olivia Pope

Dr Seuss