Crash generating mnist-tests.gguf

[rebcabin]

└─(17:35:55 on main ✹ ✭)──> python mnist-tf.py convert_tests mnist-tests                                                                                ──(Wed,Mar13)─┘
uint8
uint8
gguf: This GGUF file is for Little Endian only
Traceback (most recent call last):
  File "/Users/brian/Dropbox/Mac/Documents/GitHub/mlc/examples/mnist/mnist-tf.py", line 113, in <module>
    convert_tests(sys.argv[2])
  File "/Users/brian/Dropbox/Mac/Documents/GitHub/mlc/examples/mnist/mnist-tf.py", line 95, in convert_tests
    gguf_writer.add_tensor("x_test", np.array(x_test, dtype=np.int8))
  File "/Users/brian/miniforge3/envs/tf/lib/python3.10/site-packages/gguf/gguf_writer.py", line 229, in add_tensor
    self.add_tensor_info(name, shape, tensor.dtype, tensor.nbytes, raw_dtype = raw_dtype)
  File "/Users/brian/miniforge3/envs/tf/lib/python3.10/site-packages/gguf/gguf_writer.py", line 199, in add_tensor_info
    raise ValueError("Only F32 and F16 tensors are supported for now")
ValueError: Only F32 and F16 tensors are supported for now
(tf) ┌─(~/Documents/GitHub/mlc/examples/mnist)────────────────────────────────────────────────────────────────────────────────────────────────────(brian@MacBook-Pro:s001)─┐
└─(17:35:59 on main ✹ ✭)──> ls -la                                                                                                                  1 ↵ ──(Wed,Mar13)─┘
total 448
drwxr-xr-x@ 10 brian  staff     320 Mar 13 17:35 .
drwxr-xr-x@  3 brian  staff      96 Mar 11 18:28 ..
-rw-r--r--@  1 brian  staff    8033 Mar 13 16:56 README.md
-rw-r--r--@  1 brian  staff     123 Mar 11 18:28 environment_tf.yml
-rw-r--r--@  1 brian  staff    7739 Mar 13 16:56 inference.py
drwxr-xr-x@  7 brian  staff     224 Mar 13 17:34 mnist-cnn-model
-rw-r--r--@  1 brian  staff  139712 Mar 13 17:35 mnist-cnn-model.gguf
-rw-r--r--@  1 brian  staff       0 Mar 13 17:35 mnist-tests.gguf
-rw-r--r--@  1 brian  staff    4097 Mar 13 16:56 mnist-tf.py
drwxr-xr-x@  3 brian  staff      96 Mar 12 10:11 mnist_inference
(tf) ┌─(~/Documents/GitHub/mlc/examples/mnist)────────────────────────────────────────────────────────────────────────────────────────────────────(brian@MacBook-Pro:s001)─┐
└─(17:36:14 on main ✹ ✭)──>                                          

[rebcabin]

The issue is that add_tensor_info does not support tensor_dtype Int8DType. I don't know how this could ever run.

[certik]

Yes, you need ggml-org/llama.cpp#6045 for this to work.

[rebcabin]

at the top level outside of mlc (to prevent nested .git repos), I did

conda activate tf
git clone https://github.com/certik/llama.cpp.git
cd llama.cpp
git checkout -t origin/gguf_writer
cd gguf-py
pip install .