Follow-up from ikawrakow/ik_llama.cpp#15 (reply in thread).

Using GGUF as the format for imatrix files will be useful for further experiments (e.g. with L²QER) and compatibility with existing or future GGUF tooling (e.g. GGUF previews on HuggingFace, graphical GGUF viewer(s) #6715, some kind of gguf-diff, etc.).

There are multiple problems with imatrix which this is addressing:

• Ad-hoc format which isn't really readable by other projects (and which has no way to backward-compatibly be extended except by adding more stuff at the end)
• Non-deterministic tensor order depending on unordered_map iteration order (makes sha256sum useless to compare imatrix files made on the same dataset)
• Broken behavior at small -ub (intermediate saves happen waaay too often)
• Can't use bigger batch size than chunk size

Summary of changes

• Use GGUF to store imatrix data.
  • general.type is imatrix
  • no general.architecture
    • can't really know the architecture from old imatrix files.
  • store *.sums and *.counts for each tensors with imatrix data.
    • *.sums are the sums of activations
      • Stored in F32, like before.
    • *.counts are the number of activations (also the number of tokens), useful to calculate the mean
      • Why not simply store the mean? To allow merging imatrix files together with --in-file.
      • It's stored in F32 even though it's integer values, because when calculating the mean it would be converted to F32 anyway to perform the division.
• Add convert_legacy_imatrix_to_gguf.py to convert old imatrix.dat files to imatrix.gguf
• Like llama-perplexity since perplexity : support using multiple sequences to allow larger batch sizes #5946, allow computing multiple chunks per batch with llama-imatrix
  • This should be useful for huge models like Llama-405B when they don't fit completely in RAM.
• Use fused-multiply-add (with std::fma) when accumulating the sums of activations
  • Shouldn't hurt to somewhat reduce rounding errors
    • (obviously f64 would be even better, but I'm not use it's worth it yet. For the curious, using double for the intermediate accumulations can be tried by changing only one line in IMatrixStats: vector<float> values to vector<double> values.)
• Sort the tensor names before serializing
  • This makes the tensor order deterministic, because otherwise it depended on the iteration order of unordered_map.
    • Determinism between runs means sha256sum can be meaningfully used to compare imatrix files generated in very similar conditions.

TODO

• Compare old llama-quantize with old imatrix.dat with new llama-quantize using converted imatrix.gguf
  • Seemed to work, but need to re-test. The resulting quantized model(s) should have the same sha256sum.
• Test new llama-imatrix at different batch sizes
  • Same checksums with -ub 64 -b 512 and -ub 512 -b 2048 for a chunk size of 512 (-c 512)
• Perplexity test(s) with i-quants with old llama-imatrix vs new llama-imatrix
• Test with MoE models (perplexity with i-quants should be in the same ballpark as before)
• Test --in-file with llama-imatrix
• (maybe) Implement cleaner general.architecture exclusion.
  • Currently, this uses a subclass to make self.add_architecture() a no-op, but maybe general.architecture should simply be excluded when self.arch == "". Not sure how to prevent using the other self.add_* (in GGUFWriter) which expect self.arch to be something.
  • Or maybe the architecture should be included?
    • What about conversions from older imatrix.dat files?

• I have read the contributing guidelines
• Self-reported review complexity:
  • Medium