Use a custom batch size threshold for using a batch-specific kernel.

MrMage · Apr 25, 2024 · ae003f6 · ae003f6
1 parent e96ae25
commit ae003f6
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/ggml-metal.m b/ggml-metal.m
@@ -1421,7 +1421,7 @@ static enum ggml_status ggml_metal_graph_compute(
 
                         // find the break-even point where the matrix-matrix kernel becomes more efficient compared
                         // to the matrix-vector kernel
-                        int ne11_mm_min = 1;
+                        int ne11_mm_min = 6;
 
 #if 0
                         // the numbers below are measured on M2 Ultra for 7B and 13B models