Back to pull request #6811

`grammars`: cache decoded token codepoints for faster sampling #2196

Sign in to view logs

Run time

Learn about OS pricing on GitHub Actions

Job	Run time
bench-server-baseline (phi-2, q4_0)	15m 31s
bench-server-baseline (phi-2, q8_0)	15m 33s
bench-server-baseline (phi-2, f16)	15m 40s
	46m 44s