Improving Multi-candidate Speculative Decoding

News

pip install -r requirements.txt

dynamic_mcsd_inference.py : Dynamic MCSD Configuration Inference (ignoring quality)
static_ti_mcsd_inference.py: Static MCSD Configuration Inference (considering quality)

All results and figure generation code are stored in Dyna/results/ folder

Name		Name	Last commit message	Last commit date
Latest commit History 164 Commits
data/mt_bench		data/mt_bench
decision_model		decision_model
evaluation		evaluation
inference		inference
model/llama_tree_attn		model/llama_tree_attn
results		results
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
dynamic_mcsd_inference.py		dynamic_mcsd_inference.py
requirements.txt		requirements.txt
static_ti_mcsd_inferece.py		static_ti_mcsd_inferece.py