Decode tok/s by model
Highest decode throughput across all builds and quants per model, measured on a free GitHub Actions arm64 runner.
| # | Model | Best config | Size GB | Prefill t/s | Decode t/s | TTFT ms | KleidiAI gain | $/Mtok (free host) |
|---|---|---|---|---|---|---|---|---|
| 1 | SmolLM2-360M-Instruct | Q8_0 kleidiai-on @ 4t | 0.36 | 327.91 | 152.83 | 1561.4 | 1.18x | 0.00 (free) |
| 2 | granite-4.0-350m | Q8_0 kleidiai-on @ 4t | 0.352 | 408.03 | 151.49 | 1254.8 | 1.17x | 0.00 (free) |
| 3 | Qwen2.5-0.5B-Instruct | Q8_0 kleidiai-on @ 4t | 0.495 | 466.1 | 125.14 | 1098.5 | 1.1x | 0.00 (free) |
| 4 | Qwen3-0.6B | Q8_0 kleidiai-on @ 4t | 0.749 | 204.92 | 88.96 | 2498.5 | 1.14x | 0.00 (free) |
| 5 | LFM2-700M | Q8_0 kleidiai-on @ 4t | 0.737 | 588.75 | 88.3 | 869.6 | 1.2x | 0.00 (free) |
| 6 | Qwen2.5-1.5B-Instruct | Q8_0 kleidiai-on @ 4t | 1.533 | 186.27 | 44.55 | 2748.6 | 1.15x | 0.00 (free) |
| 7 | SmolLM2-1.7B-Instruct | Q8_0 kleidiai-on @ 4t | 1.695 | 172.66 | 37.58 | 2965.3 | 1.15x | 0.00 (free) |
| 8 | gemma-4-E2B-it | Q4_0 kleidiai-off @ 4t | 2.833 | 46.23 | 23.39 | 11075.9 | 1.0x | 0.00 (free) |
| 9 | Qwen2.5-3B-Instruct | Q8_0 kleidiai-on @ 4t | 3.06 | 107.72 | 23.28 | 4752.9 | 1.15x | 0.00 (free) |
| 10 | Phi-3.5-mini-instruct | Q4_0 kleidiai-on @ 4t | 2.027 | 54.38 | 18.57 | 9415.3 | 1.05x | 0.00 (free) |
| 11 | gemma-4-E4B-it | Q4_0 kleidiai-off @ 4t | 4.504 | 27.96 | 12.53 | 18313.3 | 1.0x | 0.00 (free) |
| 12 | Qwythos-9B | Q4_K_M kleidiai-on @ 4t | 5.243 | 21.63 | 6.49 | 23672.0 | 1.0x | 0.00 (free) |
| 13 | gemma-4-12B-it | Q4_0 kleidiai-on @ 4t | 6.276 | 16.43 | 5.97 | 31162.1 | 1.03x | 0.00 (free) |
Generated by kleidibench leaderboard · per-model
detail in each report.html · github.com/yannan000/kleidibench