Serving Decision Workloads with LLMs
One checkpoint, two paths: score choices from logits or generate chat. We evaluated SGLang across four models without fine-tuning or separate decision weights.
BFCL-derived tool selection: correct / attempted; median seconds over valid outputs.
| Checkpoint / weights | Decision correct | Chat-key correct | Chat-JSON correct | Chat-JSON valid | Decision median (s) | Chat-key median (s) | Chat-JSON median (s) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B · BF16 | 29/30 | 29/30 | 6/30 | 6/30 | 0.224 | 0.246 | 3.553 |
| Gemma-4-A4B · BF16 | 30/30 | 29/30 | 29/30 | 30/30 | 0.291 | 0.304 | 1.927 |
| GPT-OSS-120B · mixed MXFP4, experimental | 27/30 | 29/30 | 24/30 | 25/30 | 0.297 | 1.251 | 3.057 |
| Nemotron 3.5 Super VL · mixed NVFP4 | 30/30 | 30/30 | 13/30 | 13/30 | 0.431 | 0.535 | 2.704 |
Decision JSON vs chat JSON choice agreement (valid pairs only):
| Checkpoint | Decision / chat-JSON choice agreement | Valid-pair coverage |
|---|---|---|
| Qwen3.5-9B · BF16 | 100.0% (6/6) | 6/30 (20.0%) |
| Gemma-4-A4B · BF16 | 96.7% (29/30) | 30/30 (100.0%) |
| GPT-OSS-120B · mixed MXFP4, experimental | 88.0% (22/25) | 25/30 (83.3%) |
| Nemotron 3.5 Super VL · mixed NVFP4 | 100.0% (13/13) | 13/30 (43.3%) |
Agreement includes shared wrong answers, not equal probabilities. Low coverage matters. GPT-OSS uses experimental forced-final scoring that skips reasoning. These are small subsets, not official leaderboard scores.
Results and request logs · Full article · SGLang prefill-only work