Serving Decision Workloads with LLMs: One Checkpoint for Chat and Choices
A model that generates text already computes vocabulary logits. When an application needs a choice from a fixed set, it can read scores at answer positions instead of generating a response and parsing it.
We evaluated that distinction in SGLang using original checkpoints from Qwen, Gemma, OpenAI and NVIDIA. The deployment serves chat and decision requests from the same resident model weights. No learned decision head, classifier training or fine-tuning was added.
How the two paths work
SGLang’s prefill-only roadmap treats scoring and other non-generative workloads as a first-class execution path, with work on batching, cache reuse and avoiding unnecessary decode. It credits LinkedIn and TikTok contributor teams. Our work evaluates application paths in that serving ecosystem; it does not claim authorship of the upstream optimizations.
In the tested deployments, /v1/chat/completions generates text and native /v1/systemone returns structured decisions. The application selects a path per request. Input processing, buffers and cache still consume compute and memory. The current SGLang server source may differ from the tested versions.
We compared three outputs:
- Decision: a fixed choice and logit-derived candidate scores serialized by server code, without generating answer tokens in the completed runs.
- Chat key: a generated option key.
- Chat score JSON: a generated choice and probability dictionary.
Generated probabilities are self-reported estimates. Logit-derived probabilities depend on prompts, labels and normalization. They have different meanings; we did not establish calibration for either.
Checkpoints and evaluation design
Qwen3.5-9B is a dense hybrid-attention model; Gemma-4-26B-A4B and GPT-OSS-120B are MoE models; Nemotron 3.5 Super VL is a Mamba/Transformer hybrid MoE. Qwen and Gemma used BF16. GPT-OSS retained native MXFP4 expert weights with higher precision elsewhere, while Nemotron retained native mixed NVFP4.
The primary comparisons ran on the local NVIDIA GB10. Different checkpoints, precision, engines and reasoning behavior prevent isolating architecture effects. The Nemotron full comparison used text; these results do not evaluate vision.
The paired suites contain 840 requests across small tasks, BFCL-derived tool selection and RouteLLM-derived routing. Small tasks repeat the same examples across passes. Context, question and options were paired, while prompt rendering and computation differ by path. Reference answers were excluded from inference prompts.
Tool selection: validity changes the story
Correct counts include every attempt. Latency medians use only valid outputs. The columns below are calculated by explicitly selecting each path, independently of request order.
| Checkpoint / weights | Decision correct | Chat-key correct | Chat-JSON correct | Chat-JSON valid | Decision median (s) | Chat-key median (s) | Chat-JSON median (s) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B · BF16 | 29/30 | 29/30 | 6/30 | 6/30 | 0.224 | 0.246 | 3.553 |
| Gemma-4-A4B · BF16 | 30/30 | 29/30 | 29/30 | 30/30 | 0.291 | 0.304 | 1.927 |
| GPT-OSS-120B · mixed MXFP4, experimental | 27/30 | 29/30 | 24/30 | 25/30 | 0.297 | 1.251 | 3.057 |
| Nemotron 3.5 Super VL · mixed NVFP4 | 30/30 | 30/30 | 13/30 | 13/30 | 0.431 | 0.535 | 2.704 |
Decisions avoid generating the score dictionary. Against key-only chat, the time savings are generally smaller. Score-JSON formatting failures mean a valid-only latency can describe a small surviving subset; read it together with validity.
This BFCL-derived subset tests which tool to select, including irrelevance cases. It does not test generated arguments, tool execution, parallel calls or overall BFCL performance.
Do the JSON paths agree?
We pair the same case and pass and compare the selected choices when both responses are valid.
| Checkpoint | Decision / chat-JSON choice agreement | Valid-pair coverage |
|---|---|---|
| Qwen3.5-9B · BF16 | 100.0% (6/6) | 6/30 (20.0%) |
| Gemma-4-A4B · BF16 | 96.7% (29/30) | 30/30 (100.0%) |
| GPT-OSS-120B · mixed MXFP4, experimental | 88.0% (22/25) | 25/30 (83.3%) |
| Nemotron 3.5 Super VL · mixed NVFP4 | 100.0% (13/13) | 13/30 (43.3%) |
Agreement includes matching wrong answers. It does not establish equal probabilities, calibrated confidence or equivalent computation. High agreement with low coverage describes only the surviving valid pairs.
The Space index reports this comparison by model and suite, alongside counts of matching wrong answers. Model summaries include all pairwise path comparisons, and request logs expose the underlying payloads.
GPT-OSS requires an experimental path
Native decisions rejected GPT-OSS’s always-reasoning template. The successful experiment used /v1/experimental/final_decision, which forces a verified Harmony final-channel prefix and scores single-token choices through the existing engine without generating reasoning. Chat retained low reasoning effort.
This is an experimental adapter, not native decision support or an officially supported reasoning-off mode. Its time saving includes skipping work that chat performs. On the selection subset, decisions were less accurate than key-only chat. Rejected native attempts remain in the local backup archive; their fast error responses are not successful inference latency.
Small tasks and RouteLLM-derived routing
| Checkpoint | Small: decision correct | Small: key correct | RouteLLM: decision correct | RouteLLM: key correct |
|---|---|---|---|---|
| Qwen3.5-9B · BF16 | 18/20 | 18/20 | 7/20 | 6/20 |
| Gemma-4-A4B · BF16 | 20/20 | 20/20 | 7/20 | 8/20 |
| GPT-OSS-120B · mixed MXFP4, experimental | 20/20 | 20/20 | 10/20 | 7/20 |
| Nemotron 3.5 Super VL · mixed NVFP4 | 20/20 | 20/20 | 8/20 | 8/20 |
Repeated small tasks are not independent examples. RouteLLM-derived labels concern an older strong/weak model pair and include ties. GPT-OSS decision accuracy equals a constant-route baseline on the balanced subset. High agreement between paths does not establish good routing.
RouterArena: no aggregate quality improvement over a constant target
The RouterArena-derived evaluation contains 1200 requests, using deterministic domain/difficulty-stratified queries and officially cached outcomes for GPT-4o-mini, Claude 3 Haiku and Gemini 2.0 Flash. Evaluation-only outcomes stayed out of prompts. We did not run new hosted target inference.
| Router | Decision valid | Fully correct target outcomes | Mean target quality | Mean regret | Decision median (s) | Key median (s) | JSON median (s) |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B · BF16 | 100/100 | 54/100 | 0.571 | 0.080 | 0.207 | 0.240 | 6.471 |
| Gemma-4-A4B · BF16 | 100/100 | 54/100 | 0.571 | 0.080 | 0.264 | 0.291 | 1.647 |
| GPT-OSS-120B · mixed MXFP4, experimental | 100/100 | 54/100 | 0.571 | 0.080 | 0.272 | 1.577 | 2.742 |
| Nemotron 3.5 Super VL · mixed NVFP4 | 100/100 | 54/100 | 0.571 | 0.080 | 0.370 | 0.470 | 2.168 |
“Fully correct” means the selected target’s cached quality equals one. Mean quality includes partial scores; regret measures lost quality relative to the best cached target for each query. These are not answers generated by the local router. Quality averages and latency medians use valid outputs. Historical target costs exclude the unmeasured dollar cost of local routing.
The decision selection counts show why baseline context matters:
| Router | GPT-4o-mini | Claude 3 Haiku | Gemini 2.0 Flash |
|---|---|---|---|
| Qwen3.5-9B · BF16 | 0 | 0 | 100 |
| Gemma-4-A4B · BF16 | 0 | 0 | 100 |
| GPT-OSS-120B · mixed MXFP4, experimental | 0 | 0 | 100 |
| Nemotron 3.5 Super VL · mixed NVFP4 | 8 | 0 | 92 |
Qwen, Gemma and GPT-OSS always chose Gemini. Nemotron varied its selections but obtained the same aggregate quality and regret. None demonstrated aggregate quality improvement over the constant-Gemini baseline. All-zero target outcomes are not routing successes just because every target matches the oracle.
Explore and reproduce the results
The static Space has a quick summary table with Summary | Request log links for each model. Each model has a result.html and log.html grouping its saved runs. Historical failures and retries remain in a separate local backup. No GPU or live backend is needed to browse.
All numerical results tables in this article and the brief Hub post are generated by a script from the retained request records. Editable templates contain the explanatory text; rebuilding calculates the results rather than manually transcribing numbers. Source paths and hashes identify the inputs used by the Space.
Explore the live results Space.
These are small offline diagnostics. We did not establish calibrated confidence, training-data independence, production throughput, mixed-load stability or a universal latency advantage. The supported finding is narrower: compatible checkpoints can serve both chat and structured decisions, and scoring fixed choices can avoid generation work when an application only needs a decision.