SHARED WEIGHTS / EXPERIMENT NOTEBOOK

Blog post preview

Local preview of the publication draft. Hugging Face typography may differ.

Serving Decision Workloads with LLMs: One Checkpoint for Chat and Choices

A model that generates text already computes vocabulary logits. When an application needs a choice from a fixed set, it can read scores at answer positions instead of generating a response and parsing it.

We evaluated that distinction in SGLang using original checkpoints from Qwen, Gemma, OpenAI and NVIDIA. The deployment serves chat and decision requests from the same resident model weights. No learned decision head, classifier training or fine-tuning was added.

How the two paths work

SGLang’s prefill-only roadmap treats scoring and other non-generative workloads as a first-class execution path, with work on batching, cache reuse and avoiding unnecessary decode. It credits LinkedIn and TikTok contributor teams. Our work evaluates application paths in that serving ecosystem; it does not claim authorship of the upstream optimizations.

In the tested deployments, /v1/chat/completions generates text and native /v1/systemone returns structured decisions. The application selects a path per request. Input processing, buffers and cache still consume compute and memory. The current SGLang server source may differ from the tested versions.

We compared three outputs:

Generated probabilities are self-reported estimates. Logit-derived probabilities depend on prompts, labels and normalization. They have different meanings; we did not establish calibration for either.

Checkpoints and evaluation design

Qwen3.5-9B is a dense hybrid-attention model; Gemma-4-26B-A4B and GPT-OSS-120B are MoE models; Nemotron 3.5 Super VL is a Mamba/Transformer hybrid MoE. Qwen and Gemma used BF16. GPT-OSS retained native MXFP4 expert weights with higher precision elsewhere, while Nemotron retained native mixed NVFP4.

The primary comparisons ran on the local NVIDIA GB10. Different checkpoints, precision, engines and reasoning behavior prevent isolating architecture effects. The Nemotron full comparison used text; these results do not evaluate vision.

The paired suites contain 840 requests across small tasks, BFCL-derived tool selection and RouteLLM-derived routing. Small tasks repeat the same examples across passes. Context, question and options were paired, while prompt rendering and computation differ by path. Reference answers were excluded from inference prompts.

Tool selection: validity changes the story

Correct counts include every attempt. Latency medians use only valid outputs. The columns below are calculated by explicitly selecting each path, independently of request order.

Checkpoint / weights Decision correct Chat-key correct Chat-JSON correct Chat-JSON valid Decision median (s) Chat-key median (s) Chat-JSON median (s)
Qwen3.5-9B · BF16 29/30 29/30 6/30 6/30 0.224 0.246 3.553
Gemma-4-A4B · BF16 30/30 29/30 29/30 30/30 0.291 0.304 1.927
GPT-OSS-120B · mixed MXFP4, experimental 27/30 29/30 24/30 25/30 0.297 1.251 3.057
Nemotron 3.5 Super VL · mixed NVFP4 30/30 30/30 13/30 13/30 0.431 0.535 2.704

Decisions avoid generating the score dictionary. Against key-only chat, the time savings are generally smaller. Score-JSON formatting failures mean a valid-only latency can describe a small surviving subset; read it together with validity.

This BFCL-derived subset tests which tool to select, including irrelevance cases. It does not test generated arguments, tool execution, parallel calls or overall BFCL performance.

Do the JSON paths agree?

We pair the same case and pass and compare the selected choices when both responses are valid.

Checkpoint Decision / chat-JSON choice agreement Valid-pair coverage
Qwen3.5-9B · BF16 100.0% (6/6) 6/30 (20.0%)
Gemma-4-A4B · BF16 96.7% (29/30) 30/30 (100.0%)
GPT-OSS-120B · mixed MXFP4, experimental 88.0% (22/25) 25/30 (83.3%)
Nemotron 3.5 Super VL · mixed NVFP4 100.0% (13/13) 13/30 (43.3%)

Agreement includes matching wrong answers. It does not establish equal probabilities, calibrated confidence or equivalent computation. High agreement with low coverage describes only the surviving valid pairs.

The Space index reports this comparison by model and suite, alongside counts of matching wrong answers. Model summaries include all pairwise path comparisons, and request logs expose the underlying payloads.

GPT-OSS requires an experimental path

Native decisions rejected GPT-OSS’s always-reasoning template. The successful experiment used /v1/experimental/final_decision, which forces a verified Harmony final-channel prefix and scores single-token choices through the existing engine without generating reasoning. Chat retained low reasoning effort.

This is an experimental adapter, not native decision support or an officially supported reasoning-off mode. Its time saving includes skipping work that chat performs. On the selection subset, decisions were less accurate than key-only chat. Rejected native attempts remain in the local backup archive; their fast error responses are not successful inference latency.

Small tasks and RouteLLM-derived routing

Checkpoint Small: decision correct Small: key correct RouteLLM: decision correct RouteLLM: key correct
Qwen3.5-9B · BF16 18/20 18/20 7/20 6/20
Gemma-4-A4B · BF16 20/20 20/20 7/20 8/20
GPT-OSS-120B · mixed MXFP4, experimental 20/20 20/20 10/20 7/20
Nemotron 3.5 Super VL · mixed NVFP4 20/20 20/20 8/20 8/20

Repeated small tasks are not independent examples. RouteLLM-derived labels concern an older strong/weak model pair and include ties. GPT-OSS decision accuracy equals a constant-route baseline on the balanced subset. High agreement between paths does not establish good routing.

RouterArena: no aggregate quality improvement over a constant target

The RouterArena-derived evaluation contains 1200 requests, using deterministic domain/difficulty-stratified queries and officially cached outcomes for GPT-4o-mini, Claude 3 Haiku and Gemini 2.0 Flash. Evaluation-only outcomes stayed out of prompts. We did not run new hosted target inference.

Router Decision valid Fully correct target outcomes Mean target quality Mean regret Decision median (s) Key median (s) JSON median (s)
Qwen3.5-9B · BF16 100/100 54/100 0.571 0.080 0.207 0.240 6.471
Gemma-4-A4B · BF16 100/100 54/100 0.571 0.080 0.264 0.291 1.647
GPT-OSS-120B · mixed MXFP4, experimental 100/100 54/100 0.571 0.080 0.272 1.577 2.742
Nemotron 3.5 Super VL · mixed NVFP4 100/100 54/100 0.571 0.080 0.370 0.470 2.168

“Fully correct” means the selected target’s cached quality equals one. Mean quality includes partial scores; regret measures lost quality relative to the best cached target for each query. These are not answers generated by the local router. Quality averages and latency medians use valid outputs. Historical target costs exclude the unmeasured dollar cost of local routing.

The decision selection counts show why baseline context matters:

Router GPT-4o-mini Claude 3 Haiku Gemini 2.0 Flash
Qwen3.5-9B · BF16 0 0 100
Gemma-4-A4B · BF16 0 0 100
GPT-OSS-120B · mixed MXFP4, experimental 0 0 100
Nemotron 3.5 Super VL · mixed NVFP4 8 0 92

Qwen, Gemma and GPT-OSS always chose Gemini. Nemotron varied its selections but obtained the same aggregate quality and regret. None demonstrated aggregate quality improvement over the constant-Gemini baseline. All-zero target outcomes are not routing successes just because every target matches the oracle.

Explore and reproduce the results

The static Space has a quick summary table with Summary | Request log links for each model. Each model has a result.html and log.html grouping its saved runs. Historical failures and retries remain in a separate local backup. No GPU or live backend is needed to browse.

All numerical results tables in this article and the brief Hub post are generated by a script from the retained request records. Editable templates contain the explanatory text; rebuilding calculates the results rather than manually transcribing numbers. Source paths and hashes identify the inputs used by the Space.

Explore the live results Space.

These are small offline diagnostics. We did not establish calibrated confidence, training-data independence, production throughput, mixed-load stability or a universal latency advantage. The supported finding is narrower: compatible checkpoints can serve both chat and structured decisions, and scoring fixed choices can avoid generation work when an application only needs a decision.