Research · 2026
Scheduling LLM inference without reading the prompt
CB-SJF-Work is a content-blind admission policy for LLM inference. It recovers most of an oracle scheduler's gains without ever looking at what the prompt says.
Problem
LLM inference schedulers do not know how long a response will be, and the usual fix, predicting length from the prompt, is exactly what a privacy-constrained platform cannot do. First-come-first-serve suffers head-of-line blocking: one long-generating request admitted early blocks many short ones behind it.
Approach
Measure what content-blindness actually costs, on 44.1 million real requests from Microsoft Azure's production LLM inference traces. Service work splits into a prefill pass (visible at admission from context length alone) and a decode phase (not visible until the request finishes), and prefill turns out to be the larger share. CB-SJF-Work orders admissions by estimated total work using only token counts and arrival metadata, never prompt content.
State
Accepted at IC3IoT 2026, camera-ready submitted. Recovers 90.3% (conversation) and 79.3% (code) of what a perfect-information oracle scheduler attains, without reading a single prompt.
The noticing.
The established fix for head-of-line blocking is to predict how long a response will be, using the prompt itself. But reading the user's prompt is precisely what a privacy-constrained or regulated serving platform cannot do. That reframes the question: how much of the achievable scheduling benefit remains available to a scheduler that never reads content at all?
The answer turned out to be almost all of it, for a reason the prompt-reading literature had not measured. Service time has two parts: a prefill pass, whose cost is fixed by context length and therefore visible without reading anything, and a decode phase, which is not. On 44.1M production requests, prefill is the larger part of the work, at 91.2% and 98.7% of attributable marginal work across the two traces analysed.
Decisions.
Content-blind by construction, not by omission
CB-SJF-Work admits requests in increasing order of estimated total work, using context length and arrival metadata, never prompt or response text. A learned length predictor was tried and kept honest: its own contribution turned out to be almost nothing over ordering by raw context length alone, a negative result stated plainly rather than buried.
Measure the cost, not just the win
Shortest-first ordering reduces mean latency by deferring long requests, which has to degrade the tail. Rather than let that surface later, the paper reports it directly: 99th-percentile latency worsens by up to 3.1x at high load on the code workload. An oracle scheduler pays that cost too, and two control policies (longest-first, random) isolate it to the price of leaving arrival order, not to shortest-first ordering itself.
Two accountings, not the flattering one
Attribution by marginal cost and by simulated engine time tell slightly different stories at low load. Both get reported, not just whichever makes the result look better.
State.
Traces analysed
44.1M requests
Simulator
trace-driven
Status
IEEE camera-ready
Next
The simulator, the analysis code and the figures are public, and the whole run reproduces in a few minutes on CPU. The open problem it leaves standing, content-blind scheduling under prefix caching, is where the next work starts.