cloud scheduling · accepted
Quantifying the Cost of Content-Blindness in LLM Inference Scheduling
Every serious fix for head-of-line blocking in LLM serving reads the prompt to guess how long the response will be. This paper asks what a scheduler forfeits when it cannot, and finds the answer smaller than expected.
Problem
Predicting response length from prompt text is the standard fix for head-of-line blocking in LLM serving, and exactly what a privacy-constrained platform cannot do. Whether anything is actually lost by not reading the prompt had never been measured directly.
Approach
44.1 million real requests from Microsoft Azure's production LLM inference traces, split into a conversation and a code workload. Service work decomposes into an observable prefill component (fixed by context length) and an unobservable decode component. The paper measures how much of each there actually is, then evaluates CB-SJF-Work, a scheduler that orders admissions on the observable part alone.
Findings
Prefill is 91.2% and 98.7% of attributable marginal work in the two traces, so the component a content-blind scheduler can see is the larger one. CB-SJF-Work cuts mean normalised latency by 51.5% and 58.8% against first-come-first-serve, recovering 90.3% and 79.3% of what a perfect-information oracle attains, at a measured tail-latency cost the paper reports rather than hides.
The problem.
Continuous-batching inference engines hold a request's slot for as long as it takes to generate its response, so one long-generating request admitted early blocks many short ones behind it. The standard literature answer is to predict output length from the prompt and approximate shortest-job-first. That requires reading the user's prompt, which is a genuine obstacle for a platform under data-protection obligations or offering confidential-computing guarantees.
The question underneath the constraint had not been answered empirically: how much of the achievable scheduling benefit is actually locked inside the prompt?
Approach.
The paper analyses the Azure LLM Inference Trace 2024, 44.1 million real production requests across a conversation and a code workload, and decomposes service work into a prefill pass (proportional to context length, visible at admission) and a decode phase (proportional to output length, not visible until the request completes). It then evaluates CB-SJF-Work, ordering admissions by estimated total work using only context length and arrival metadata, benchmarked against first-come-first-serve and against oracles with perfect output-length knowledge.
For these workloads the information a scheduler is denied is largely the information it least needs.
— paper abstract
Findings.
Prefill share, conversation
91.2%
Prefill share, code
98.7%
Latency cut, conversation
51.5%
Latency cut, code
58.8%
Oracle gap closed, conversation
90.3%
Oracle gap closed, code
79.3%
Two findings qualify the headline. The gain is paid for in the tail: 99th-percentile latency degrades by up to 3.1x at high load on the code workload, a cost an oracle scheduler also incurs, and one isolated to the act of leaving arrival order rather than to shortest-first ordering itself. And a learned content-blind length predictor earns almost nothing over simply ordering by raw context length, a negative result the paper states rather than omits.
Accepted at IC3IoT 2026, Track 4 (Blockchain, Cloud Computing and Big Data Analytics), 24 to 25 September 2026. Co-authored with Kavita Sri.
Next
The simulator, the analysis code, the generated figures and the LaTeX tables are public, and the whole run reproduces in a few minutes on CPU. The open problem it leaves standing is prefix-cache-aware content-blind scheduling: multi-turn conversation traffic serves repeated prefixes from cache, and the traces used here record context length but not cache residency, so the current result cannot answer what that does.