IMPAKT
Back to Decision Library
Updated LLM Serving, Inference Engineering, AI Performance

Prefill, Decode, and Goodput: Where LLM Inference Actually Bottlenecks

Diagnose LLM serving by separating prefill, decode, queueing, batching, speculative decoding, and external work. The inference measurement record keeps phase metrics tied to representative arrival patterns, tail latency, failures, and accepted workflow goodput so a local optimization cannot masquerade as an operating improvement.

A field note by for IMPAKT.

Large language model (LLM) inference has at least two different serving stages: prefill reads the supplied context and builds attention state; decode produces output tokens sequentially. An optimization can improve one stage while leaving the user-visible workflow unchanged or worse. Diagnose with phase metrics, but decide with accepted goodput: completed work that passes quality, severe-failure, service, authority, and recovery thresholds under representative load. This distinction tells a CTO whether to change prompts, scheduling, batching, speculative decoding, hardware, or the service design itself.

Prefill reads the dossier

Prefill processes the input tokens and constructs the state used during generation. Prompt length, attention implementation, runtime, and hardware influence its cost and time to first token. Long requests can also delay shorter interactive work when they share queues and resources.

The DistServe paper studies separate resource provisioning for prefill and decode and introduces a goodput-oriented serving design under its evaluated workloads. Its results are configuration-specific. The transferable mechanism is that the two phases can have different resource characteristics and service effects.

For document-heavy work, prefill may dominate the wait before any output appears. Repeated stable prefixes may make caching material. For short prompts with long outputs, decode may dominate. Measure the workload distribution to locate the bottleneck.

Decode writes one step at a time

Decode generates a token and then uses the updated state for the next token. Inter-token delay shapes the streaming experience. Memory bandwidth, retained context, batch composition, hardware utilization, and runtime scheduling can influence the rate.

Tokens per second is useful diagnostic evidence when its unit and conditions are explicit. Evaluate output acceptance, tool completion, and exception work separately. A faster invalid completion remains an attempted result.

Capture time to first token, inter-token delay, and end-to-end latency separately. Report medians and tail percentiles at named concurrency. A single unloaded run provides a point sample rather than an operating service level.

Continuous batching changes the queue

Static batching waits for requests that can advance together. Iteration-level scheduling lets completed sequences leave and new work enter at token-step boundaries. The Orca serving-system paper reports such scheduling and evaluates it in the authors' system.

The elevator analogy is useful: riders can enter and leave at more frequent stops instead of waiting for an entire group to finish. Higher utilization can follow under suitable traffic. Queue policy, giant prompts, very long outputs, and admission control still determine whether short requests receive fair service.

Before enabling a batching change, segment traffic into service classes such as interactive, background, and long-running. Measure accepted throughput and tail latency for each class. A global average can conceal starvation.

Speculative decoding drafts, then verifies

Speculative decoding uses a cheaper proposal mechanism to draft several possible tokens. The target model evaluates the draft in parallel, accepts a valid prefix, and applies the method's correction after a rejection. The speculative decoding paper establishes distribution preservation under its algorithm and assumptions while reporting speedups in tested settings.

The target remains the authority for accepted tokens under the method; the proposal model gains no independent authority. Benefit depends on draft acceptance, proposal cost, target verification cost, sequence shape, batch, and hardware utilization. Poor acceptance or an already saturated system can remove the advantage.

Record accepted draft length, rejection frequency, draft and target compute, sampling settings, correctness checks, and end-to-end latency. Requalify after either model, tokenizer, runtime, or sampling policy changes.

Goodput is the decision metric

Throughput counts attempted work under a stated load. Goodput counts work that meets a specified service objective. IMPAKT extends that idea to accepted workflow goodput: completions must also clear the task rubric, severe-failure gate, authority boundary, and recovery requirement.

Create an inference measurement record with these fields:

  • model, artifact, runtime, hardware, precision, scheduler, and cache state;
  • prompt and output-length distributions by service class;
  • arrival rate, concurrency, batch, queue policy, and test duration;
  • first-token, inter-token, and end-to-end latency distributions;
  • attempted requests, completed requests, accepted completions, and severe failures;
  • retries, tool calls, fallback, review minutes, and recovery events;
  • prefill, decode, queueing, and external-tool time where observable;
  • raw evidence location, owner, date, expiry, and change triggers.

Comparable goodput requires a stable denominator, acceptance rubric, and difficult-case set across candidates.

Reproduce the request arrival pattern

Serving behavior depends on when work arrives as well as what each request contains. Build a test trace from representative inter-arrival times, prompt and output lengths, service classes, cancellation, timeouts, and bursts. Warm and cold cache states should be separate runs. Hold the request mix stable when comparing schedulers or hardware.

Run long enough to observe queue growth and recovery after a burst. Report the measurement window and completed sample count. A queue still accumulating at the end indicates an unsustainable throughput rate. Background work paused to protect interactive latency remains deferred demand.

Also inspect failure handling. A timeout that retries immediately can double load precisely when the system is saturated. Admission control, backpressure, bounded retries, and degraded service may improve accepted outcomes more than another local optimization. Include their effects in the same run record.

Match the intervention to the measured stage

If prefill dominates, reduce unnecessary context, improve evidence selection, test prefix reuse, protect interactive queues, or evaluate phase-specific capacity. If decode dominates, test compatible kernels, batch policy, speculative decoding, or a different qualified artifact. If external tools dominate, focus latency work on that path.

If quality or recovery fails, a latency optimization is premature. Return to the decision-grade benchmark contract and preserve the configuration tuple, cases, acceptance, raw outputs, and failures.

The countercondition is also important. A slower phase metric may be acceptable if it improves end-to-end acceptance or reduces exception burden. Optimize the operating outcome ahead of the most visible dashboard number.

Synthetic worked workload — support-case summary. Workload SUP-SUM-01 produces an evidence-linked case summary after a support ticket closes. Use two service classes: an interactive request at closure and an overnight backlog over the same accepted-output rubric. Record input and output tokens per request, requests per minute, and time to first token and tail latency in milliseconds. Also record accepted summaries per hour, queue recovery after bursts, review minutes, and cost per accepted summary; leave every target and result to measure. Long case histories may expose prefill pressure while summary length may expose decode pressure. The measured phase selects the intervention. Qualification remains scoped to this workload, queue, model, evidence packet, and service objective.

Price the qualified service classes through the completed-workflow cost model after accepted goodput is visible.

Decision rule

Use prefill, decode, queue, and cache metrics to locate the constraint. Enable batching, speculative decoding, or phase-specific capacity only when representative accepted goodput improves without breaking tail latency, fairness, quality, authority, or recovery gates.

What this does not prove

This article does not claim a universal serving bottleneck, throughput multiplier, latency target, or preferred runtime. The cited systems papers report results under their own hardware, models, workloads, and methods. No published result is presented as an IMPAKT benchmark or as evidence of production economics.

Editorial process

This article was extracted from the IMPAKT LLM Operating Playbook with AI-assisted structure, drafting, editing, and metadata preparation. It underwent an independent critique and substantive revision loop against IMPAKT's publication rubric; primary sources are linked beside supported claims, and synthesis, recommendations, and evidence boundaries remain explicit.

Sources