Executive Inference Glossary
Translate inference mechanisms into the workload, capacity, quality, and cost questions they can actually change.
Start with the actual decision
Inference terms are useful when they explain a constraint or a tradeoff. They become distracting when a technical metric is presented as a business outcome.
Use each definition to ask what changed for the workload, which unit was measured, and what the result does not establish.
Intended user
Executives, product and finance leaders, architects, and practitioners who need a shared language for evaluating inference proposals and benchmarks.
Collect before comparing
- 01The business workflow and acceptable outcome
- 02Request shape, context length and repetition, output length, volume, concurrency, and peaks
- 03Quality, first-response, completion-time, throughput, and reliability requirements
- 04Hardware, serving software, model, quantization, batching, cache, and routing configuration
- 05Measurement date, units, comparison basis, test duration, and excluded costs
Test every material criterion against the workload
KV or prefix cache
Can repeated context avoid repeated computation, and how often does that exact reuse occur?
Potentially changes latency, capacity, and unit economics for repeated workloads; it proves little for highly variable prompts.
Quantization
What memory or throughput benefit is gained, and does quality remain acceptable on the real task?
Lower numerical precision may broaden hardware options or capacity, with workload-dependent quality tradeoffs.
Speculative decoding
Does a draft-and-verify path improve accepted generation speed on this model, hardware, and output distribution?
Can improve throughput or latency, but the benefit depends on acceptance behavior and added complexity.
Continuous batching
Can requests share serving capacity efficiently under the expected arrival pattern?
May raise utilization and throughput while changing queueing and latency behavior.
Time to first token (TTFT)
How quickly does the response begin, and is that the delay users or downstream systems actually feel?
Useful for interactive responsiveness, but it does not describe total completion time or quality.
Goodput
How much useful work meets the stated quality and service threshold per unit of time?
More decision-relevant than raw throughput when acceptance and latency requirements are explicit.
Treat an inference metric as decision-relevant only when it changes a documented workload requirement or cost component under a matched comparison. If the workload, unit, quality threshold, configuration, date, or boundary is missing, use the result as a question—not a conclusion.
This glossary simplifies implementation details for decision use. It is not a tuning guide, benchmark result, hardware recommendation, or guarantee that an optimization will help. Definitions remain stable longer than vendor behavior, prices, and measured performance.
Editorial version 1.0
Initial public glossary reviewed August 2026. Definitions should be revisited as serving implementations and common measurement practices change.
StandardsBuild a cost-per-workflow view
Put the relevant mechanisms into a comparison that includes quality, exceptions, labor, and operating cost.
Continue