Operating a Private 27B LLM: What Makes a Benchmark Decision-Grade?

A private-model benchmark becomes decision-grade only when someone can connect the result to an architecture choice and repeat the comparison. A tokens-per-second headline alone is insufficient. A dated benchmark record must cover the hardware, model and quantization, serving stack, cache state, concurrency, workload, prompts, repetitions, units, quality threshold, matched baseline, raw outputs, and failure conditions. If any of those changes materially, the conclusion should be reopened.

That standard protects both sides of the placement decision. It prevents an attractive lab result from becoming an unsupported self-hosting case. It also prevents a poorly configured local test from dismissing a potentially useful private option.

The useful question is narrower than “How fast is the model?”

IMPAKT operates and benchmarks a private, OpenAI-compatible 27-billion-parameter language-model environment with serving extensions, caches, and agent-tool integrations. IMPAKT is deliberately not publishing unpublished performance numbers here. The more useful field note is the measurement contract IMPAKT requires before a number can influence a chief technology officer’s decision.

The named operating context is a synthetic, tool-assisted document task in the private lab. A fixed document packet supplies context. The model must produce a structured answer and, for selected cases, choose a read-only tool with valid arguments. No client or employer data is part of this description. The business analogue is a repeated knowledge workflow, but the lab fixture is not evidence of enterprise performance.

The decision under test might be: can this configuration meet an acceptable-output threshold and a response-time objective for a stable internal workload, or should the workload remain on a managed API? That is answerable. “Is private AI better?” is not.

The benchmark record is the reusable artifact

The record starts with a one-sentence decision and a versioned workload. Then it captures the following fields before execution:

• Date and change set: test date, code revision, serving-engine version, model identifier, tokenizer, and evaluation version.
• Hardware envelope: processor and accelerator class, accelerator memory in gigabytes, system memory in gigabytes, storage path, power mode, and relevant topology. Sensitive identifiers can remain private while the comparison-relevant specification is preserved.
• Model representation: parameter class, quantization method, context limit used in tokens, and any adapter or modification.
• Serving state: batch policy, concurrency in active requests, cache configuration, speculative-decoding configuration, tool interface, and cold or warm start state.
• Workload: number of cases, input tokens per case, requested output tokens per case, shared-prefix proportion, tool-call proportion, and arrival pattern in requests per minute.
• Method: warm-up policy, repetition count per case, randomization or fixed seed where applicable, timeout in seconds, and failure classification.
• Acceptance: deterministic schema checks, task-specific correctness criteria, human-review rubric where needed, and the minimum acceptable-output rate.
• Preservation: raw prompts or approved hashes, raw outputs, timing events, errors, evaluator decisions, and the script needed to reproduce the summary.

This is a benchmark card, not administrative decoration. It prevents comparison drift. A cache-enabled run against repeated prefixes should not be compared with an uncached run against variable prompts as though only the server changed.

Measure the completed path, then inspect the mechanism

The first result is task acceptance. Report accepted completed cases divided by attempted cases, alongside the count and classification of failures. A fast invalid tool call is not goodput. Goodput means acceptable work completed per unit of time under the stated conditions.

Next record end-to-end latency in seconds, including preprocessing, retrieval, model prefill, generation, tool execution, and validation. Preserve the median and a tail percentile rather than only an average. Record time to first token in seconds and output-token rate in tokens per second, but keep them subordinate to accepted completion.

Throughput needs an explicit load. Report attempted and accepted completions per hour at a stated concurrency in active requests. Also record queue time in seconds, peak memory in gigabytes, and failure counts by category. If energy or infrastructure economics matter, measure energy in kilowatt-hours and cost in a named currency per accepted completed case. Do not infer those values from model speed alone.

Serving mechanisms can then explain a result. PagedAttention was proposed to manage key-value cache memory more efficiently for large-language-model serving; its paper evaluates system throughput under specified workloads (PagedAttention paper). That supports examining memory behavior, not assuming the paper’s results transfer to IMPAKT's configuration.

Prefix caching is similarly conditional. In its version 0.5.3.post1 documentation, vLLM describes reusing key-value cache blocks when requests share a prefix, avoiding recomputation of that shared prompt segment (vLLM prefix-caching documentation). The benchmark must expose the shared-prefix proportion and measure prefix processing separately from generation rather than assigning all change to the cache.

Speculative decoding uses a faster proposal process and a verification step to accelerate generation while preserving the target distribution under the method’s assumptions (Speculative Decoding paper). It should be tested with its configuration named and quality checks retained. A gain on one prompt-length or acceptance pattern should not become a universal throughput multiplier.

Match the baseline to the decision

A managed-API baseline should receive the same approved prompt content, output contract, case set, timeout policy, and acceptance evaluation. Compare end-to-end completion rather than isolating only model generation. Record provider and model identifiers, test date, region or endpoint class when relevant, concurrency, retry policy, and price basis. If an exact match is impossible, name the asymmetry instead of hiding it.

A self-hosted baseline should change one decision variable at a time when the goal is causal explanation. Change the quantization while holding the model, workload, server settings, and evaluation stable. Change cache state while holding the request sequence stable. Change concurrency only after preserving the single-request reference. Real systems have interactions, but controlled comparisons reveal which mechanism deserves further testing.

The MLCommons inference working group describes its purpose as creating fair, representative benchmarks and identifies architecture-neutral, representative, reproducible assessment as the need. Its deliverables include benchmark definitions, reference software, and submission rules (MLCommons Inference). An internal benchmark need not reproduce that program to borrow its discipline: specify the scenario, constrain quality, and make the comparison basis inspectable.

The enterprise translation comes last. Stable accepted throughput may inform capacity planning. Tail latency may expose a user-experience or queueing constraint. Cache benefit may reveal that repeated context changes the economics. Failure categories may show that evaluation or tool design, rather than inference speed, is the current bottleneck. These are hypotheses for the named workload, not direct return-on-investment claims.

The benchmark answers one evidence question inside the broader workload-placement matrix. If a candidate survives the quality and operating gates, carry its measured acceptance, capacity, and failure data into the completed-workflow cost model instead of translating tokens per second directly into an investment claim.

Decision rule

Let a private benchmark influence architecture only when it passes five gates:

1. The workload and acceptance threshold represent the intended task.
2. Configuration, cache state, concurrency, repetitions, and units are recorded.
3. The baseline uses the same case set and completion definition, with asymmetries disclosed.
4. Raw outputs, timing events, evaluator decisions, and failures are preserved.
5. The result is repeated after a clean restart and remains decision-relevant under a material sensitivity test.

If the candidate meets quality but misses latency, test serving and workload mechanisms. If it meets latency but misses acceptance, do not declare an infrastructure win. If the workload is rare, highly bursty, or depends on capability unavailable in the candidate, a managed API may remain the stronger decision regardless of local throughput.

The next useful action is to complete the benchmark card before running another test. Empty fields reveal where an impressive number could not survive architecture review.

What this does not prove

Operating one private 27-billion-parameter environment does not establish production resilience, concurrency, security, service levels, total cost, or superiority over APIs. A synthetic document-and-tool fixture does not establish behavior on an organization’s data, users, permissions, or demand distribution. No benchmark number in this article claims an observed result.

The cited papers and documentation explain mechanisms under their own configurations. They do not validate IMPAKT's environment or predict its benefit. A decision-grade lab record is evidence for a bounded proof of concept. Production placement still requires workload-specific security, privacy, legal, operational, financial, and recovery review.

Editorial process

This article was developed from IMPAKT's editorial direction with AI-assisted drafting and independent editorial review. Primary sources are linked beside the claims they support and listed below; direct observations, synthesis, recommendations, hypothetical examples, and material boundaries are identified where they appear.

Sources

• MLCommons Inference — accessed August 27, 2026.
• PagedAttention paper — submitted September 12, 2023; accessed August 27, 2026.
• Speculative Decoding paper — published in the 2023 International Conference on Machine Learning proceedings; accessed August 27, 2026.
• vLLM automatic prefix-caching documentation — version 0.5.3.post1; accessed August 27, 2026.