IMPAKT
Back to Decision Library
Updated LLM Architecture, Retrieval-Augmented Generation, Model Adaptation

Prompting, Retrieval, Fine-Tuning, Distillation, or Routing? Choose the Smallest Eligible Intervention

Match an observed AI failure to the least irreversible intervention that can resolve it. This decision ladder separates prompting, retrieval, fine-tuning, distillation, and routing by mechanism, new operating burden, evaluation need, countercondition, and rollback instead of treating them as a technology maturity sequence.

A field note by for IMPAKT.

Choose a large language model (LLM) intervention from the failure you measured instead of a technology ladder. Prompting changes instructions. Retrieval supplies external evidence. Fine-tuning changes learned behavior through a trained artifact, which may include adapter weights. Distillation trains another artifact. Routing sends different cohorts through different qualified paths. Start with the least irreversible option that can clear the workload gate, but skip directly to a stronger intervention when evidence shows the simpler one falls short. Every branch still needs its own evaluation, operating owner, and rollback.

Begin with a measured failure class

“The answers are weak” is too broad to choose an intervention. Separate the observed failure:

  • the instruction is ambiguous or output format is unstable;
  • required evidence is missing, stale, inaccessible, or poorly ranked;
  • behavior is inconsistent across a recurring task distribution;
  • the qualified model is too expensive or heavy for the target envelope;
  • different cohorts have materially different capability, control, or service needs;
  • the system lacks authority, verification, recovery, or an accountable owner.

The last failure requires system-boundary work. Fine-tuning leaves payment authority, recovery, and lawful data routing unchanged. Fix those controls before changing model behavior.

Prompt when the task is present but underspecified

Prompting changes instructions, demonstrations, constraints, and output structure without changing model weights. The GPT-3 few-shot paper reports in-context behavior across its evaluated tasks and configurations. Transfer across tasks and versions remains a local evaluation question.

Prompting is a strong first intervention when the system already has the necessary information and capability, while the observed failure concerns task framing, examples, decomposition, or schema. It is comparatively reversible: version the template, rerun the case set, and roll back.

It can still be brittle. Demonstration order, labels, context position, user input, and model version may change behavior. Treat prompt text as part of the released configuration. Preserve counterexamples and adversarial cases alongside successful demonstrations.

Retrieve when the missing asset is current evidence

Retrieval-augmented generation (RAG) supplies selected external material at request time. The original RAG paper evaluates a retrieval-plus-generation approach on specified knowledge-intensive tasks. Its results support a technical pattern; factual support still depends on the corpus, retrieval, and final claim.

Use retrieval when the workload depends on current, owned, attributable, or deletable evidence that should remain outside the weights. The operating system then inherits new responsibilities: access control, ingestion, versioning, chunking, ranking, freshness, poisoning resistance, citation, entailment, deletion, and fallback.

Evaluate retrieval and answer support separately. A document can be relevant without supporting the claim. A cited passage can be stale or outside the user's permissions. Record which source version entered the context and whether the final claim follows from it.

Fine-tune or add adapters when behavior must change repeatedly

Fine-tuning produces a trained artifact for selected behavior. Some methods update the base model; parameter-efficient methods can instead train added weights while keeping the base frozen. The LoRA paper is one primary record of the latter pattern. The InstructGPT paper reports supervised fine-tuning and reinforcement learning from human feedback in its own model and evaluation setting. These studies show that post-training can materially change observed behavior; currency, authority, and correctness remain system-level gates.

Consider fine-tuning when a stable task distribution repeatedly requires behavior beyond the qualified prompting envelope. Examples include specialized output patterns, tool selection, domain language, or task-specific response behavior. Compare it with the best qualified prompt and retrieval baseline.

The artifact introduces data provenance, consent, contamination, training, storage, evaluation, rollback, and refresh duties. It can also encode unwanted behavior or lose capability on rare slices. Version the training set, recipe, base model, checkpoints, and acceptance evidence.

Distill when a bounded behavior needs a different operating envelope

Distillation trains a smaller or otherwise different artifact from teacher outputs or behavior. The official DeepSeek-R1 repository publishes distilled checkpoints and describes their lineage. That record establishes artifact existence and lineage; teacher equivalence and enterprise cost advantage remain workload questions.

Use distillation when a selected behavior is sufficiently stable and valuable to justify a new model lifecycle, while deployment fit, latency, or capacity requirements rule out the teacher path. Evaluate the student independently. Measure where it fails, how often it falls back, and whether the combined student-plus-fallback system improves accepted outcomes.

Distillation is more committed than routing to an existing service. It adds training and artifact governance. Prefer it only when repeated demand can repay that burden.

Route when cohorts are materially different

Routing selects among prompts, models, tools, or execution paths using task, risk, capacity, or evidence. It is useful when cohorts need materially different capabilities or controls. A simple deterministic rule may be enough; a learned router adds its own error distribution.

Test the route decision as a first-class component. Record misclassification, fallback, duplicate context, inconsistent behavior, and cross-route recovery. Include retries and severe failures in route cost. Prefer one simple eligible configuration whenever cohort differences fail to justify the hybrid seam.

This article selects the intervention that addresses a measured failure. Use the IMPAKT workload-placement matrix afterward when that intervention creates a materially different access family or execution venue.

Use an intervention ladder without treating it as maturity

For each candidate intervention, complete a short record:

  1. Name the observed failure and the cases that expose it.
  2. State the mechanism the intervention changes.
  3. State which material variables the mechanism leaves unchanged.
  4. Define the accepted outcome and severe failures.
  5. Compare against the least complex eligible baseline.
  6. Record new data, tools, operators, costs, and failure modes.
  7. Define rollback, fallback, refresh, and retirement.
  8. Promote only when repeated matched evidence clears the gate.

The order is a reversibility heuristic, and eligibility determines the actual sequence. If the workload requires current private evidence, retrieval may be the first design. If a small offline artifact is a hard constraint, distillation or another qualified artifact may enter immediately. If a prompt already clears the gate, added training creates unnecessary burden.

Counterconditions that reverse the choice

Return from fine-tuning to retrieval when facts change faster than a training lifecycle. Return from retrieval to a fixed deterministic source when the task needs one authoritative value rather than semantic search. Remove routing when cohorts no longer differ materially. Retire a distilled path when fallback and maintenance consume its capacity advantage.

Change the architecture rather than the model when the binding issue is identity, permissions, validation, monitoring, human review, or recovery. Intervention choice should reduce the measured failure and keep any shifted burden visible.

Synthetic worked workload — policy answer with citations. Workload HR-POL-QA-01 answers an employee question from approved, current policy text and returns cited passages for accountable review. The observed failure is missing current evidence, so retrieval is the first eligible mechanism; a stronger prompt still lacks the documents. Measure retrieval recall, passage support, accepted answers, stale-source failures, latency in milliseconds, review minutes, and cost per accepted answer. Fine-tuning or adapters remain candidates only for a repeated behavioral failure after retrieval passes. Distillation enters only if a bounded offline or capacity requirement justifies a new artifact lifecycle. Routing enters when jurisdictions require materially different corpora or controls. Each branch retains the fixed prompt baseline and prior qualified configuration as rollback. The qualified scope ends at cited policy preparation for this jurisdiction and excludes employment decisions.

Decision rule

Choose the eligible intervention with the lowest validated lifecycle burden that clears the named failure and preserves a tested rollback. Reopen the choice when the failure class, evidence, task distribution, artifact, route, or recovery path changes.

What this does not prove

This article does not rank prompting, retrieval, fine-tuning, distillation, or routing universally. The cited research reports bounded methods and results; it does not establish IMPAKT performance, production savings, or transfer to an untested workload. “Smaller intervention” does not mean lower risk when its data, authority, or failure path is more consequential.

Editorial process

This article was extracted from the IMPAKT LLM Operating Playbook with AI-assisted structure, drafting, editing, and metadata preparation. It underwent an independent critique and substantive revision loop against IMPAKT's publication rubric; primary sources are linked beside supported claims, and synthesis, recommendations, and evidence boundaries remain explicit.

Sources