Your team built the demo in a week. The CEO loved it. That was four months ago. The LLM-powered feature is still not in production.
The demo-to-production gap for LLMs is wider than for traditional ML. The model itself works fine. Everything around it doesn't: prompt management is ad hoc, there's no regression testing, latency is unpredictable, costs are uncontrolled, and nobody has thought about what happens when the model says something wrong.
This is the unglamorous engineering checklist between a working prototype and a production LLM service.
Prompts are code. Treat them accordingly.
Version control. Store prompts in Git, not in application code as string literals. Prompt changes should go through code review. A one-word change in a system prompt can alter behaviour across every user interaction.
Templating. Separate prompt structure from dynamic content. Use a templating system (Jinja2, Mustache, or your framework's native approach) so prompts are readable and testable.
A/B testing. Prompt changes need the same rollout discipline as code changes. Run the new prompt on a subset of traffic. Compare output quality metrics before full deployment.
Prompt registry. For organisations with multiple LLM features, maintain a registry of active prompts with metadata: owner, last updated, evaluation results, associated model version.
Prompts are code. Version them, review them, test them, and roll them out progressively. A one-word system prompt change can alter behaviour for every user.
LLM output is non-deterministic. You can't write assert statements against exact responses. But you can build an evaluation framework that catches regressions.
Build an eval dataset. 100+ input-output pairs covering your key use cases, edge cases, and known failure modes. This is the most valuable asset in your LLM pipeline.
Automated scoring. Use a combination of:
Run evals on every change. Prompt updates, model version upgrades, system prompt modifications, and retrieval pipeline changes all trigger the eval suite. If scores drop below baseline, the change doesn't ship.
Track metrics over time. Eval scores should be dashboarded and trended. Gradual degradation is harder to catch than sudden drops.
| Eval type | What it catches | Automation level |
|---|---|---|
| Format checks | Broken JSON, missing fields, length violations | Fully automated |
| LLM-as-judge | Quality regressions, tone shifts, factual errors | Semi-automated (requires calibration) |
| Embedding similarity | Semantic drift from expected answers | Fully automated |
| Human review | Subtle quality issues, brand voice, nuance | Manual (sample-based) |
Production LLMs need boundaries. Without them, you're one edge case away from an incident.
Input validation. Filter or reject inputs that are adversarial, off-topic, or contain injection attempts. Prompt injection is a real attack vector, not a theoretical concern.
Output validation. Check every response before serving it:
Fallback behaviour. Define what happens when the model fails or guardrails trigger:
Never serve an unvalidated LLM response to a user in a production system.
Never serve an unvalidated LLM response to a user. Every output needs format checks, content filtering, and a fallback path for when things go wrong.
LLM inference is slow and expensive compared to traditional ML. Both require explicit engineering.
Latency budgets. Define your P50 and P99 targets. A chatbot needs sub-2-second first-token latency. A batch summarisation job can tolerate 30 seconds. Design your architecture around the constraint.
Streaming. For user-facing applications, stream tokens as they're generated. Perceived latency drops dramatically even when total generation time is unchanged.
Caching. Cache responses for identical or semantically similar inputs. Exact-match caching is simple. Semantic caching (embedding-based similarity) catches paraphrased queries. Both reduce cost and latency.
Model selection. Not every request needs your largest model. Route simple queries to smaller, cheaper models. Use the expensive model only when the task demands it. This pattern (model routing) can reduce costs by 50–70%.
Async patterns. For non-interactive use cases (document processing, batch analysis), process requests asynchronously. Queue jobs, process in parallel, return results via webhook or polling.
| Cost lever | Savings potential | Implementation effort |
|---|---|---|
| Semantic caching | 20–40% of LLM calls eliminated | Medium |
| Model routing (small/large) | 50–70% cost reduction | Medium |
| Prompt optimisation (fewer tokens) | 10–30% per request | Low |
| Batch processing (off-peak) | 30–50% with spot pricing | Low |
Not every request needs your most expensive model. Route simple queries to smaller models and reserve the large model for complex tasks. This alone can halve your LLM costs.
Production LLMs need audit trails, especially in regulated industries.
Log everything. Input prompt, retrieved context (if RAG), model response, guardrail decisions, latency, cost, and user feedback. Store logs immutably.
Model cards. Document each LLM deployment: model version, intended use, known limitations, evaluation results, responsible team, and escalation path.
Cost controls. Per-user and per-feature rate limits. Budget alerts. Token consumption tracking by feature and team.
Incident response. Define what constitutes an LLM incident (harmful output served, data leakage, sustained quality degradation). Write runbooks. Practice them.
Before shipping an LLM feature to production, verify:
If half of these are missing, you're not ready for production. You're ready for a demo.