LLMOps is MLOps for the LLM era. Different bottlenecks (cost, latency, hallucination), different tools (LangSmith, Helicone, LiteLLM, Braintrust), different metrics (groundedness, RAG faithfulness, refusal rate). Same goal: ship reliably without burning $50K/month on tokens.
Learning Objectives
After this lesson, you will be able to:
Explain how running LLMs in production is different from traditional ML -- and why you need new tools for it
Set up prompt versioning and testing so that changing a prompt is as safe and traceable as changing code
Build an LLM gateway that routes requests, limits costs, tracks spending, and switches providers when one goes down
If you have used ChatGPT or any LLM API, you have already experienced LLMOps challenges without knowing it: inconsistent responses, surprise costs, prompt tweaks that break things. This lesson gives you the tools and patterns to manage all of that professionally.
What Do You Think?
Your LLM endpoint costs $50K/month. What is the first optimization to try?
The answer is D. Before optimizing anything, you need to understand what you are spending on. Traffic analysis often reveals that 20-40% of LLM calls are unnecessary -- queries that could be handled by keyword search, rule engines, or cached responses. The cheapest LLM call is the one you never make.
Prompts are the "code" of LLM applications. They need the same rigor as software code: version control, review, testing, and rollback.
The problem: A team of 5 engineers iterating on prompts in a Google Doc. Nobody knows which version is in production. A "small tweak" to the system prompt breaks 30% of edge cases. There is no way to roll back except manually pasting the old prompt.
The solution: Prompt versioning systems.
Tool
Type
Key Feature
Humanloop
SaaS platform
Visual prompt editor + A/B testing
PromptLayer
SaaS platform
Prompt version history + analytics
Braintrust
SaaS platform
Eval-first prompt development
Langfuse
Open source
Prompt management + observability
Git + JSON files
DIY
Free, full control, integrates with existing CI
A production-grade prompt versioning workflow
This is exactly how software CI/CD works -- applied to prompts instead of code.
Guardrails are the safety nets that prevent LLMs from going off the rails. They sit between your application and the LLM, intercepting both inputs and outputs.
Input guardrails (before the LLM sees the query)
Prompt injection detection (is the user trying to override the system prompt?)
PII detection and redaction (remove credit card numbers, SSNs before processing)
Topic filtering (is this query within the allowed scope?)
Token budget enforcement (reject queries that would cost too much)
Output guardrails (before the user sees the response)
Hallucination detection (does the response contain claims not supported by context?)
Content safety filtering (toxic, harmful, or inappropriate content)
Format validation (is the JSON output actually valid JSON?)
Try it! Take any LLM API (or use a free one) and try the same prompt 5 times. Notice how the outputs differ each time -- same input, different output. Now imagine versioning and testing those prompts automatically. That is what LLMOps tools do.
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Example: Guardrails AI for structured output validation
from guardrails import Guard
from guardrails.hub import ValidRange, ToxicLanguage
guard = Guard().use_many(
ValidRange(min=0, max=100, on_fail="fix"),
ToxicLanguage(on_fail="filter"),
)
# The guard wraps your LLM call and validates the output
raw_response, validated_response, *rest = guard(
llm_api=openai.chat.completions.create,
model="gpt-4o",
messages=[{"role": "user", "content": "Rate this product 1-100"}],
)
# If the LLM returns 150, the guard fixes it to 100
# If the LLM outputs toxic language, the guard filters it
Arbitrage across providers, useful for latency-tolerant traffic
Kong AI Gateway
Open source
Enterprise API gateway with AI extensions
AWS Bedrock / Azure AI Foundry
Managed (cloud-native)
VPC + IAM + provisioned throughput
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# LiteLLM: unified interface to any LLM provider
from litellm import completion
# Same function, different providers — swap models without code changes
response_openai = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Explain LLMOps"}],
)
response_anthropic = completion(
model="claude-3-5-sonnet-20241022",
messages=[{"role": "user", "content": "Explain LLMOps"}],
)
response_local = completion(
model="ollama/llama3",
messages=[{"role": "user", "content": "Explain LLMOps"}],
)
Why gateways matter: Without a gateway, switching from GPT-4 to Claude requires changing every API call in your codebase. With a gateway, you change one configuration line. This flexibility is not a nice-to-have -- it is essential when a provider has an outage, raises prices, or a better model launches.
Prefix / prompt caching — the highest-ROI LLMOps lever. Anthropic, OpenAI, Google Gemini, DeepSeek, and most self-hosted runtimes (vLLM, SGLang) now support server-side caching of common prompt prefixes. For RAG systems where the system prompt + retrieval template is identical across requests, cached tokens cost 10-25% of fresh tokens and time-to-first-token drops 30-70%.
The math is worth running explicitly — most teams underestimate the savings.
LLM evaluation is fundamentally harder than classical ML evaluation. You cannot just compute accuracy on a test set because there is no single "correct" answer for most LLM tasks.
Evaluation dimensions
Dimension
What It Measures
How to Measure
Correctness
Is the answer factually right?
Human review, LLM-as-judge
Relevance
Does it address the question?
Semantic similarity, LLM-as-judge
Faithfulness
Is it grounded in provided context?
NLI models, citation checking
Helpfulness
Is it useful to the end user?
User feedback, A/B tests
Safety
Is it free of harmful content?
Content classifiers, red teaming
Cost
How much did it cost per response?
Token counting, gateway metrics
Latency
How fast is the response?
Time-to-first-token, total time
Key evaluation tools
Tool
Type
Best For
promptfoo
Open source CLI
Prompt regression testing in CI
LangSmith
SaaS platform
Tracing + evaluation for LangChain apps
Braintrust
SaaS platform
Eval datasets + scoring + experiments
Ragas
Open source
RAG-specific evaluation metrics
DeepEval
Open source
Unit testing framework for LLMs
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Tests · Run both prompt versions against the test suite. Verify the new prompt handles off-topic queries better. Check the deployment decision logic.
LLM costs behave nothing like traditional infrastructure costs. A web server at 10x traffic costs ~10x. An LLM application at 10x traffic can cost 10x more per user because each request involves heavy computation.
The cost management stack
Layer 1: Visibility
Track cost per request, per user, per feature, per team
Use gateway-level metrics (LiteLLM, Portkey) for real-time cost dashboards
The key insight: LLMOps is not one tool -- it is a system of interconnected components. Start with the gateway (routing + cost tracking), add guardrails (safety), then build evaluation (quality). Do not try to build all five layers on day one.
The LLMOps stack is moving as fast as the models. Notable additions since 2024:
Serving runtimes: vLLM 0.6+, SGLang 0.4+, and TensorRT-LLM 0.18+ ship continuous batching, prefix caching, speculative decoding, FP8 / FP4 inference (B200), and disaggregated prefill/decode by default — what required custom infra in 2024 is one config flag in 2026.
Memory-first agent runtimes: Letta (formerly MemGPT, 2025) productized self-managed agent memory; CrewAI Studio and LangGraph Studio added visual graph IDEs for multi-agent workflows; Autogen 2.0 standardized the multi-agent contract.
Optimization frameworks: DSPy 2.5+ (2025) made declarative prompt + program optimization mainstream — define a signature, let DSPy compile it against your eval set.
Apple Silicon serving: MLX (Apple 2024+) lets you run Qwen 3, Llama 4 Scout, and DeepSeek distills on M-series Macs for prototyping and on-device inference.
Serverless GPU + edge: Modal, Replicate, Cloudflare Workers AI, and Together / Fireworks / Groq cover the spectrum from pay-per-second GPUs to global edge inference; pick by latency budget and request shape.
Function calling 2.0: parallel and batched tool calls are now standard across OpenAI, Anthropic, Google, and most open-weights serving stacks — your gateway must handle tool-call concurrency, not just sequential calls.
Native multimodal output: Claude voice, GPT-4o/GPT-5 audio-out, and Gemini native image-out generate non-text modalities directly. Logging now needs to capture audio and image bytes, not just text.
Treating prompts like configuration, not code -- Prompts change behavior as much as code changes. They need version control, code review, testing, and rollback capabilities.
No evaluation before deployment -- "The new prompt looks good to me" is not a deployment strategy. Run it against 50+ test cases and compare to the baseline.
Single-provider dependency -- If 100% of your traffic goes to one LLM provider and they have a 4-hour outage, your product is down for 4 hours. Use a gateway with automatic failover.
Ignoring cost until it is too late -- Set up cost tracking on day one, not after your first $50K bill. Gateway-level cost attribution is cheap to implement and invaluable for budgeting.
Over-engineering guardrails -- Start with basic input/output validation. Add sophisticated guardrails (NeMo, custom classifiers) only after you have real data on what goes wrong.
Prompts are code, not config. Version-control every prompt, run regression tests before deploying changes, and maintain rollback capability; a single bad prompt update can silently degrade your entire application
Guardrails protect both directions. Input guardrails catch prompt injection and PII before the LLM processes them; output guardrails catch hallucination and harmful content before users see them; skip either side and you have a half-protected system
LLM gateways eliminate provider lock-in. A single unified API (LiteLLM, Portkey) gives you automatic failover, per-team cost tracking, rate limiting, and semantic caching without changing application code
Evaluation never stops. Run promptfoo or LangSmith in CI to catch regressions before production; track correctness, relevance, safety, and cost as separate dimensions, not a single score
Prompt caching is the highest-ROI quick win. For RAG systems where the system prompt is identical across queries, Anthropic prompt caching can reduce input token costs by 90% and time-to-first-token by 50%
You now have the operational stack for LLM systems: versioned prompts, gateways, evals, and observability. Next up: Model Serving & Inference -- the lower-level engineering that decides whether your LLMOps stack runs at 200ms p99 or 10s p99 at the same QPS.