Cost Architecture

Legacy GPT-4 at $30/M output tokens vs Claude Haiku 4.5 at $5/M output, and $1/M in versus $30/M in — is a 6-30x swing on the exact same request. The architecture decisions you make in week 1 lock in 10x of your bill. Here's how to design AI products for the bill — model routing, prompt caching, batch APIs, and the math that decides which GPU to rent.

Learning Objectives

After this lesson, you will be able to:

Compare GPU options (A100, H100, L4, T4) and pick the right hardware for training versus serving
Design a routing system that sends easy questions to cheap models and hard questions to expensive ones
Apply token-saving strategies that cut LLM API costs by 40-70% without making the output worse

Prerequisites

Before You Start

Review

Why

Why This Matters

Why running ChatGPT costs OpenAI $700,000/day, and how to make AI cheaper — every ChatGPT response requires expensive GPU computation; at 100 million+ users, that adds up to hundreds of millions per year in serving costs alone; this is why OpenAI created GPT-4o-mini (cheaper for simple queries), why Anthropic offers prompt caching (reuse the same system prompt without re-paying), and why Google has multi-tier pricing; understanding these cost strategies is the difference between a sustainable AI product and a money bonfire
Your side project could cost $500/month if you are not careful — using GPT-4 for every single request in a popular app can easily hit $5,000+/month in API calls; but if you route simple questions ("what time is it?") to a cheap model and only send hard questions ("explain quantum computing") to the expensive one, you cut costs by 10x without users noticing any quality drop
The math that kills AI startups — if 10 users cost you $1/day, 10,000 users cost $1,000/day, and 1 million users cost $100,000/day; without cost optimization (caching, batching, model routing, prompt compression), your app gets more popular AND more unprofitable at the same time; this lesson teaches you how to bend that cost curve
Build this → Build a cost calculator: input your expected users, queries per day, and average token usage, then model the monthly cost under different strategies (all GPT-4, GPT-4o-mini routing, semantic caching, prompt compression) — find the cheapest architecture that still meets your quality bar

#The $100K/Month Surprise

This lesson might save you (or your company) more money than any other topic in this track. Cost surprises kill more AI startups than bad models. The good news: most cost problems are solvable once you know the patterns.

Intuition

Build Your Intuition

A startup builds a ChatGPT-like product. During development, they test with 100 requests per day. Costs are negligible. They launch, get 100K users, and their first cloud bill is $87,000 for a single month of GPU compute. The company has 8 months of runway. At this rate, they have 3 months.

This is not hypothetical — it happens constantly. AI costs are fundamentally different from traditional software costs. A web server handling 10x more traffic costs roughly 10x more. An LLM serving 10x more users can cost 10x more per user because each request does heavy computation, not just database lookups.

Cost architecture is the discipline of making AI systems economically viable. It is not about being cheap — it is about being efficient. The best cost architecture delivers the same user experience at 5-10x lower cost. That is the difference between a sustainable business and a slow-motion bankruptcy.

#The Real Numbers: What ML Actually Costs

pythonplayground.py · Pyodide

# Real ML project cost estimation
# These numbers reflect actual cloud pricing (approximate, mid-2025)

training_cost = {
  "GPU": "A100 80GB x 4 = $12/hr x 72 hours = $864",
  "Storage": "500GB training data on S3 = $11.50/month",
  "Compute": "Preprocessing on c5.4xlarge = $0.68/hr x 24 hrs = $16.32",
  "Total training": "$891.82 per training run",
}

inference_cost = {
  "GPU": "T4 16GB = $0.35/hr x 730 hrs/month = $255.50",
  "API calls": "1M requests/month x $0.002/request = $2,000",
  "Bandwidth": "1M x 5KB avg response = 5GB = $0.45",
  "Total serving": "$2,255.95/month",
}

print("=== TRAINING COSTS (one-time per run) ===")
for key, val in training_cost.items():
  print(f"  {key}: {val}")

print()
print("=== SERVING COSTS (recurring monthly) ===")
for key, val in inference_cost.items():
  print(f"  {key}: {val}")

# Calculate annual total cost of ownership
training_total = 891.82
serving_monthly = 2255.95
annual_cost = serving_monthly * 12 + training_total

print()
print("=== TOTAL COST OF OWNERSHIP (Year 1) ===")
print(f"  Training (one-time):  ${training_total:,.2f}")
print(f"  Serving (12 months):  ${serving_monthly * 12:,.2f}")
print(f"  Annual total:         ${annual_cost:,.2f}")
print()
print(f"  Training is {training_total / annual_cost * 100:.1f}% of annual cost")
print(f"  Serving is {serving_monthly * 12 / annual_cost * 100:.1f}% of annual cost")
print()
print("KEY INSIGHT: Serving costs dominate. A model that is")
print("3x cheaper to train but 10% more expensive to serve")
print("will cost MORE over its lifetime.")
print()
print("CHALLENGE: What if you retrain monthly instead of once?")
print("How does that change the training vs serving ratio?")

# --- Try it! ---
# Change daily_active_users to YOUR expected user count.
# Change avg_queries_per_user to match your app.
# See how fast the monthly bill grows. Then try adding
# a "cache_hit_rate" of 0.3 (30% of queries are repeats)
# and subtract that from API calls. How much do you save?

Tests · Verify annual cost calculation is correct. Confirm serving costs exceed training costs. Calculate the retraining scenario.

#Training GPUs

GPU	VRAM	FP16 TFLOPS	$/hr (cloud)	Best For
H100 SXM	80 GB	990	$3.00-4.00	Large model training (70B+)
A100 SXM	80 GB	312	$2.00-2.50	Most training workloads
A100 PCIe	40/80 GB	312	$1.50-2.50	Budget training
L4	24 GB	121	$0.40-0.60	Fine-tuning, small model training
H200	141 GB	990	$5.00-6.00	Very large models (140B+)

Prices are approximate as of early 2026 and vary by cloud provider and region.

Key insight: H100 is ~3x faster than A100 for training, but only ~1.5-2x more expensive. For large training runs, H100 is almost always more cost-effective despite the higher hourly rate because you need fewer hours.

#Inference GPUs

GPU	VRAM	INT8 TOPS	$/hr (cloud)	Best For
H100	80 GB	1,979	$3.00-4.00	High-throughput LLM serving
A100	80 GB	624	$2.00-2.50	Standard LLM serving
L4	24 GB	485	$0.40-0.60	Cost-efficient inference, small models
T4	16 GB	130	$0.35-0.50	Budget inference, INT8 models
L40S	48 GB	733	$1.00-1.50	Mid-range, good for 13B-34B models

Prices are approximate as of early 2026 and vary by cloud provider and region.

The L4 sweet spot: For many inference workloads, NVIDIA L4 offers the best cost-performance ratio. It has enough VRAM for quantized 7B-13B models and costs 75% less per hour than A100. Four L4s can serve the same throughput as one A100 for many model sizes, at half the cost.

What Do You Think?

You need to serve a Llama 70B model. An A100-80GB costs $2/hr and an L4-24GB costs $0.60/hr. The 70B model in INT4 needs ~37GB. Which is more cost-effective?

This is a trick question — two L4s only have 48 GB total, and tensor parallelism overhead means you likely need the A100. But the real answer is D: cost-effectiveness depends on throughput. If the A100 serves 30 requests/second and two L4s serve 15 req/s, the per-request cost is identical. You need to benchmark your specific model and traffic pattern.

#The Cost Architecture Decision Process

Building a cost-efficient AI system is not about being cheap — it is about being systematic. Here is the step-by-step process for designing an architecture that balances performance and cost:

#Step 1: Estimate Request Volume

Start with your traffic projections. How many requests per day? What is the peak-to-average ratio? A chatbot serving 10K requests/day has fundamentally different economics than one serving 1M. Volume determines whether API providers or self-hosting is more economical.

#Step 2: Calculate API Costs

Price out the API option. At 50K requests/day with 2,000 input tokens and 500 output tokens each, GPT-4-turbo costs ~$30K/month. GPT-5 mid-tier costs ~$7-8K/month. Claude Haiku 4.5 / GPT-5-mini costs ~$1-3K/month. Model choice is the single biggest cost lever.

#Step 3: Calculate Self-Hosted GPU Costs

Price out the self-hosting option. Two A100-80GB GPUs at $2/hr each cost $2,880/month fixed. Add engineering time for setup, monitoring, and maintenance. Self-hosting has a high fixed cost but near-zero marginal cost per request.

#Step 4: Find the Break-Even Point

The break-even point is where API costs equal self-hosted costs. Below this volume, APIs win (simpler, no ops burden). Above it, self-hosting wins (lower marginal cost). For most teams, the break-even is around $15-20K/month in API spend, factoring in engineering time.

#Step 5: Choose Architecture

Select your architecture based on the analysis. Options include: all-API (simple, scales automatically), all-self-hosted (cheapest at scale), or hybrid (self-host a small model for easy queries, API for hard ones). Multi-model routing is almost always the right answer at scale.

#Step 6: Implement Caching and Batching

Layer on cost optimizations. Semantic caching intercepts 15-25% of repeated queries. Prompt compression reduces input tokens by 40-60%. Batch APIs (50% cheaper) handle non-real-time tasks. Each optimization compounds multiplicatively with the others.

#Step 7: Monitor and Optimize

Track cost per request, cost per user, and cost per feature daily. Alert on anomalies (prompt drift making prompts longer, traffic spikes, retry storms). Review cost metrics weekly in sprint reviews. The best cost architecture evolves continuously as traffic patterns and model pricing change.

#Token Optimization: Spending Less on Every Request

For LLM API users (OpenAI, Anthropic, Google), tokens are the primary cost driver:

#1. Prompt Engineering for Efficiency

Before optimization

You are a helpful customer service agent for Acme Corp. You were founded
in 1985 and are known for your excellent products and customer service.
When responding to customers, always be polite, professional, and helpful.
Make sure to address their concern directly and provide actionable steps.
If you don't know the answer, say so honestly rather than making
something up. Always end with asking if there's anything else you can help with.

Customer: What is your return policy?

Token count: ~100 input tokens for system prompt

After optimization

[Acme Corp CS agent. Be concise, helpful, honest. If unsure, say so.]

Customer: What is your return policy?

Token count: ~20 input tokens for system prompt (80% reduction)

Impact at scale: 100K requests/day x 80 tokens saved x $0.01/1K tokens = $80/day = $2,400/month saved from one prompt optimization.

#2. Caching Strategies

Prompt caching (provider-level)

Anthropic and OpenAI offer prompt caching: repeated system prompts are cached server-side
First request pays full price; subsequent requests with the same prefix get a 90% discount on cached tokens
Effective for applications with static system prompts

Semantic caching (application-level)

Hash incoming queries and cache responses
Use embedding similarity to match similar (not identical) queries to cached responses
Cache hit rate of 10-30% for customer support bots, higher for FAQ-style products
Invalidation strategy: TTL-based or event-triggered

#3. Context Window Management

pythonplayground.py · Pyodide

# Token cost calculator and optimization simulator

PRICING = {
  "gpt-4-turbo": {"input": 10.00, "output": 30.00},     # per 1M tokens
  "gpt-4o": {"input": 2.50, "output": 10.00},
  "gpt-4o-mini": {"input": 0.15, "output": 0.60},
  "claude-3-5-sonnet": {"input": 3.00, "output": 15.00},
  "claude-3-5-haiku": {"input": 0.80, "output": 4.00},
}

def estimate_monthly_cost(model: str, daily_requests: int,
                        avg_input_tokens: int, avg_output_tokens: int) -> dict:
  pricing = PRICING[model]
  monthly_requests = daily_requests * 30
  input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * pricing["input"]
  output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * pricing["output"]
  return {"model": model, "input_cost": input_cost, "output_cost": output_cost,
          "total": input_cost + output_cost, "monthly_requests": monthly_requests}

# Baseline: everything goes to GPT-4-turbo
baseline = estimate_monthly_cost("gpt-4-turbo", 50000, 2000, 500)
print("=== Baseline: All traffic to GPT-4-turbo ===")
print(f"Monthly cost: ${baseline['total']:.2f}")
print(f"  Input: ${baseline['input_cost']:.2f} | Output: ${baseline['output_cost']:.2f}")

print("\n=== Optimization Strategies ===\n")

# Strategy 1: Switch to GPT-4o
opt1 = estimate_monthly_cost("gpt-4o", 50000, 2000, 500)
print(f"1. Switch to GPT-4o: ${opt1['total']:.2f}"
    f" (saving {(1 - opt1['total']/baseline['total']) * 100:.0f}%)")

# Strategy 2: Use GPT-4o-mini for 70% of requests
opt2a = estimate_monthly_cost("gpt-4o-mini", 35000, 2000, 500)
opt2b = estimate_monthly_cost("gpt-4o", 15000, 2000, 500)
opt2_total = opt2a["total"] + opt2b["total"]
print(f"2. Route 70% to mini, 30% to 4o: ${opt2_total:.2f}"
    f" (saving {(1 - opt2_total/baseline['total']) * 100:.0f}%)")

# Strategy 3: Same as 2 + reduce prompt from 2000 to 800 tokens
opt3a = estimate_monthly_cost("gpt-4o-mini", 35000, 800, 400)
opt3b = estimate_monthly_cost("gpt-4o", 15000, 800, 500)
opt3_total = opt3a["total"] + opt3b["total"]
print(f"3. + Optimize prompts (2000->800 tokens): ${opt3_total:.2f}"
    f" (saving {(1 - opt3_total/baseline['total']) * 100:.0f}%)")

# Strategy 4: Same as 3 + 20% cache hit rate
opt4_total = opt3_total * 0.80
print(f"4. + 20% semantic cache hit: ${opt4_total:.2f}"
    f" (saving {(1 - opt4_total/baseline['total']) * 100:.0f}%)")

print(f"\nBaseline: ${baseline['total']:.2f}/month")
print(f"Fully optimized: ${opt4_total:.2f}/month")
print(f"Total savings: {(1 - opt4_total/baseline['total']) * 100:.0f}%")

print("\nCHALLENGE: Add self-hosted Llama-4-Maverick / DeepSeek-V3.1 as an option.")
print("Assume 2x A100 at \$2/hr each = \$2,880/month fixed cost,")
print("handling up to 100K requests/day. When is self-hosting cheaper?")

Tests · Calculate the baseline cost and verify each optimization strategy reduces it. Find the break-even point for self-hosting.

#Multi-Model Routing: The Right Model for the Job

Routing strategies

#1. Rule-Based Routing

Rule-Based Routing Logic

Incoming Query

IF length < 50 tokens AND no code → Small Model

IF contains 'explain' or 'analyze' → Large Model

IF non-English language → Multilingual Model

DEFAULT → Medium Model

Simple but effective for well-understood traffic patterns. Works for 60-70% of use cases.

#2. Classifier-Based Routing

Train a small, fast classifier (BERT-tiny, 10ms latency) to predict query difficulty:

Class 0: Easy (route to mini model) — $0.001/request
Class 1: Medium (route to standard model) — $0.01/request
Class 2: Hard (route to premium model) — $0.05/request

The classifier costs $0.0001/request. Even if it misroutes 10% of queries, the overall cost savings are massive.

#3. Cascade / Fallback Routing

Try the small model first
If confidence is low or output quality is poor (detected by a cheap quality check), escalate to the large model
Only 20-30% of queries typically need escalation

Cost analysis: If 70% of queries are resolved by the small model at $0.001 each and 30% escalate to the large model at $0.05 each, the blended cost is $0.0157/request vs. $0.05/request for all-large-model. That is a 68% savings.

Try it: Calculate Your Inference CostsInteractive

Loading visualization...

#LLM Gateways: The Cost Control Plane

The 2024-2026 production pattern is to put every model call through an LLM gateway — a proxy that handles routing, caching, retries, fallback, observability, and budget enforcement in one place. Without it, cost controls scatter across application code; with it, you have a single chokepoint to enforce policy.

Gateway	Open source?	Strongest feature	Notes
LiteLLM	Yes	Unified OpenAI-compatible API across 100+ providers	The de-facto standard for self-hosted gateways; built-in budgets and routing rules.
Portkey	Hybrid	Smart routing, fallback chains, prompt versioning	Strong fallback graphs (try GPT-5, on fail try Claude Sonnet 5, on fail try DeepSeek-V3.1).
Helicone	Yes	Drop-in OpenAI proxy with cost analytics	Minimal code change; pairs well with Langfuse for tracing.
Cloudflare AI Gateway	Managed	Edge caching + rate limiting + observability	Lowest latency overhead; pay only for traffic.
OpenRouter	Managed	Marketplace pricing across providers	Useful for arbitrage when latency-tolerant.
AWS Bedrock / Azure AI Foundry	Managed	Native cloud auth + VPC + provisioned throughput	Pick if you are already deep in that cloud.

What you should enforce at the gateway:

Per-customer / per-feature budgets — hard cutoff plus soft alerts at 80%.
Prefix caching — hash system prompts and reuse server-side KV; many gateways ship this.
Semantic cache — embed and match the user message; serve cached completions for high-similarity hits.
Fallback chains. Primary fails or rate-limits, secondary handles, tertiary handles, and graceful degradation if all fail.
Provider rotation. When a provider has elevated p99, shift weight automatically.
Audit logging. Every prompt, completion, model, latency, token count, and cost tagged by user/feature.

What Do You Think?

Your team is deciding whether to call OpenAI directly from each microservice or to route everything through a self-hosted LiteLLM gateway. Annual LLM spend is projected at $1.2M. What is the single strongest argument for the gateway?

#FinOps Math: Per-Million-Tokens, GPU Hours, and Spot Economics

Pricing models for AI workloads are a mess of dimensions. Work them out in the same units before deciding.

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

#Spot Instances and Preemptible Compute

Training on spot instances

AWS Spot, GCP Preemptible, Azure Low Priority: 60-90% cheaper than on-demand
Risk: instances can be terminated with 30-120 seconds notice
Mitigation: checkpoint every N minutes, auto-resume from latest checkpoint
Best practice: Checkpoint every 15-30 minutes. Use a persistent storage volume (EFS, GCS) for checkpoints.

Inference on spot instances

More risky than training (user-facing latency)
Use spot for batch inference (non-real-time)
For real-time: use spot for overflow capacity with on-demand for baseline

Watch Out

Common Mistake

Ignoring output tokens. Output tokens are 3-6x more expensive than input tokens for most providers. A verbose system prompt that encourages long answers dramatically increases output token costs. Add "Be concise" or set max_tokens to limit output cost.

Not accounting for retries and failures. If 5% of requests fail and are retried, your effective cost is 5% higher. If retries also fail and cascade to a larger model, costs compound. Monitor retry rates and root-cause failures.

Premature self-hosting. Self-hosting sounds cheaper but adds ops burden: GPU procurement, driver updates, model deployment, monitoring, on-call rotation. At fewer than 50K requests/day, API providers are usually cheaper when you factor in engineering time. Self-host when API costs exceed $15-20K/month and you have the team to manage infrastructure.

Not monitoring cost per request over time. Prompt drift (prompts getting longer as features are added), traffic pattern changes, and model updates can silently increase costs. Track cost per request daily and alert on anomalies.

#Key Takeaways

GPU selection depends on workload type. H100s for large-scale training, A100s for medium training and inference, L4/T4 for cost-effective inference; matching hardware to workload avoids 2-5x overspending
Multi-model routing cuts costs dramatically. Route simple queries to cheap small models and hard queries to expensive large models; 70-80% of queries are often simple enough for the small model
Token optimization reduces LLM API costs by 40-70%. Prompt compression, caching frequent responses, batching requests, and truncating unnecessary context all reduce token consumption without degrading quality
Cost must be a first-class architectural concern. Designing for cost-efficiency from the start (choosing the right model size, implementing caching, using spot instances) is far easier than optimizing after launch

#Quick Check

Quick Check1 / 4

Why is the NVIDIA L4 often the most cost-effective GPU for inference?

Cost optimization at the model level (KV-cache reuse, quantization, prefix caching) is covered in depth in the NLP track's inference efficiency lesson. Training-data economics (carbon, GPU hours, dataset curation cost) come from the same FinOps playbook applied earlier in the lifecycle.

#Exit Ticket

With cost architecture mastered, you can build AI products that are both powerful and economically sustainable. Next up: Security & Compliance — protecting your AI systems from attacks and meeting regulatory requirements.

Cost Architecture

Learning Objectives

After this lesson, you will be able to:

Compare GPU options (A100, H100, L4, T4) and pick the right hardware for training versus serving
Design a routing system that sends easy questions to cheap models and hard questions to expensive ones
Apply token-saving strategies that cut LLM API costs by 40-70% without making the output worse

Prerequisites

Before You Start

Review

Why

Why This Matters

Why running ChatGPT costs OpenAI $700,000/day, and how to make AI cheaper — every ChatGPT response requires expensive GPU computation; at 100 million+ users, that adds up to hundreds of millions per year in serving costs alone; this is why OpenAI created GPT-4o-mini (cheaper for simple queries), why Anthropic offers prompt caching (reuse the same system prompt without re-paying), and why Google has multi-tier pricing; understanding these cost strategies is the difference between a sustainable AI product and a money bonfire
Your side project could cost $500/month if you are not careful — using GPT-4 for every single request in a popular app can easily hit $5,000+/month in API calls; but if you route simple questions ("what time is it?") to a cheap model and only send hard questions ("explain quantum computing") to the expensive one, you cut costs by 10x without users noticing any quality drop
The math that kills AI startups — if 10 users cost you $1/day, 10,000 users cost $1,000/day, and 1 million users cost $100,000/day; without cost optimization (caching, batching, model routing, prompt compression), your app gets more popular AND more unprofitable at the same time; this lesson teaches you how to bend that cost curve
Build this → Build a cost calculator: input your expected users, queries per day, and average token usage, then model the monthly cost under different strategies (all GPT-4, GPT-4o-mini routing, semantic caching, prompt compression) — find the cheapest architecture that still meets your quality bar

#The $100K/Month Surprise

Intuition

Build Your Intuition

#The Real Numbers: What ML Actually Costs

pythonplayground.py · Pyodide

# Real ML project cost estimation
# These numbers reflect actual cloud pricing (approximate, mid-2025)

print("=== TRAINING COSTS (one-time per run) ===")
for key, val in training_cost.items():
  print(f"  {key}: {val}")

print()
print("=== SERVING COSTS (recurring monthly) ===")
for key, val in inference_cost.items():
  print(f"  {key}: {val}")

# Calculate annual total cost of ownership
training_total = 891.82
serving_monthly = 2255.95
annual_cost = serving_monthly * 12 + training_total

Tests · Verify annual cost calculation is correct. Confirm serving costs exceed training costs. Calculate the retraining scenario.

#Training GPUs

GPU	VRAM	FP16 TFLOPS	$/hr (cloud)	Best For
H100 SXM	80 GB	990	$3.00-4.00	Large model training (70B+)
A100 SXM	80 GB	312	$2.00-2.50	Most training workloads
A100 PCIe	40/80 GB	312	$1.50-2.50	Budget training
L4	24 GB	121	$0.40-0.60	Fine-tuning, small model training
H200	141 GB	990	$5.00-6.00	Very large models (140B+)

Prices are approximate as of early 2026 and vary by cloud provider and region.

#Inference GPUs

GPU	VRAM	INT8 TOPS	$/hr (cloud)	Best For
H100	80 GB	1,979	$3.00-4.00	High-throughput LLM serving
A100	80 GB	624	$2.00-2.50	Standard LLM serving
L4	24 GB	485	$0.40-0.60	Cost-efficient inference, small models
T4	16 GB	130	$0.35-0.50	Budget inference, INT8 models
L40S	48 GB	733	$1.00-1.50	Mid-range, good for 13B-34B models

Prices are approximate as of early 2026 and vary by cloud provider and region.

What Do You Think?

You need to serve a Llama 70B model. An A100-80GB costs $2/hr and an L4-24GB costs $0.60/hr. The 70B model in INT4 needs ~37GB. Which is more cost-effective?

#The Cost Architecture Decision Process

Building a cost-efficient AI system is not about being cheap — it is about being systematic. Here is the step-by-step process for designing an architecture that balances performance and cost:

#Step 1: Estimate Request Volume

#Step 2: Calculate API Costs

#Step 3: Calculate Self-Hosted GPU Costs

#Step 4: Find the Break-Even Point

#Step 5: Choose Architecture

#Step 6: Implement Caching and Batching

#Step 7: Monitor and Optimize

#Token Optimization: Spending Less on Every Request

For LLM API users (OpenAI, Anthropic, Google), tokens are the primary cost driver:

#1. Prompt Engineering for Efficiency

Before optimization

You are a helpful customer service agent for Acme Corp. You were founded
in 1985 and are known for your excellent products and customer service.
When responding to customers, always be polite, professional, and helpful.
Make sure to address their concern directly and provide actionable steps.
If you don't know the answer, say so honestly rather than making
something up. Always end with asking if there's anything else you can help with.

Customer: What is your return policy?

Token count: ~100 input tokens for system prompt

After optimization

[Acme Corp CS agent. Be concise, helpful, honest. If unsure, say so.]

Customer: What is your return policy?

Token count: ~20 input tokens for system prompt (80% reduction)

Impact at scale: 100K requests/day x 80 tokens saved x $0.01/1K tokens = $80/day = $2,400/month saved from one prompt optimization.

#2. Caching Strategies

Prompt caching (provider-level)

Anthropic and OpenAI offer prompt caching: repeated system prompts are cached server-side
First request pays full price; subsequent requests with the same prefix get a 90% discount on cached tokens
Effective for applications with static system prompts

Semantic caching (application-level)

Hash incoming queries and cache responses
Use embedding similarity to match similar (not identical) queries to cached responses
Cache hit rate of 10-30% for customer support bots, higher for FAQ-style products
Invalidation strategy: TTL-based or event-triggered

#3. Context Window Management

pythonplayground.py · Pyodide

# Token cost calculator and optimization simulator

print("\n=== Optimization Strategies ===\n")

# Strategy 4: Same as 3 + 20% cache hit rate
opt4_total = opt3_total * 0.80
print(f"4. + 20% semantic cache hit: ${opt4_total:.2f}"
    f" (saving {(1 - opt4_total/baseline['total']) * 100:.0f}%)")

print(f"\nBaseline: ${baseline['total']:.2f}/month")
print(f"Fully optimized: ${opt4_total:.2f}/month")
print(f"Total savings: {(1 - opt4_total/baseline['total']) * 100:.0f}%")

Tests · Calculate the baseline cost and verify each optimization strategy reduces it. Find the break-even point for self-hosting.

#Multi-Model Routing: The Right Model for the Job

Routing strategies

#1. Rule-Based Routing

Rule-Based Routing Logic

Incoming Query

IF length < 50 tokens AND no code → Small Model

IF contains 'explain' or 'analyze' → Large Model

IF non-English language → Multilingual Model

DEFAULT → Medium Model

Simple but effective for well-understood traffic patterns. Works for 60-70% of use cases.

#2. Classifier-Based Routing

Train a small, fast classifier (BERT-tiny, 10ms latency) to predict query difficulty:

Class 0: Easy (route to mini model) — $0.001/request
Class 1: Medium (route to standard model) — $0.01/request
Class 2: Hard (route to premium model) — $0.05/request

The classifier costs $0.0001/request. Even if it misroutes 10% of queries, the overall cost savings are massive.

#3. Cascade / Fallback Routing

Try the small model first
If confidence is low or output quality is poor (detected by a cheap quality check), escalate to the large model
Only 20-30% of queries typically need escalation

Try it: Calculate Your Inference CostsInteractive

Loading visualization...

#LLM Gateways: The Cost Control Plane

Gateway	Open source?	Strongest feature	Notes
LiteLLM	Yes	Unified OpenAI-compatible API across 100+ providers	The de-facto standard for self-hosted gateways; built-in budgets and routing rules.
Portkey	Hybrid	Smart routing, fallback chains, prompt versioning	Strong fallback graphs (try GPT-5, on fail try Claude Sonnet 5, on fail try DeepSeek-V3.1).
Helicone	Yes	Drop-in OpenAI proxy with cost analytics	Minimal code change; pairs well with Langfuse for tracing.
Cloudflare AI Gateway	Managed	Edge caching + rate limiting + observability	Lowest latency overhead; pay only for traffic.
OpenRouter	Managed	Marketplace pricing across providers	Useful for arbitrage when latency-tolerant.
AWS Bedrock / Azure AI Foundry	Managed	Native cloud auth + VPC + provisioned throughput	Pick if you are already deep in that cloud.

What you should enforce at the gateway:

Per-customer / per-feature budgets — hard cutoff plus soft alerts at 80%.
Prefix caching — hash system prompts and reuse server-side KV; many gateways ship this.
Semantic cache — embed and match the user message; serve cached completions for high-similarity hits.
Fallback chains. Primary fails or rate-limits, secondary handles, tertiary handles, and graceful degradation if all fail.
Provider rotation. When a provider has elevated p99, shift weight automatically.
Audit logging. Every prompt, completion, model, latency, token count, and cost tagged by user/feature.

What Do You Think?

#FinOps Math: Per-Million-Tokens, GPU Hours, and Spot Economics

Pricing models for AI workloads are a mess of dimensions. Work them out in the same units before deciding.

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

#Spot Instances and Preemptible Compute

Training on spot instances

AWS Spot, GCP Preemptible, Azure Low Priority: 60-90% cheaper than on-demand
Risk: instances can be terminated with 30-120 seconds notice
Mitigation: checkpoint every N minutes, auto-resume from latest checkpoint
Best practice: Checkpoint every 15-30 minutes. Use a persistent storage volume (EFS, GCS) for checkpoints.

Inference on spot instances

More risky than training (user-facing latency)
Use spot for batch inference (non-real-time)
For real-time: use spot for overflow capacity with on-demand for baseline

Watch Out

Common Mistake

#Key Takeaways

GPU selection depends on workload type. H100s for large-scale training, A100s for medium training and inference, L4/T4 for cost-effective inference; matching hardware to workload avoids 2-5x overspending
Multi-model routing cuts costs dramatically. Route simple queries to cheap small models and hard queries to expensive large models; 70-80% of queries are often simple enough for the small model
Token optimization reduces LLM API costs by 40-70%. Prompt compression, caching frequent responses, batching requests, and truncating unnecessary context all reduce token consumption without degrading quality
Cost must be a first-class architectural concern. Designing for cost-efficiency from the start (choosing the right model size, implementing caching, using spot instances) is far easier than optimizing after launch

#Quick Check

Quick Check1 / 4

Why is the NVIDIA L4 often the most cost-effective GPU for inference?

Cost Architecture

#The $100K/Month Surprise

#The Real Numbers: What ML Actually Costs

#GPU Selection: The Hardware Menu

#Training GPUs

#Inference GPUs

#The Cost Architecture Decision Process

#Step 1: Estimate Request Volume

#Step 2: Calculate API Costs

#Step 3: Calculate Self-Hosted GPU Costs

#Step 4: Find the Break-Even Point

#Step 5: Choose Architecture

#Step 6: Implement Caching and Batching

#Step 7: Monitor and Optimize

#Token Optimization: Spending Less on Every Request

#1. Prompt Engineering for Efficiency

Before optimization

After optimization

#2. Caching Strategies

Prompt caching (provider-level)

Semantic caching (application-level)

#3. Context Window Management

#Multi-Model Routing: The Right Model for the Job

Routing strategies

#1. Rule-Based Routing

#2. Classifier-Based Routing

#3. Cascade / Fallback Routing

#LLM Gateways: The Cost Control Plane

#FinOps Math: Per-Million-Tokens, GPU Hours, and Spot Economics

#Spot Instances and Preemptible Compute

Training on spot instances

Inference on spot instances

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

#Exit Ticket

Your Reflection

Continue Learning

Discussion

Cost Architecture

#The $100K/Month Surprise

#The Real Numbers: What ML Actually Costs

#GPU Selection: The Hardware Menu

#Training GPUs

#Inference GPUs

#The Cost Architecture Decision Process

#Step 1: Estimate Request Volume

#Step 2: Calculate API Costs

#Step 3: Calculate Self-Hosted GPU Costs

#Step 4: Find the Break-Even Point

#Step 5: Choose Architecture

#Step 6: Implement Caching and Batching

#Step 7: Monitor and Optimize

#Token Optimization: Spending Less on Every Request

#1. Prompt Engineering for Efficiency

Before optimization

After optimization

#2. Caching Strategies

Prompt caching (provider-level)

Semantic caching (application-level)

#3. Context Window Management

#Multi-Model Routing: The Right Model for the Job

Routing strategies

#1. Rule-Based Routing

#2. Classifier-Based Routing

#3. Cascade / Fallback Routing

#LLM Gateways: The Cost Control Plane

#FinOps Math: Per-Million-Tokens, GPU Hours, and Spot Economics

#Spot Instances and Preemptible Compute

Training on spot instances

Inference on spot instances

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

#Exit Ticket

Your Reflection

Continue Learning

Discussion