Cost Architecture

#The $100K/Month Surprise

This lesson might save you (or your company) more money than any other topic in this track. Cost surprises kill more AI startups than bad models. The good news: most cost problems are solvable once you know the patterns.

#The Real Numbers: What ML Actually Costs

#GPU Selection: The Hardware Menu

#Training GPUs

Prices are approximate as of early 2026 and vary by cloud provider and region.

#Inference GPUs

Prices are approximate as of early 2026 and vary by cloud provider and region.

This is a trick question -- two L4s only have 48 GB total, and tensor parallelism overhead means you likely need the A100. But the real answer is D: cost-effectiveness depends on throughput. If the A100 serves 30 requests/second and two L4s serve 15 req/s, the per-request cost is identical. You need to benchmark your specific model and traffic pattern.

#The Cost Architecture Decision Process

Building a cost-efficient AI system is not about being cheap -- it is about being systematic. Here is the step-by-step process for designing an architecture that balances performance and cost:

#Token Optimization: Spending Less on Every Request

For LLM API users (OpenAI, Anthropic, Google), tokens are the primary cost driver:

#1. Prompt Engineering for Efficiency

Before optimization

You are a helpful customer service agent for Acme Corp. You were founded
in 1985 and are known for your excellent products and customer service.
When responding to customers, always be polite, professional, and helpful.
Make sure to address their concern directly and provide actionable steps.
If you don't know the answer, say so honestly rather than making
something up. Always end with asking if there's anything else you can help with.

Customer: What is your return policy?

Token count: ~100 input tokens for system prompt

After optimization

[Acme Corp CS agent. Be concise, helpful, honest. If unsure, say so.]

Customer: What is your return policy?

Token count: ~20 input tokens for system prompt (80% reduction)

#2. Caching Strategies

Prompt caching (provider-level)

• Anthropic and OpenAI offer prompt caching: repeated system prompts are cached server-side
• First request pays full price; subsequent requests with the same prefix get a 90% discount on cached tokens
• Effective for applications with static system prompts

Semantic caching (application-level)

• Hash incoming queries and cache responses
• Use embedding similarity to match similar (not identical) queries to cached responses
• Cache hit rate of 10-30% for customer support bots, higher for FAQ-style products
• Invalidation strategy: TTL-based or event-triggered

#3. Context Window Management

#Multi-Model Routing: The Right Model for the Job

Routing strategies

#1. Rule-Based Routing

Simple but effective for well-understood traffic patterns. Works for 60-70% of use cases.

#2. Classifier-Based Routing

Train a small, fast classifier (BERT-tiny, 10ms latency) to predict query difficulty:

• Class 0: Easy (route to mini model) -- $0.001/request
• Class 1: Medium (route to standard model) -- $0.01/request
• Class 2: Hard (route to premium model) -- $0.05/request

The classifier costs $0.0001/request. Even if it misroutes 10% of queries, the overall cost savings are massive.

#3. Cascade / Fallback Routing

1. Try the small model first
2. If confidence is low or output quality is poor (detected by a cheap quality check), escalate to the large model
3. Only 20-30% of queries typically need escalation

#LLM Gateways: The Cost Control Plane

What you should enforce at the gateway:

1. Per-customer / per-feature budgets — hard cutoff plus soft alerts at 80%.
2. Prefix caching — hash system prompts and reuse server-side KV; many gateways ship this.
3. Semantic cache — embed and match the user message; serve cached completions for high-similarity hits.
4. Fallback chains — primary fails or rate-limits, secondary handles, tertiary handles, and graceful degradation if all fail.
5. Provider rotation — when a provider has elevated p99, shift weight automatically.
6. Audit logging — every prompt, completion, model, latency, token count, and cost tagged by user/feature.

#FinOps Math: Per-Million-Tokens, GPU Hours, and Spot Economics

Pricing models for AI workloads are a mess of dimensions. Work them out in the same units before deciding.

#Spot Instances and Preemptible Compute

Training on spot instances

• AWS Spot, GCP Preemptible, Azure Low Priority: 60-90% cheaper than on-demand
• Risk: instances can be terminated with 30-120 seconds notice
• Mitigation: checkpoint every N minutes, auto-resume from latest checkpoint
• Best practice: Checkpoint every 15-30 minutes. Use a persistent storage volume (EFS, GCS) for checkpoints.

Inference on spot instances

• More risky than training (user-facing latency)
• Use spot for batch inference (non-real-time)
• For real-time: use spot for overflow capacity with on-demand for baseline

#Key Takeaways

• GPU selection depends on workload type. H100s for large-scale training, A100s for medium training and inference, L4/T4 for cost-effective inference; matching hardware to workload avoids 2-5x overspending
• Multi-model routing cuts costs dramatically. Route simple queries to cheap small models and hard queries to expensive large models; 70-80% of queries are often simple enough for the small model
• Token optimization reduces LLM API costs by 40-70%. Prompt compression, caching frequent responses, batching requests, and truncating unnecessary context all reduce token consumption without degrading quality
• Cost must be a first-class architectural concern. Designing for cost-efficiency from the start (choosing the right model size, implementing caching, using spot instances) is far easier than optimizing after launch

#Quick Check

#Exit Ticket