Most ML failures are data failures. Schema drift in an upstream Kafka topic silently corrupts a feature, a model gets retrained on bad data, and accuracy quietly drops 8% over six weeks. The fix isn't a better model — it's better pipelines. Airflow, dbt, Kafka, Great Expectations, and data contracts.
Learning Objectives
After this lesson, you will be able to:
Distinguish batch (Airflow/dbt) from streaming (Kafka/Flink) data pipelines and pick the right architecture for your latency/freshness target
Use data contracts and schema registries (Confluent, Apicurio, Avro/Protobuf) to prevent silent data corruption between producers and consumers
Apply DAG-orchestrated workflows (Airflow, Prefect, Dagster) with idempotency, retries, and observability for production reliability
Recognize the 'silent data quality' failure modes — schema drift, late-arriving data, dedup races — and the patterns that prevent them
The biggest production failure mode: a producer team changes the schema of the data your ML pipeline consumes, your pipeline silently produces wrong features, your model accuracy drops. Data contracts prevent this.
A contract specifies: schema (field names, types, nullability), invariants (no negative ages, prices in $), update SLAs, ownership.
Contract violation⇒pipeline halts at boundary, alerts producer team — does NOT silently corrupt downstream
A feature store is the system that promises one thing: the feature value your training pipeline saw is identical to the feature value your inference pipeline serves. Without that guarantee, you get training-serving skew — the most insidious ML bug, because metrics look fine offline but the production model silently underperforms.
The dominant feature stores in 2026:
Store
Origin
Sweet spot
Trade-offs
Feast
Tecton open-source (2019)
Most lightweight; team owns the infra
DIY ingestion + serving; you wire it up
Tecton
Tecton SaaS
Managed end-to-end + low-latency online serving
Cost; vendor lock-in
Hopsworks
LogicalClocks/Hopsworks AI
On-prem + EU sovereignty; PySpark heritage
Heavier deploy footprint
Databricks Feature Store
Databricks
Native to Delta Lake / Unity Catalog users
Locked to Databricks
SageMaker Feature Store
AWS
Native AWS integration
AWS-only
The common architecture under all of them is the same: an offline store (Parquet / Delta / Iceberg / BigQuery) holding historical feature values for training, and an online store (Redis / DynamoDB / Cassandra / Bigtable) holding the latest feature values for serving. Ingestion writes to both; the feature store's job is to guarantee the values match.
Notice that the online store is 10-100x more expensive per GB. You don't load ALL features into the online store — only the ones with realtime traffic. The feature store handles the partitioning.
This is the single most important concept in feature engineering for ML. Point-in-time correctness means: when you build a training row at timestamp T, every feature value in that row must reflect what your system would have known at T — not what it knows today.
A concrete failure: you're training a model to predict purchase intent. You join user_features to purchase_events on user_id. If you naively JOIN ON user_id, you'll pick up user_features as they exist today — including features computed from purchases that happen AFTER the prediction timestamp. The model "learns" using future information, scores 99% in offline eval, and lays a 60% accuracy egg in production. This is feature leakage, and it's why point-in-time joins exist.
The correct join is: for each event at T_event, join to the latest feature value with feature_timestamp <= T_event. Most feature stores call this an as_of_join or point_in_time_join and implement it on top of Spark / Polars / DuckDB.
Loading visualization...
Run that. Notice the difference: the naive join shows avg_purchases_per_week = 2.3 for the event on 2026-05-10, but that value didn't exist until 2026-05-15. A model trained on the naive join would silently encode future information; the point-in-time join gives the model 0.7, the value the system would actually have known at prediction time. This is the bug feature stores were built to eliminate.
What Do You Think?
Your team is training a churn model. Offline AUC is 0.94; production AUC is 0.71. You verify the model code, the feature definitions, and the production training data — all match. The most likely diagnosis?
Feature versioning. When you change a feature's definition (avg_purchases_per_week now excludes returns), you don't want to silently invalidate every model trained on the old definition. The feature store assigns the new version a new identity; old models keep using the old version; new models pin the new version. Without this, you get the worst of both worlds — silent retraining drift.
Data validation. Tools like Great Expectations, Deequ, and Soda let you express assertions over data (expect_column_values_to_be_between('age', 0, 120)) and run them as quality gates in the pipeline. Feature stores integrate these — invalid rows are quarantined to a dead-letter queue, not silently passed downstream.
ML metadata stores like MLMD (TFX) and OpenMetadata track the full provenance: which feature version, which training run, which model, which deployment. EU AI Act compliance leans heavily on this lineage.
Quick check
A teammate proposes 'we can skip the feature store — we'll just rerun the same SQL in training and inference.' Why is this brittle in practice?
Tests · Verify the DAG fails at the 'validate' task because of the negative purchase. Verify retries work (3 attempts). Verify successful execution lands the parquet file in the feature-store path.
Your data team adopts Feast. They define features in YAML, ingest from Kafka and S3, and serve via Redis. A month later, a model trained against feature version v2 is in production. The data team renames a column upstream and rolls out v3. What protects the production model from breaking?
Production data pipelines are the foundation. Next: feature stores, the system that serves these pipelines' output to both training and inference with guaranteed consistency.