"Stripe processes 250 million API calls a day. Every one of them is a chance to silently lose or duplicate a row. The companies that win at ML aren't the ones with the smartest models — they're the ones whose data ingestion never quietly drops a payment, never double-counts a webhook, and never reads stale data into training. Ingestion correctness compounds; ingestion bugs compound faster."
Learning Objectives
After this lesson, you will be able to:
Identify the four major places ML data lives — APIs, databases, files, and event streams — and pick the right ingestion pattern for each based on freshness, scale, and cost
Choose between row-oriented (CSV, JSON) and columnar (Parquet, ORC) file formats based on whether you'll read whole rows or just a few columns at a time
Pull data reliably from rate-limited paginated APIs using retries with exponential backoff, idempotency keys, and incremental cursors so retries don't double-count records
Pick a sampling strategy — uniform, weighted, stratified, or reservoir — that gives your model an honest view of the data distribution without blowing up storage or training time
Don't worry if APIs, databases, and streams feel like four different worlds — they kind of are, but the same patterns (paginate, retry, validate, store) show up in all of them.
Every ML project starts by pulling data out of one or more of these four worlds. The mechanics differ, but the contract is the same: you need to extract data faithfully (no quiet drops, no quiet duplicates), reliably (network failures shouldn't corrupt anything), and observably (you need to know when extraction breaks).
APIs are how you pull data from someone else's running system in real time. They are the most common source for ML data in the wild because most data lives in SaaS tools today.
The four ingestion patterns for APIs
Pull (you call them): Standard REST/GraphQL. You send a request, they return data. Used for one-shot extracts and scheduled batch pulls.
Push / Webhooks (they call you): They send events to your endpoint when something happens. Used by Stripe, GitHub, Slack, and most event-driven SaaS.
Long-polling / SSE / WebSocket: You hold a connection open and they stream events down it. Used by Twitter/X firehose, Slack RTM, and live feeds.
GraphQL: One endpoint, you specify exactly which fields you want. Less wasted bandwidth, but more complex to cache.
#Pagination, Rate Limits, and Retries: the three things that always bite you
APIs almost never return your full dataset in one response. They paginate (return results in chunks), rate-limit (cap how many requests you can make per minute or hour), and fail intermittently (network blips, server hiccups). Production-grade ingestion has to handle all three.
Rtotal=⌈N/S⌉where N=total records,S=page size
delayn=min(base⋅2n+jitter,cap)
What Do You Think?
Your script extracts 50,000 records from a paginated API. Halfway through page 250, the network drops and your script crashes. You restart it. What's the safest way to resume?
The right answer is resume from a cursor or timestamp the API gave you on the last successful request. This is called incremental extraction. Page numbers can shift if records get inserted or deleted between calls; cursors and timestamps don't. Most modern APIs (Stripe, GitHub, Notion) return a next_cursor or updated_after parameter for exactly this reason.
Databases are where structured business data lives. The first distinction to learn:
OLTP (Online Transaction Processing) -- PostgreSQL, MySQL, SQL Server. Optimized for many small reads/writes (your app's user table). Row-oriented.
OLAP (Online Analytical Processing) -- BigQuery, Snowflake, Redshift, ClickHouse. Optimized for big aggregate queries (revenue by month). Columnar under the hood.
You almost never train ML directly off the production OLTP database -- your training queries would slow down the app. Instead, you extract from a read replica, a CDC feed, or a warehouse mirror of the production data.
Full extracts:SELECT * FROM users once a day. Simple, but expensive at scale.
Incremental extracts:SELECT * FROM users WHERE updated_at > :last_run. Requires a reliable update timestamp on every row. Most production tables have one.
Change Data Capture (CDC): Listen to the database's transaction log directly (Postgres WAL, MySQL binlog) using tools like Debezium. You get every insert, update, and delete in near real time, without slamming the database with queries.
For ML, CDC is the gold standard when you need fresh features at low latency, because it gives you a stream you can replay and reprocess.
When you need data within seconds of it happening -- fraud detection, recommendation systems reacting to live behavior, autonomous vehicle sensors -- you need streams.
A stream is an append-only log of events. Producers write to the end; consumers read forward through it. Each event has an offset (position in the log) so consumers can resume exactly where they left off after a crash.
The streaming ML pattern:
App produces events to Kafka (clicks, purchases, sensor readings)
A streaming job (Flink, Spark Streaming) reads events, computes features
Features land in a feature store with low-latency lookup
Online inference reads from the feature store at request time
You don't need streams for most ML projects. Use batch unless your business requires sub-minute freshness. A daily batch pipeline is 10x simpler and 10x cheaper than a streaming one. Streams earn their cost only when stale predictions cost real money.
Real ML data almost never lives in one table. You'll join users with orders with events with products. The mechanics of joining are easy; the cardinality is what bites people.
∣A⋈B∣≤∣A∣×∣B∣
The four join types you actually use:
Inner join: Keep only rows that match in both tables. Loses unmatched rows on either side. Use when you genuinely need data from both.
Left join: Keep all rows from the left table; fill nulls where the right table has no match. The most common ML join -- "for every user, attach their last 30 days of orders, even if they have none."
Full outer join: Keep everything from both. Rare in ML.
Anti-join: Keep rows in the left that have no match in the right. Useful for "users who never placed an order."
A common ML mistake is fan-out: a user has 5 orders, you left-join orders into the user table, and now your "user" rows are duplicated 5x. Aggregate before joining (SUM(order_total) GROUP BY user_id) when you only want one row per entity.
When your full dataset is too big to train on directly, you sample. The sampling strategy determines whether your model sees an honest distribution or a distorted one.
The four sampling strategies:
Uniform random: Every row has equal probability. Default choice, but bad for rare classes.
Weighted: Each row has a probability proportional to some weight. Useful when records have different importance.
Stratified: Sample within each class proportionally (or equally). Guarantees rare classes are represented. Always your starting point for imbalanced data.
Reservoir: Sample k items from a stream of unknown length, with each item having equal probability. Used when you can't store the full stream.
Walk through how data flows from these four worlds into one ingestion layerInteractive
Loading visualization...
Notice how the early stages -- Sources, Extract -- are exactly where this lesson lives. Everything you've learned about APIs, databases, files, and streams plugs into the "Sources" box. The patterns from later stages (validation, transformation, monitoring) come up in the next few lessons.
Four worlds, one contract. APIs, databases, files, and streams all need extraction that is faithful (no quiet drops), reliable (retries don't corrupt), and observable (you know when it breaks). The mechanics differ; the discipline doesn't.
Pagination, rate limits, and retries always show up. Build exponential backoff with jitter, idempotency keys, and cursor-based resumption into your ingestion code from day one. Bolting them on later means rewriting the whole thing.
Pick file format by query pattern. Parquet (or ORC) for tabular ML and analytics, JSON/JSONL for nested log data, CSV only for tiny human-editable files, Avro for evolving streaming schemas. Default to Parquet.
Sample with care. Uniform sampling silently breaks rare-class problems. Use stratified sampling whenever class balance matters, and reservoir sampling when the stream length is unknown.
Use streams only when you need them. Daily batch pipelines are 10x simpler and 10x cheaper than streaming pipelines. Streams earn their complexity only when stale predictions cost real money in real seconds.
An API returns 100 records per page and you need 1M records. The rate limit is 60 requests per minute. What's the minimum wall-clock time to extract everything, ignoring retries?
Modern columnar formats (Parquet, Iceberg, Delta) and zero-copy engines (DuckDB, Polars) have collapsed the cost of "store everything raw and decide later" -- which is why every serious ML team in 2026 ingests into a lakehouse before doing any feature engineering.
You can pull data out of any source now. Next up: Exploratory Data Analysis -- what to actually look at once the data has landed, and why staring at your data for an hour beats jumping straight to a model every time.