Causal Inference: ATE, Propensity Scoring & Double ML
A pharma company runs a regression and finds patients who took drug X recovered 12 days sooner. Ship it? An ad team computes that users who saw the new banner converted at 8% versus 3% for the rest. Roll it out everywhere? An ed-tech startup notices students who use the practice app score 18 points higher on the final exam. Buy more app licenses? Every one of these conclusions is wrong — not because the numbers are fake, but because the question being asked ("does X cause Y?") is fundamentally different from the question the data answered ("are X and Y correlated?"). This lesson is the bridge between the two.
Learning Objectives
After this lesson, you will be able to:
Articulate the fundamental problem of causal inference — that you only ever observe one of two potential outcomes per unit — and use the Rubin potential-outcomes framework to define ATE, ATT, and CATE precisely
Recognize the three pillars of identification (unconfoundedness, overlap/positivity, SUTVA) and explain how a confounder, a collider, and a mediator each play a different role in a DAG
Choose between regression adjustment, propensity-score methods (matching / IPW / stratification), doubly robust estimators (AIPW), and Double / Debiased ML — and justify the choice from the data, not the vibes
Simulate a confounded dataset where naive OLS is biased, recover the true ATE with IPW, and implement Double ML by hand using cross-fitted nuisance estimators
Diagnose the four classic failure modes — collider conditioning, positivity violations, unmeasured confounding, and conditioning on a post-treatment variable — and know which sensitivity analysis to reach for
Don't worry if "causal inference" feels like an entirely new branch of statistics — in some ways it is, but the tools (logistic regression, regression, cross-validation, ensembles) are all things you have already met. What is new is the way we frame the question: not "what can I predict?" but "what would happen if I intervened?".
A growth team at a subscription company ran a regression on six months of historical data: customers who received a 20% coupon had a 14% higher retention rate than those who did not. The team concluded that coupons drive retention and built a campaign that mailed coupons to every customer at risk of churn. Retention dropped by 4%.
What happened? The original 14% lift was not caused by the coupons. The marketing system had been targeting coupons at the most loyal customers — people who were already going to renew. The coupon recipients had higher retention because they were already a self-selected high-retention group. When the team mailed coupons indiscriminately, they paid the discount cost without the underlying behavioral lift, and the gross margin collapsed.
This is the story of correlation versus causation in production: a confounder (customer loyalty score) was simultaneously causing both the treatment assignment (who got the coupon) and the outcome (who renewed). A regression of retention ~ coupon will happily measure their correlation; a causal estimate of "what would happen if we sent everyone a coupon?" requires accounting for the confounder. That is the entire subject of this lesson.
Donald Rubin formalized this in the 1970s. For every unit i and a binary treatment T ∈ {0, 1}, we postulate two potential outcomes:
Y_i(1) — what would happen to unit i if treated
Y_i(0) — what would happen to unit i if not treated
The observed outcome is then:
Yi=Ti⋅Yi(1)+(1−Ti)⋅Yi(0)
The individual treatment effect Y_i(1) - Y_i(0) is therefore never observable for any single unit. But aggregates of it are estimable, under assumptions.
The third estimand is the conditional average treatment effect (CATE), also called the individual treatment effect when X is rich enough to identify a single person:
τ(x)=E[Y(1)−Y(0)∣X=x]
Estimand
Population averaged over
Typical use
ATE
Everyone
Roll-out / policy decisions
ATT
Those who were treated
Program evaluation
ATU
Those who were untreated
Counterfactual policy
CATE / ITE
Subgroups defined by X
Personalization, targeting
What Do You Think?
A pharma company runs a trial of a new diabetes drug. The trial enrolled only patients with HbA1c > 9% (severe). The trial estimates an ATE of -1.4 percentage points reduction. Marketing wants to claim that the drug 'reduces HbA1c by 1.4 points on average for diabetic patients.' What is the most important caveat?
(Y(0), Y(1)) ⊥ T | X — given the observed covariates X, the potential outcomes are independent of the treatment assignment. Said differently: once you condition on X, the treatment looks "as if random."
This is the load-bearing assumption. It is untestable from data — you have to argue it from domain knowledge. If a key variable that influences both T and Y is missing from X, your estimate is confounded no matter how fancy your estimator.
0 < P(T = 1 | X = x) < 1 for every x in the population. For every covariate profile, there must be at least some chance of being treated and at least some chance of not being treated. Otherwise, you have no comparable controls (or no comparable treated) for that profile, and the counterfactual is genuinely unidentified.
In practice: if a feature like is_premium_customer perfectly predicts who gets the coupon (P(T=1 | premium) = 1), there is no way to learn what would happen to a premium customer who didn't get one. The data carries no evidence.
#3. SUTVA (Stable Unit Treatment Value Assumption)
Two parts: (a) no interference — one unit's treatment does not affect another unit's outcome; (b) one version of treatment — there is a single, well-defined treatment, not a mixture of variants.
Interference is the big one and is violated all over the place: network effects (vaccinating Ana protects Ben), marketplace spillovers (a discount to one seller cannibalizes another), peer effects in classrooms. Whenever SUTVA is implausible, you need cluster randomization, marketplace experiments, or specialized estimators.
Quick check
A social network runs an experiment where 50% of users randomly see a new 'reactions' button. After 4 weeks, treated users post 12% more. The team plans to ship it. What is the most defensible critique?
A directed acyclic graph (DAG) is a quick sketch of the causal structure you believe holds. Nodes are variables; arrows are direct causal influences. No need for fancy graphics — drawing this in your notebook is enough.
text
age (W)
/ \
v v
smoking (T) ---> cancer (Y)
Here age causes both smoking (older cohorts smoked more) and cancer (older bodies, more cancer). age is a confounder — it shares an arrow into both T and Y. To estimate the causal effect of smoking on cancer, you must adjust for age, otherwise the smoking coefficient will absorb age's contribution.
But there is a trap. Consider:
text
smoking (T) ---> hospital admission (S) <--- car accident (Y)
hospital admission is a collider — both T and Y point into it. Conditioning on (filtering by, or adjusting for) a collider opens a spurious path between T and Y. If you condition on S, you will find a spurious correlation between smoking and car accidents that does not exist in the population at large. This is collider bias, also called Berkson's bias.
Toggle the confounder, mediator, and collider on this graph to see which adjustments open or block a path between treatment and outcome.
Loading visualization...
What Do You Think?
Researchers want to estimate the causal effect of smoking on lung cancer. They have data on smoking status, lung cancer diagnosis, age, sex, occupation, AND a 'lung function test' (FEV1) measured 10 years after smoking initiation. Which variables should they adjust for?
When you can randomize, do. Random assignment guarantees T ⊥ (Y(0), Y(1))by construction (no confounders, observed or unobserved), and the simple difference of group means is an unbiased estimator of ATE:
ATERCT=YˉT=1−YˉT=0
RCTs are the gold standard. A/B testing at tech companies is just RCT-at-scale, and the entire field of online experimentation is downstream of this. The catch: many treatments cannot be randomized (you cannot randomly assign smoking, or income, or which school a child attends). That is where the rest of this lesson lives.
The most familiar starting point. Fit a model Y ~ T + X and read off the coefficient on T. If the model form is correct and you have included all confounders, that coefficient is the ATE.
ATEReg=n1i=1∑n[μ^(1,Xi)−μ^(0,Xi)]
Strengths: simple, efficient when the model is right, gives uncertainty intervals via standard errors. Weaknesses: if mu(t, x) is misspecified (wrong functional form, missing interactions), the estimate is biased. Linear regression assumes a constant treatment effect across X; if effects are heterogeneous, that constant is a weighted average that may not equal the ATE.
The propensity score is e(x) = P(T = 1 | X = x) — the probability of receiving treatment given the covariates. Rosenbaum & Rubin proved in 1983 that, under unconfoundedness, conditioning on the scalar e(x) is sufficient to remove confounding — you do not need to condition on the full X vector. This collapses a high-dimensional adjustment problem into a one-dimensional one.
For each treated unit, find the nearest-neighbor control unit on e(x), and compute the average difference in outcomes across matched pairs. Clean conceptually; brittle when overlap is poor; loses data for unmatched units.
Re-weight each unit by the inverse of its probability of receiving the treatment it actually received. The weighted sample then mimics a randomized experiment:
IPW's appeal is that, under unconfoundedness and overlap, it is unbiased even when the outcome model is mis-specified. Its weakness is its sensitivity to extreme propensities: a unit with e(x) = 0.01 gets weight 100, and one outlier can swing the estimate badly. The fix is propensity trimming, clipping, or stabilized weights.
Bin units into 5 (or so) strata by propensity score, compute the treated-minus-control difference inside each stratum, and average across strata weighted by stratum size. Crude but robust; the standard default in older epidemiology papers.
What Do You Think?
You estimate propensity scores for an ad-lift study. In the treated group (saw the ad), 30% of users have e(x) > 0.95. In the control group (did not see the ad), 25% have e(x) < 0.05. What is the right move?
What if your outcome model OR your propensity model is wrong? Augmented IPW (AIPW) combines both, with a beautiful property: as long as at least one of the two models is correctly specified, the estimator is consistent.
If you want to use modern ML (Random Forests, gradient boosting, neural networks) to estimate the nuisance functions mu(t, x) and e(x), you run into a problem: ML methods regularize, and that regularization bias contaminates the treatment-effect estimate. The classical solution would be to use unbiased estimators for the nuisance functions, but those are exactly what we cannot easily build in high dimensions.
Double / Debiased Machine Learning (Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey, Robins 2018) solves this with two ingredients:
Neyman-orthogonal scores — moment conditions whose first derivative with respect to the nuisance functions is zero at the true value. Small errors in the nuisance estimates do not cascade into large errors in the treatment-effect estimate. AIPW is one such orthogonal score; the more general DML framework extends this.
Cross-fitting — split the data into K folds; for each fold, estimate the nuisance functions on the other K-1 folds and apply them on the held-out fold. This removes overfitting bias: the same data is never used to both fit and apply a nuisance estimator.
ATE is one number. The world is heterogeneous: a coupon that lifts conversion by 5% on average might lift it by 20% for price-sensitive users and 0% for everyone else. CATE estimators target tau(x) = E[Y(1) - Y(0) | X = x].
The four classical "meta-learners":
S-learner. Fit a single model mu(t, x) (treatment is just another feature). CATE is mu(1, x) - mu(0, x). Simple, biased toward zero when T has weak influence relative to other features (regularization shrinks the treatment coefficient).
T-learner. Fit two separate outcome models: mu_1(x) on the treated and mu_0(x) on the controls. CATE = mu_1(x) - mu_0(x). No regularization toward zero, but the two models can fit very different functional shapes and the difference is noisy.
X-learner (Künzel et al. 2019). Like T-learner but with a clever cross-imputation step: impute counterfactuals for each group using the other group's model, then fit a final model on the imputed CATEs. Better when treatment groups are imbalanced.
R-learner (Nie & Wager 2021). Built on the Robinson decomposition (the same residualization idea as DML), fits CATE by minimizing a doubly robust loss on cross-fitted residuals. The current state of the art for tabular heterogeneous-treatment-effect estimation.
Method
When it shines
When it fails
S-learner
Strong, clear treatment signal
Treatment effect is small relative to outcome variance
T-learner
Balanced group sizes, similar models on both arms
Imbalanced groups, noisy difference
X-learner
One group much smaller (e.g., rare treatment)
Bias if base learners are mis-specified
R-learner
Heterogeneous effects across many features
Small samples — residualization eats degrees of freedom
In practice: start with R-learner via EconML or DoubleML libraries. If you must roll your own, T-learner is the easy default.
Already covered. The fix is: draw the DAG; only condition on pre-treatment variables; never condition on anything caused by both the treatment and the outcome.
If e(x) is near 0 or 1 for some x, IPW blows up. Trim to common support (drop units outside [0.05, 0.95] propensity range), report the trimmed estimand explicitly, and check overlap with a propensity-score histogram by treatment group before estimating.
This is the elephant in the room. Unconfoundedness is untestable; you cannot prove from data alone that you measured every confounder. The honest workaround is sensitivity analysis: ask "how strong would an unmeasured confounder need to be to explain away my result?" If the answer is "it would need to be implausibly strong, with no candidate biology / no candidate variable," the result is robust. If the answer is "even a modest confounder could explain it," your result is fragile.
The two standard formalisms: Rosenbaum bounds (how large a hidden-bias parameter Γ would flip the conclusion; for the smoking-lung-cancer studies, Γ ≈ 6 was required, which is enormous) and E-values (VanderWeele & Ding 2017 — the minimum risk ratio that the confounder would need with both T and Y to explain the observed association). Both give a numerical handle on "how robust is this?".
#4. Conditioning on Post-Treatment Variables (Bad Controls)
Anything caused by the treatment is either a mediator or a collider. Adjusting for it is wrong. Standard examples: adjusting for "job satisfaction" when estimating the effect of a training program on wages (satisfaction is post-treatment); adjusting for "ad engagement" when estimating ad effect on purchase (engagement is post-treatment). The diagnostic is mechanical: was this variable measured before treatment, and is it not caused by treatment? If not, leave it out.
When randomization is impossible AND unconfoundedness is too strong to swallow, quasi-experimental designs exploit a feature of how the treatment is assigned to recover causal estimates with weaker assumptions.
Difference-in-Differences (DiD). Compare the change in outcome before and after treatment in a treated group versus the change in a control group. The key identifying assumption is parallel trends: absent treatment, the two groups would have moved together. Canonical use: minimum wage changes in one US state, neighboring state as control (Card & Krueger 1994). DiD is the workhorse of modern policy evaluation; the recent econometrics literature (Goodman-Bacon, Callaway & Sant'Anna, de Chaisemartin & D'Haultfœuille) has rewritten how to do it correctly under staggered treatment timing.
Instrumental Variables (IV). Find an instrument Z that affects T but has no direct effect on Y except through T, and is unconfounded with the T-Y error term. Then the IV estimator recovers the local average treatment effect (LATE) on the units whose treatment status was moved by Z (the "compliers"). Canonical use: military draft lottery as an instrument for military service in studying veteran earnings (Angrist 1990). Good instruments are scarce and the assumptions are hard to defend; weak-instrument bias is a well-known landmine.
Regression Discontinuity (RD). When treatment is assigned by a sharp cutoff in a running variable (test scores above 70 get scholarships; loans approved when credit score crosses 620), the units just above and just below the cutoff are essentially identical except for treatment. The discontinuity in Y at the cutoff is the treatment effect. Canonical use: scholarship eligibility, election close-call studies (Lee 2008), school admissions cutoffs. Beautiful when the cutoff is sharp and unmanipulated; falls apart when units sort around the cutoff.
These three (DiD, IV, RD) plus RCTs and propensity-based methods cover ~90% of applied causal inference work today. Each makes a different identifying assumption; the art is matching the assumption to the institutional details of the setting.
A team has fit a Random Forest outcome model mu_hat(t, x) and a Random Forest propensity model e_hat(x). They take mu_hat(1, x) - mu_hat(0, x), average over the sample, and call it the ATE. Their colleague suggests switching to AIPW with the same two nuisance models. What is the main benefit?
#Causal Inference in Production: What the Toolkit Actually Looks Like
The fundamental problem of causal inference is missing data. You observe at most one of Y(1) and Y(0) per unit; every estimator is a recipe for imputing the missing potential outcome using comparable other units, under assumptions you cannot test from the data alone
Three assumptions buy you identification. Unconfoundedness (no hidden common causes), overlap (every covariate profile has some treated and some untreated units), and SUTVA (no interference, one version of treatment). Lose one and the estimate is no longer causal
Propensity scores collapse the adjustment problem to one dimension. IPW, matching, and stratification all work because Rosenbaum & Rubin showed that conditioning on e(x) is sufficient under unconfoundedness; the catch is that you need overlap and the propensity model has to be roughly right
Doubly robust + cross-fitting = honest ML-flavored causal inference. AIPW gives you two chances to specify a model correctly; cross-fitting in DML lets you use Random Forests and gradient boosting for the nuisances without the regularization bias contaminating the treatment effect, and you keep valid confidence intervals
The most common production mistake is adjusting for the wrong variable. Colliders and post-treatment variables (mediators) flip causal estimates more often than missing confounders do. Draw the DAG first; for every candidate adjuster, verify it was measured before treatment and is not caused by treatment
Which of these does NOT belong in a list of valid pre-treatment confounders to adjust for in an observational study of 'did the loyalty program raise spend?'
You now know the difference between "correlated" and "caused," and the toolkit (potential outcomes, propensity scoring, AIPW, Double ML, quasi-experiments) that actually closes the gap. Every A/B test, every observational lift study, every CATE-driven personalization system in production builds on these ideas. Next stop: time series forecasting — where the unit of analysis is a moment in time, and the confounders include the past itself.