Causal Inference, Part 2: Doubly Robust Estimators, Double ML and Quasi-Experiments
Part 1 gave you the question, the three assumptions and three estimators: a trial, regression and propensity scores. The last two can each fail on their own. If the outcome model is wrong, regression adjustment is biased, and if the propensity model is wrong, IPW is. This part builds estimators that survive one wrong model, shows how to use flexible machine learning without breaking the maths, and ends with what to do when no estimator is enough.
Learning Objectives
After this lesson, you will be able to:
Explain why AIPW is doubly robust, and read its bias as the product of two errors
Implement Double ML by hand with cross-fitted nuisance models, and say what cross-fitting protects against
Choose between the S, T, X and R meta-learners for effects that differ across people
Diagnose the four pitfalls that sink real studies, and reach for a sensitivity analysis when unconfoundedness is in doubt
Pick difference-in-differences, instrumental variables or regression discontinuity from how treatment was assigned, and use the decision table to choose an estimator
In plain words: Part 1 assumed we could model either the outcome or the chance of treatment well enough. Part 2 asks what happens when we cannot be sure of either. The answer is to combine the two models so that one good model is enough, to keep the flexible learners honest by never scoring a person with a model that saw them, and to say out loud how fragile the result would be if a confounder were missing.
Words you need10 terms
A model we must fit along the way, such as the outcome model or the propensity model, but whose own predictions are not what we report.
An estimator that stays correct if either one of its two models is right, so there are two chances to get it right.
Fitting each nuisance model on some folds and using it only on the other folds, so no person is scored by a model that saw them.
What is left after a prediction is subtracted: the actual value minus the predicted one.
Conditional average treatment effect: the average effect for people who share the same characteristics.
A treatment effect that differs from person to person, for example a coupon that helps price-sensitive users and does nothing for others.
A group of people who are always or never treated, so there is nobody to compare them with.
Asking how strong a hidden confounder would have to be to explain away the result.
A variable that moves the treatment but affects the outcome only through the treatment.
The assumption that, without treatment, the treated and control groups would have changed in the same way over time.
What if your outcome model OR your propensity model is wrong? Augmented IPW (AIPW) combines both, with a beautiful property: as long as at least one of the two models is correctly specified, the estimator is consistent.
The "double" in doubly robust is exactly this: two chances to be right. AIPW is the foundation for the next step.
Why do two chances to be right appear? One identity answers it exactly, and it also says how wrong the estimate is when both models are a little wrong. We derive it for the treated half of the formula; the control half is the same with 1 - T.
If you want to use modern ML (Random Forests, gradient boosting, neural networks) to estimate the nuisance functions mu(t, x) and e(x), you run into a problem: ML methods regularize, and that regularization bias contaminates the treatment-effect estimate. The classical solution would be to use unbiased estimators for the nuisance functions, but those are exactly what we cannot easily build in high dimensions.
Double / Debiased Machine Learning (Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey, Robins 2018) solves this with two ingredients:
Neyman-orthogonal scores. Moment conditions whose first derivative with respect to the nuisance functions is zero at the true value. Small errors in the nuisance estimates do not cascade into large errors in the treatment-effect estimate. AIPW is one such orthogonal score; the more general DML framework extends this.
Cross-fitting. Split the data into K folds; for each fold, estimate the nuisance functions on the other K-1 folds and apply them on the held-out fold. This removes overfitting bias: the same data is never used to both fit and apply a nuisance estimator.
In Double Machine Learning, what role does cross-fitting play?
Lab 2 below uses the partialling-out form of Double ML, whose estimator is a ratio of two sums of residuals. Open the derivation to see why that ratio recovers the treatment effect.
ATE is one number. The world is heterogeneous: a coupon that lifts conversion by 5% on average might lift it by 20% for price-sensitive users and 0% for everyone else. CATE estimators target tau(x) = E[Y(1) - Y(0) | X = x].
The four classical "meta-learners":
S-learner. Fit a single model mu(t, x) (treatment is just another feature). CATE is mu(1, x) - mu(0, x). Simple, biased toward zero when T has weak influence relative to other features (regularization shrinks the treatment coefficient).
T-learner. Fit two separate outcome models: mu_1(x) on the treated and mu_0(x) on the controls. CATE = mu_1(x) - mu_0(x). No regularization toward zero, but the two models can fit very different functional shapes and the difference is noisy.
X-learner (Künzel et al. 2019). Like T-learner but with a clever cross-imputation step: impute counterfactuals for each group using the other group's model, then fit a final model on the imputed CATEs. Better when treatment groups are imbalanced.
R-learner (Nie & Wager 2021). Built on the Robinson decomposition (the same residualization idea as DML), fits CATE by minimizing a doubly robust loss on cross-fitted residuals. The current state of the art for tabular heterogeneous-treatment-effect estimation.
Method
When it shines
When it fails
S-learner
Strong, clear treatment signal
Treatment effect is small relative to outcome variance
T-learner
Balanced group sizes, similar models on both arms
Imbalanced groups, noisy difference
X-learner
One group much smaller (e.g., rare treatment)
Bias if base learners are mis-specified
R-learner
Heterogeneous effects across many features
Small samples — residualization eats degrees of freedom
In practice: start with R-learner via EconML or DoubleML libraries. If you must roll your own, T-learner is the easy default.
Covered in Part 1. The fix is: draw the DAG; only condition on pre-treatment variables; never condition on anything caused by both the treatment and the outcome.
If e(x) is near 0 or 1 for some x, IPW blows up. Trim to common support (drop units outside [0.05, 0.95] propensity range), report the trimmed estimand explicitly, and check overlap with a propensity-score histogram by treatment group before estimating.
This is the elephant in the room. Unconfoundedness is untestable; you cannot prove from data alone that you measured every confounder. The honest workaround is sensitivity analysis: ask "how strong would an unmeasured confounder need to be to explain away my result?" If the answer is "it would need to be implausibly strong, with no candidate biology / no candidate variable," the result is robust. If the answer is "even a modest confounder could explain it," your result is fragile.
The two standard formalisms: Rosenbaum bounds (how large a hidden-bias parameter Γ would flip the conclusion; for the smoking-lung-cancer studies, Γ ≈ 6 was required, which is enormous) and E-values (VanderWeele & Ding 2017 — the minimum risk ratio that the confounder would need with both T and Y to explain the observed association). Both give a numerical handle on "how robust is this?".
#4. Conditioning on Post-Treatment Variables (Bad Controls)
Anything caused by the treatment is either a mediator or a collider. Adjusting for it is wrong. Standard examples: adjusting for "job satisfaction" when estimating the effect of a training program on wages (satisfaction is post-treatment); adjusting for "ad engagement" when estimating ad effect on purchase (engagement is post-treatment). The diagnostic is mechanical: was this variable measured before treatment, and is it not caused by treatment? If not, leave it out.
When randomization is impossible AND unconfoundedness is too strong to swallow, quasi-experimental designs exploit a feature of how the treatment is assigned to recover causal estimates with weaker assumptions.
Difference-in-Differences (DiD). Compare the change in outcome before and after treatment in a treated group versus the change in a control group. The key identifying assumption is parallel trends: absent treatment, the two groups would have moved together. Canonical use: minimum wage changes in one US state, neighboring state as control (Card & Krueger 1994). DiD is the workhorse of modern policy evaluation; the recent econometrics literature (Goodman-Bacon, Callaway & Sant'Anna, de Chaisemartin & D'Haultfœuille) has rewritten how to do it correctly under staggered treatment timing.
Instrumental Variables (IV). Find an instrument Z that affects T but has no direct effect on Y except through T, and is unconfounded with the T-Y error term. Then the IV estimator recovers the local average treatment effect (LATE) on the units whose treatment status was moved by Z (the "compliers"). Canonical use: military draft lottery as an instrument for military service in studying veteran earnings (Angrist 1990). Good instruments are scarce and the assumptions are hard to defend; weak-instrument bias is a well-known landmine.
Regression Discontinuity (RD). When treatment is assigned by a sharp cutoff in a running variable (test scores above 70 get scholarships; loans approved when credit score crosses 620), the units just above and just below the cutoff are essentially identical except for treatment. The discontinuity in Y at the cutoff is the treatment effect. Canonical use: scholarship eligibility, election close-call studies (Lee 2008), school admissions cutoffs. Beautiful when the cutoff is sharp and unmanipulated; falls apart when units sort around the cutoff.
These three (DiD, IV, RD) plus RCTs and propensity-based methods cover ~90% of applied causal inference work today. Each makes a different identifying assumption; the art is matching the assumption to the institutional details of the setting.
A team has fit a Random Forest outcome model mu_hat(t, x) and a Random Forest propensity model e_hat(x). They take mu_hat(1, x) - mu_hat(0, x), average over the sample, and call it the ATE. Their colleague suggests switching to AIPW with the same two nuisance models. What is the main benefit?
#Causal Inference in Production: What the Toolkit Actually Looks Like
Doubly robust + cross-fitting = honest ML-flavored causal inference. AIPW gives you two chances to specify a model correctly; cross-fitting in DML lets you use Random Forests and gradient boosting for the nuisances without the regularization bias contaminating the treatment effect, and you keep valid confidence intervals
Effects differ across people. CATE meta-learners turn an average effect into a targeting rule, and the choice between the S, T, X and R learners depends on group sizes and how weak the effect is
Unmeasured confounding cannot be tested away. Report a sensitivity number such as an E-value next to the estimate, and reach for difference-in-differences, instrumental variables or regression discontinuity when the way treatment was assigned gives you a quasi-experiment
What is the central role of Neyman-orthogonal scoring (the AIPW-style influence function) in Double Machine Learning?
You now know the difference between "correlated" and "caused," and the toolkit (potential outcomes, propensity scoring, AIPW, Double ML, quasi-experiments) that actually closes the gap. Every A/B test, every observational lift study, every CATE-driven personalization system in production builds on these ideas. That completes Track 3: you now command the classical arsenal — regression, classification, trees, ensembles, SVMs, clustering, evaluation, and causal reasoning. Next stop: Track 4, Deep Learning, where networks learn the representations you have been hand-engineering all along.