Why Your Finest Predictive Mannequin Offers the Fallacious Therapy Impact

A query that sounds simple however isn’t

the flawed purpose. Let’s say we wish to estimate the impact of an publicity on an final result, e.g., air air pollution on hospital admissions or a drug versus time to restoration. We’ve got observational knowledge, one publicity variable, and a pile of candidate covariates. Some are confounders. Some are noise. Now the most important query is — which of them go within the mannequin?

The same old reflex arms the selection to a variety rule. Run Stepwise routines, examine BIC values, and even cross-validate Lasso. Full Bayesian averaging proceeds over candidate specs.

Nonetheless, the issue with these approaches is that each one among these instruments optimizes predictive match. A mannequin can predict the result superbly whereas being systematically flawed concerning the impact we care about.

This difficulty was the topic of a 2012 Biometrics paper by Chi Wang, Giovanni Parmigiani, and Francesca Dominici, who referred to as the underlying difficulty adjustment uncertainty. It’s the uncertainty about which variables to regulate for, and so they proposed a repair referred to as Bayesian Adjustment for Confounding (BAC) [1]. It additionally has a well-known frequentist twin that most individuals don’t realise is similar perception: post-double-selection and double machine studying [3][4].

On this submit, I’ll display three issues. I’ll present you the failure mode in runnable Python, construct the repair, after which lastly present you the repair’s personal sharp edge: the case the place “regulate for something that predicts publicity” is strictly the flawed recommendation and makes your estimate worse.

If in case you have been studying about propensity scores, subclassification, and matching, that is the query these strategies assume you’ve got already answered: how do you resolve, from knowledge, which covariates a sound adjustment wants?

A great predictor and a confounder will not be the identical

Why does a specification tuned for prediction produce a flawed impact estimate? As a result of the 2 duties reward distinct traits. Prediction favors a covariate that accounts for final result variance. Impact estimation favors a covariate tied to each publicity and final result.

Now, the issue arises when a confounder is strongly related to the publicity however solely weakly with the result. To a predictive criterion, that variable appears to be like almost ineffective as a result of it explains little final result variance. Additionally, a lot of what it explains is already captured by the (correlated) publicity. In sensible finite samples, BIC might want dropping it. However dropping it adjustments what the publicity coefficient means. The coefficient now absorbs the confounder’s path, and the “impact” is now truly contaminated.

Wang and coauthors illustrate the purpose with 5 candidate covariates. U1, U2, and U3 every hyperlink to each X and Y. U4 predicts Y alone. U5 is noise.

Now think about the complete argument lowered to 2 t-statistics. On the simulated knowledge under, the variable U2 exhibits a t-statistic of 1.2 inside the result mannequin. That yields p equal to 0.23. It seems indistinguishable from noise. Contained in the publicity mannequin, the identical variable exhibits a t-statistic of 31.5. Identical variable, similar knowledge set. One mannequin calls it irrelevant. The opposite calls it central.

Each routine we usually invoke examines solely the primary column. This final result follows from arithmetic. Omitted variable bias from excluding U2 equals:

It’s the product of U2’s coefficient within the final result equation occasions its coefficient within the regression of X on U2 (i.e., delta). The primary issue stays small. That’s precisely why BIC stays unmoved. The second issue grows massive as a result of U2 tracks publicity intently. Their product contaminates the goal estimate. No final result match rule ever inspects the second issue. Two portions. Match guidelines register just one.BIC stays choice constant. With adequate observations, it might retain U2. The current issue is finite pattern. The asymptotic disappearance of bias affords little solace when the info set at hand accommodates roughly 1,000 rows. The Code under follows the sample already described.

The failure mode, in Python

Let’s simulate the setup in Determine 1 and match two final result fashions, i.e., the totally adjusted one and one which drops U2.

import numpy as np, pandas as pd
import statsmodels.api as sm

rng = np.random.default_rng(11)
n, M = 1000, 5

U = rng.regular(dimension=(n, M))                              # candidates U1..U5
X = U[:,0] + U[:,1] + 0.1*U[:,2] + rng.regular(dimension=n)    # publicity
Y = 0.1*X + U[:,0] + 0.1*U[:,1] + U[:,2] + U[:,3] 
    + rng.regular(dimension=n)                                 # true impact = 0.1

cols = [f"U{j+1}" for j in range(M)]
df = pd.DataFrame(U, columns=cols); df["X"] = X; df["Y"] = Y

def fit_outcome(consists of):
    feats = ["X"] + [c for c, inc in zip(cols, includes) if inc]
    return sm.OLS(df["Y"], sm.add_constant(df[feats])).match()

full  = fit_outcome((1,1,1,1,0))   # adjusts for U1..U4
noU2  = fit_outcome((1,0,1,1,0))   # silently drops confounder U2

The outcomes (true impact is 0.1):

adjusts for U1,U2,U3,U4   β̂ = 0.144   95% CI (0.083, 0.206)   BIC = 2907.6
drops U2                  β̂ = 0.171   95% CI (0.127, 0.214)   BIC = 2902.1

Right here, three issues occurred:

1. The mannequin lacking a real confounder has the higher BIC. By the same old logic, it’s the mannequin we’re going to select.
2. Its confidence interval excludes the true worth. We’d report a major impact of ~0.17 with confidence and truly be fully flawed.
3. The totally adjusted mannequin, penalized for carrying a “ineffective” predictor, is the one whose interval truly covers the reality.

The 2 fashions match the info equally effectively. Nonetheless, they estimate completely different parameters. Solely one among them is the causal amount we needed.

Bayesian Mannequin Averaging has the identical blind spot

Suppose we don’t belief any single chosen mannequin and resolve to common them. Bayesian mannequin averaging (BMA) weights every mannequin by its posterior likelihood and blends the estimates.

Right here, the catch is that with the same old flat prior over fashions, the posterior weights are pushed by marginal probability. Once more, it’s the similar criterion that simply betrayed our estimates. Run by way of all 2⁵ = 32 final result fashions, apply the same old BIC weights, and watch the place the likelihood settles.

BMA assigns roughly 90 % of its mass to the model that drops U2. Therefore, we are able to conclude that averaging doesn’t assist when the weights themselves level on the flawed goal. Wang et al. noticed precisely the identical sample of their paper and located that BMA’s 95% intervals coated the reality much less typically than the nominal degree would counsel. [1].

The BAC thought: ask the publicity mannequin who the confounders are

A confounder, by definition, should predict the publicity in addition to the result. This implies we must always not choose covariates utilizing solely the result mannequin; as a substitute, we must always match a second mannequin for the publicity and let that mannequin veto the result mannequin’s choices.

BAC formalizes this with two linked variable-selection issues:

• an final result mannequin: Y as a operate of X and candidate covariates,
• an publicity mannequin: X as a operate of the identical candidates,

These two are joined by a dependence parameter ω, which represents the prior odds {that a} covariate enters the result mannequin on condition that it’s already within the publicity mannequin. At ω = 1, the fashions decouple, and we get well plain BMA. As ω → ∞, any covariate that the publicity mannequin needs is pressured into the result mannequin, which provides us the complete confounding adjustment [1].

If this logic reminds you of propensity scores, that’s completely intuitive. The publicity mannequin is a propensity mannequin, and the textbook recommendation from the causal inference literature consists of covariates associated to therapy project when developing our adjustment [2]. That is precisely the instinct BAC encodes as a previous. The essential distinction is that BAC doesn’t drive us to decide to a single hand‑picked specification. As a substitute, it captures the uncertainty in that choice and propagates it into the ultimate inference.

A minimal, enumerated model of BAC is proven under. Real BAC makes use of MCMC, however with solely 5 covariates we are able to brute‑drive each attainable mannequin mixture.

from itertools import product

def bic_weights(bics):
    b = np.array(bics)
    w = np.exp(-0.5 * (b - b.min()))
    return w / w.sum()

# All 32 final result fashions
out_models, out_bics, betas = [], [], []
for inc in product([0,1], repeat=M):
    m = fit_outcome(inc)
    out_models.append(inc); out_bics.append(m.bic)
    betas.append(m.params["X"])
w_bma = bic_weights(out_bics)

# All 32 publicity fashions -> per-covariate inclusion likelihood
exp_bics, exp_models = [], []
for inc in product([0,1], repeat=M):
    feats = [c for c, i in zip(cols, inc) if i]
    Xm = sm.add_constant(df[feats]) if feats else np.ones((n,1))
    exp_bics.append(sm.OLS(df["X"], Xm).match().bic)
    exp_models.append(inc)
w_exp = bic_weights(exp_bics)
incl_prob = [sum(w for w, inc in zip(w_exp, exp_models) if inc[j])
             for j in vary(M)]

# BAC-style prior with omega -> infinity:
# final result fashions should include each covariate the publicity mannequin calls for
required = [p > 0.5 for p in incl_prob]
okay = np.array([all(inc[j] for j in vary(M) if required[j])
               for inc in out_models])
w_bac = w_bma * okay
w_bac = w_bac / w_bac.sum()

print("publicity inclusion probs:", np.spherical(incl_prob, 3))
print("BMA estimate:", np.dot(w_bma, betas))
print("BAC estimate:", np.dot(w_bac, betas))

The printed output exhibits that the publicity mannequin appropriately recovers the confounder construction:

publicity inclusion probs: [1.    1.    1.    0.031 0.032]
BMA estimate: 0.169
BAC estimate: 0.144

The inclusion possibilities for U1, U2, and U3 are all roughly one, whereas U4 and U5 have possibilities very near zero. When the result‑mannequin weights are conditioned on this information, the therapy impact estimate strikes from 0.169 right down to 0.144. Determine 2 exhibits how the mannequin weights flip decisively towards the totally adjusted specs.

A single dataset may very well be a fortunate draw, so I repeated the entire experiment 300 occasions.

This bias is just not random noise. It’s structural. The prediction‑pushed weights systematically favour underneath‑adjusted fashions, and the ultimate estimates inherit that systematic slant.

The catch: an instrument appears to be like precisely like a confounder

That is the half that almost all write‑ups of BAC skip, and it explains why the ω knob truly exists. The rule “regulate for something that predicts the publicity” is just not a really protected recommendation. Let’s think about a variable Z that predicts X strongly however has no path to Y in anyway besides by way of X. That variable is an instrument, and an publicity mannequin can’t inform it other than a real confounder.

Now let’s see what occurs once we obey the publicity mannequin in a setting the place UC is an unmeasured confounder (as there at all times is in actual knowledge) and Z is a pure instrument:

Uc = rng.regular(dimension=n)                  # unmeasured confounder
Z  = rng.regular(dimension=n)                  # instrument: impacts X solely
X  = Uc + Z + rng.regular(dimension=n)
Y  = 0.1*X + Uc + rng.regular(dimension=n)     # Z does NOT seem right here

Over 500 replications with a real impact of 0.1, the outcomes are hanging:

no adjustment      β̂ = 0.434    (bias +0.334)
regulate for Z       β̂ = 0.601    (bias +0.501)

Adjusting for Z made issues 50% worse. The estimate didn’t turn out to be much less environment friendly however it grew to become extra biased. This phenomenon is called bias amplification or Z‑bias [5][6]. The instinct works as follows: conditioning on Z strips out the a part of X’s variation that was clear (the instrument‑pushed half). This leaves behind a residual X that’s extra dominated by the unmeasured confounder than earlier than. So, we’ve principally thrown away our good variation and saved the contaminated variation, so the residual confounding received concentrated.

At $\gamma =3$, ignoring $Z$ leaves a bias of $+0.09$, whereas adjusting for Z leaves $+0.50$. And we can’t detect this distinction from an publicity mannequin alone. In a simulation the place Z is a pure instrument and C is a real confounder, each coming into X identically, we get hold of the next:

publicity mannequin t-statistics:   Z = 32.2    C = 33.6

These numbers are statistically indistinguishable. Nonetheless, causally they’re reverse. No quantity of information will separate them, as a result of the distinction lies within the DAG, not within the covariance matrix. That is exactly why BAC’s $\omega$ is a dial and never a swap. The $\omega \to \mathrm{\infty }$ setting is the aggressive model that forces in each publicity predictor. A finite $\omega$ softens the rule for publicity. The selection of $\omega$ is a causal assumption we’re making, not a parameter we are able to estimate from the info. If you happen to take one factor from this submit, take this sentence.

The identical thought, 3 ways:

BAC dates from 2012, however the core perception was independently rebuilt in econometrics. It will definitely grew to become Double Machine Studying, and is now used underneath numerous completely different names

1. Publish‑double‑choice (Belloni, Chernozhukov & Hansen, 2014 [3]: The process is kind of easy. We run a lasso of Y on the covariates and run a lasso of X on the covariates. Then take the union of the chosen variables, and refit OLS on that union. The lasso‑on‑X step exists for precisely the rationale BAC’s publicity mannequin exists i.e., to rescue confounders which are weak on Y however sturdy on X. The affect is actual. Utilizing the theoretically motivated plug‑in penalty from the BCH paper on the identical simulation, the result lasso drops U2​ in 100 out of 300 datasets (33% of the time). Single‑choice (lasso on Y solely) yields a imply estimate of 0.112 with bias $+0.012$, whereas double‑choice (the union) yields a imply of 0.101 with bias $+0.001$ and 95% confidence interval protection of 0.94. A 3rd of the time, the result lasso throws away a real confounder however the publicity lasso at all times catches it, and the interval protection returns to nominal.
2. Double/debiased machine studying (Chernozhukov et al., 2018) [4]: This is similar two‑mannequin construction generalised. We predict Y from the covariates and X from the covariates. Then we take the residuals of each, and regress residual‑on‑residual. Cross‑becoming is added to get rid of the overfitting bias. Now the 2 nuisance fashions may be gradient boosting or random forests as a substitute of lassos, and the impact estimate stays ​constant. A sanity test on the identical knowledge‑producing course of confirms that DML lands at 0.101 i.e., the identical place that BAC and double choice attain.
3. Final result‑adaptive lasso (Shortreed & Ertefaie, 2017) [7]: Right here is the counter‑punch, which exists exactly due to the bias‑amplification downside described above. Final result‑adaptive lasso builds the propensity mannequin however penalises every covariate by how weakly it pertains to the result. It intentionally down-weights variables that predict publicity and nothing else. That is the precise reverse intuition from BAC with $\omega \to \mathrm{\infty }$. It’s a wager that pure devices are the larger hazard; BAC’s wager is that dropped confounders are. Each bets are defensible.

So what’s the conclusion?

1. We have to cease letting predictive standards decide our adjustment set. BIC, AIC, lasso paths, and stepwise p-values all rating fashions on final result match, and none of them is aware of what a confounder is. A mannequin can win on all of those metrics and nonetheless hand us a biased impact.
2. Second, we have to match the publicity mannequin even when we by no means run BAC. It prices us 4 traces of code, and it’s the solely diagnostic in our toolkit that may see the suitable‑hand column of that t-statistic desk. When a covariate could be very outstanding within the publicity mannequin however silent within the final result mannequin, we’ve simply discovered the variable that our choice process is about to throw away.
3. However, additionally we must always not blindly obey the publicity mannequin. Earlier than we embrace a robust publicity‑predictor, ask the query the info can’t reply: may this variable have an effect on Y by way of any path aside from X? If the reply is sure, it’s a confounder and we must always embrace it. If the reply is genuinely no, it’s an instrument, and adjusting for it would amplify no matter unmeasured confounding we’ve left. That is the step the place we’ve to suppose, and no estimator will do it for us.
4. The asymmetry between underneath‑adjustment and over‑adjustment ought to form our selections. Over-adjustment for variables which are merely prognostic often prices effectivity, whereas adjusting for devices can truly enhance bias. [1].

Lastly, we have to deal with adjustment uncertainty as actual uncertainty. If our confidence interval is conditional on one hand‑chosen mannequin, it understates what we have no idea.

The uncomfortable abstract is that the instruments most of us attain for first had been constructed for a unique job. Prediction asks, “Which mannequin explains Y?” Causal inference asks, “Which mannequin isolates the impact of X on Y?” These questions have completely different solutions. Figuring out that’s many of the battle, and realizing that the second query carries an assumption that no quantity of information will resolve for us.

References

[1] C. Wang, G. Parmigiani and F. Dominici, Bayesian Impact Estimation Accounting for Adjustment Uncertainty (2012), Biometrics, 68(3), 661–671.

[2] A. Gelman, J. Hill and A. Vehtari, Regression and Different Tales (2020), Cambridge College Press, Ch. 20.

[3] A. Belloni, V. Chernozhukov and C. Hansen, Inference on Therapy Results after Choice amongst Excessive-Dimensional Controls (2014), Assessment of Financial Research, 81(2), 608–650.

[4] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey and J. Robins, Double/Debiased Machine Studying for Therapy and Structural Parameters (2018), The Econometrics Journal, 21(1), C1–C68.

[5] J. Pearl, Invited Commentary: Understanding Bias Amplification (2011), American Journal of Epidemiology, 174(11), 1223–1227.

[6] J. A. Myers, J. A. Rassen, J. J. Gagne, Ok. F. Huybrechts, S. Schneeweiss, Ok. J. Rothman, M. M. Joffe and R. J. Glynn, Results of Adjusting for Instrumental Variables on Bias and Precision of Impact Estimates (2011), American Journal of Epidemiology, 174(11), 1213–1222.

[7] S. M. Shortreed and A. Ertefaie, Final result-Adaptive Lasso: Variable Choice for Causal Inference (2017), Biometrics, 73(4), 1111–1122.