Okay, so in earlier posts I laid out this factor I hold working into. An individual will kind of say this:
If I run diff-in-diff with out covariates, and discover a quantity, however then run it with covariates and discover a completely different quantity, that is dangerous and diff-in-diff is invalid.
It’s a intestine test, vibe factor for positive. In fact, although, there’s such a factor as “conditional parallel tendencies” and I gave an instance a month in the past. My instance was assume two teams — men and women — every of whom are on completely different earnings, Y, tendencies. However there are two tendencies in outcomes per group. And they’re this:
-
Y(0) tendencies. These are the tendencies in earnings if untreated. Say “highschool solely tendencies in earnings”.
-
Y(1) tendencies. These are the tendencies in earnings if handled. So “faculty educated employee earnings tendencies”.
See for diff-in-diff, because the goal parameter is the ATT, then you need to ask your self — which of these two random variables are we involved about? And the reply is whichever one is lacking. And for the ATT, we’re lacking this one:
Lacking counterfactual: E[Y(0)|D=1]
We’re lacking the Y(0) pattern, and we’re lacking it for a inhabitants of people that have been handled which on this case is the faculty educated employees.
However for diff-in-diff it’s much more slim than that. The diff-in-diff recall is a 2×2, or as I wish to say “4 averages and three subtractions. Nicely guess what the parallel tendencies equation is? The parallel tendencies equation is itself a 2×2. It’s curiously itself “4 averages and three subtractions”. The truth is, in lots of circumstances finest I can inform, the calculation to estimate a remedy impact has a range bias time period that’s merely the identical calculation measured on the lacking counterfactual. So whenever you calculate diff-in-diff:
DiD = Delta Y_1 – Delta Y_0
the place Delta Y_1 is “after minus earlier than earnings for the faculty educated employees”, and Delta Y_0 is “after minus earlier than earnings for the highschool educated employees.” Guess what that equals:
DiD = ATT + PT equation
The left hand aspect is as I stated Delta Y_1 – Delta Y_0. However you realize what the parallel tendencies (PT) equation is?
PT equation = Delta Y(0)_1 – Delta Y(0)_0
Lots of people don’t this, however do you know that the parallel tendencies equation is itself a diff-in-diff? That’s proper. The bias of diff-in-diff is one other diff-in-diff. It’s simply that it’s a diff-in-diff on Y(0), which recall is “tendencies in earnings if highschool just for the faculty educated employee” minus “tendencies in earnings if highschool just for the highschool solely employee”.
That’s what makes it a range bias time period. As a result of choice bias for estimates of the ATT are merely the unique calculation, measured in realized final result Y, however this time for Y(0), which is lacking for the remedy group!
However isn’t it attention-grabbing nonetheless? It’s attention-grabbing to me that we are saying “parallel tendencies” however actually what it truly is solely one other diff-in-diff!
Nicely take a look at it once more shut. The bias is variations in common Y(0) tendencies for 2 teams. What does that imply? Nicely, for one, it means this:
Common Y(0) pattern for remedy group doesn’t equal common Y(0) for management group
I meant that’s what it means actually. It means the technique of two group’s Y(0) tendencies (after minus earlier than within the Y(0) final result) shouldn’t be the identical. It doesn’t imply the tendencies in Y will not be the identical, as a result of variations. within the tendencies in Y is actually the diff-in-diff calculation and if there’s any remedy impact, then beneath parallel tendencies that distinction is the ATT. No, that is an imbalance in counterfactual Y(0) tendencies. Counterfactual as a result of Y(0) — “highschool solely earnings” — is lacking for the remedy group (faculty educated) the interval after they obtained their faculty diploma as a result of the interval after they obtained their faculty diploma, their Y grew to become Y(1) not Y(0).
So, that is precisely why parallel tendencies is solely a query about covariates. Why? As a result of what are the covariates which are chargeable for tendencies in Y(0) and are these covariates unequally distributed throughout the 2 teams.
Apologies for yelling with a bunch of daring and italics however I get excited!
Anyway, so right here is my instance. All highschool educated males earn +10 {dollars} yr to yr. They get raises, in different phrases. However highschool educated females earn +8 {dollars} a yr. What did I simply write down? Developments in Y(0) for 2 teams — men and women.
Okay in order that’s “heterogeneous group tendencies in Y(0)”. That isn’t imbalance. Imbalance is concerning the faculty educated employees versus the highschool educated employees in Y(0). And if the males are simply as equal within the remedy group than within the management group, then that covariate is irrelevant to a parallel tendencies violation. Why? As a result of watch. Let’s say 40% of all faculty educated employees are male and 40% of all highschool educated employees are male. Then test this out:
School educated pattern is: 0.4 x 10 + 0.6 x 8 =8.8
Highschool educated pattern is 0.4 x 10 + 0.6 x 8 =8.8
And thus 8.8 – 8.8 = 0. Why does that matter? As a result of the distinction within the imply pattern in Y(0) in D=1 and D=0 alongside these dimensions is balanced and thus parallel tendencies holds.
what meaning? Meaning even with heterogenous tendencies in Y(0) by observable teams, you truly don’t want to manage for intercourse. As a result of intercourse is balanced and despite the fact that there are heterogenous pattern by intercourse, they’re neutered, they’re canceled out due to the stability.
That’s precisely the identical purpose why in an RCT you don’t have to manage for covariates until the randomization is conditional on these covariates or if these covariates are so deeply correlated with the result that together with them can scale back the usual errors. My previous colleague, Rebecca Thornton, would say that so much when she would clarify this to her college students and it simply at all times caught with me that she was saying it. It’s the identical although with diff-in-diff. It’s the very same factor. Simply because you could have covariates which are predictive of tendencies and even trigger these tendencies shouldn’t be sufficient of a justification to situation on it in a mannequin.
And actually, I’d warning you from doing that, particularly if you’re utilizing Callaway and Sant’Anna. Why? As a result of you need to pay for these covariates. They aren’t free. There isn’t any such factor as a free lunch. In CS, you usually are incorporating these covariates into the mannequin with a propensity rating and logistic regressions really want a good quantity of handled items per covariate. Generally as many as 7 or 10 handled items per covariate. Nicely, in state stage panel knowledge, the US solely has 50 states within the first place! So how rapidly does that collapse you suppose into the curse of dimensionality and make it such that you simply actually are getting nonsensical coefficients on the logit (which btw are suppressed anyway as output in many of the CS packages I do know of so that you don’t even see first hand these coefficients anyway).
However let’s say you employ regression adjustment. Nicely, simply know that for those who’re utilizing regression adjustment, you don’t actually get out of jail with the curse there both. The curse is at all times there. It’s simply that for those who use covariates in OLS specs, you’re going to be figuring out off the purposeful kind as a result of in case you have the curse of dimensionality occurring, you’re going to be projecting — OLS recall is one of the best linear predictor — into an area the place there isn’t any knowledge. So that you higher be fairly assured about that regression specification and its purposeful kind as a result of it’s going to be imputing the Y(0) pattern for the handled group off the Y(0) fitted pattern for the management group. Dropping even one polynomial or one interplay shall be technically an incorrect specification and may land you anyplace — particularly in 2026 when nobody is keen to calm down a perception in heterogeneity.
So, what then can we do? Right here’s what we do.
-
We test if the covariates that we expect are inflicting the Y(0) tendencies are balanced for the remedy and management group at baseline, after which
-
We estimate a propensity rating, plot the histogram, test the distribution, see if there’s each overlap and if the max worth of the propensity rating within the management group is nearly 1. As a result of whether it is nearly 1, then the inverse chance weight will explode.
Why? As a result of the load in a diff-in-diff with inverse chance weights, which the unique Abadie 2005 Restud had (“Semi Parametric Diff-in-Diff”) and the Sant’Anna and Zhao 2020 and Callaway and Sant’Anna 2021 even have, is the same as p(x)/[ 1- p(x) ], and that weight solely applies to the management group in diff-in-diff. Why solely the management group? As a result of, we’re estimating the ATT, and nothing is mistaken with the remedy group’s final result. We’re solely lacking the counterfactual for the handled group, and we’re utilizing the management group to get it because the management group is the one group that has the Y(0).
Nicely, take a look at what occurs to that weight if the propensity rating for the management group is “nearly 1”. Assume it’s 0.999, which you’ll be able to simply get with quite a lot of heterogeneity in these Xs, weak to no help for a few of them, and a big pattern of management items. 0.999/[0.001] = 999.
What’s 999? That’s the load on a single unit’s final result. A single unit’s first distinction final result in diff-in-diff utilizing IPW will develop into that final result occasions 999. That’s an outlier, it has huge leverage over the estimate. And you must test it! You might want to know if it’s there as a result of that single statement might actually flip the signal or drive it to who is aware of the place. What if that one unit is Elon Musk!
We speak about this in part 4.2 of our Journal of Financial Literature. I encourage you to learn it carefully. I’m nonetheless not fairly prepared but to undergo my empirical instance to indicate you conclusively for my part about easy methods to convincingly present you, and not using a simulation thoughts you, that correcting for covariates matter, but additionally how you right for covariates issues simply as a lot, however I’m going to. Simply needed to get that rant out of my system first!
