An earlier submit had the incorrect video so I needed to delete it and repost. Apologies for the duplication!
Right this moment’s entry continues to be a sequence on utilizing Claude Code for quantitative social scientific tasks. However not like different explainers on-line, it’s not written from the angle of a software program engineer written to an viewers of software program engineers. Neither is it as a Claude Code influencer speaking abstractly about Claude Code. Reasonably, I’m simply going to be utilizing Claude Code to revive and lengthen an outdated challenge on abortion clinic closures. And the extension shall be to make use of the brand new conditionally accepted AER paper on steady diff-in-diff by Brantly Callaway, Andrew Goodman-Bacon and Pedro Sant’Anna.
I’ll proceed to make these open to the general public, however be aware that after a number of days, all of the posts on right here go behind the paywall, so when you’re new to this, you’ll have to return and subscribe to get caught up. This substack is a labor of affection meant for the neighborhood to be taught extra about econometrics (and now additionally AI brokers for empirical work) in addition to a medium for self expression. So please think about changing into a paying subscriber because it’s solely $5/month or $50/12 months! And thanks everybody for supporting me all these years on this stuff!
So I’m again with one other lengthy video. This one clocks in at about 1.5 hours. I apologize upfront. Be happy to skip round.
Let me clarify what we’re doing right here as a result of I believe it’s value stating upfront: this isn’t meant to be some authoritative explainer on something. It’s extra like a livestream of me utilizing Claude Code to revive an outdated challenge and lengthen it utilizing new strategies. Consider it as watching somebody work by an actual empirical downside in actual time, warts and all.
The paper in query is from the Journal of Human Assets (2019) — me, Andrea Schlosser, Jason Lindo, and Caitlin Myers finding out how Texas HB2 affected abortion entry. We’re taking a look at what occurs when clinics shut and ladies should journey farther to get an abortion. Basic difference-in-differences setup with a steady therapy (distance).
However I’ve bought larger plans. Brantly Callaway, Andrew Goodman-Bacon, and Pedro Sant’Anna have a paper that’s conditionally accepted on the AER on steady difference-in-differences. And I need to take this outdated challenge and run it by that new methodology. Not simply replicate what we did earlier than. Not simply clear up the outdated code. However really re-evaluate and re-interpret what distance does to outcomes utilizing their framework.
So this sequence is doubling as a number of issues without delay:
-
A sequence on utilizing AI brokers (particularly Claude Code) for empirical analysis
-
A sequence on steady diff-in-diff
-
A case examine in conversational challenge administration
Right here’s the factor that was bugging me: Andrea and I had constructed a dataset years in the past for her thesis. However then when Jason and Caitlin and all of us joined pressured on what would finally develop into the printed JHR paper, the journey knowledge utilized in that new paper modified from what Andrea and I had been utilizing in our earlier work. And that’s largely, I believe, as a result of if reminiscence serves, Caitlin had finished meticulous year-by-year clinic monitoring that included out-of-state clinics in ways in which me and Andrea had been lacking as we had been relying totally on Texas licensure knowledge from the state itself, however we had been a lot much less assured concerning the contiguous states location of abortion corporations. So I had two datasets floating round on this folder and I wanted to know precisely what was totally different between them.
Andrea’s thesis knowledge backdated the 2010 distances all the best way again to 2006 which I had solely vaguely remembered in any respect earlier than Claude Code discovered it within the outdated do file doing simply that. We took the 2010 distance, then we simply resaved it a number of occasions as a 2009 dataset, a 2008 dataset, and so forth. earlier than then merging these years to every county. Andrea, I now recall, had defined that was far again as she might discover, and since we had consequence knowledge going again to 2006, we had been going to make the belief that previous to HB2, there had been no clinic closures — which was seemingly incorrect, however that was the belief we had been making. Which implies there’s no within-county variation within the pre-period — and that’s form of an issue when your whole identification technique depends on within-county variation.
However as I stated, the JHR knowledge doesn’t have this downside. Caitlin tracked precise clinic openings and closings 12 months by 12 months utilizing a wide range of sources and shoe leather-based. Actual Jon Snow vitality.
The principle variations, as you’ll see within the deck and the video, is that we had been lacking fairly badly the distances to clinics on the western facet of Texas, particularly distances the place the closest clinic was not inside Texas after HB2, however moderately to clinics in New Mexico and Oklahoma. We match as much as the JHR 92% of the time, however that’s the place we’re incorrect, and also you’ll see within the video me discovering that as I direct Claude Code to determine some issues out for me. however particularly, for counties within the Panhandle, it issues loads. Lubbock exhibits up as 307 miles from a clinic within the thesis knowledge however solely 78 miles within the JHR knowledge. That’s a 229-mile distinction as a result of the thesis missed the nearer clinics throughout the state line. Principally, what’s going on is that me and Andrea in her thesis (and my equal paper alongside hers) had been introducing systematic error in 8% of our knowledge the place we had been imputing too far of journey distances. However the Caitlin and Jason journey distance that they delivered to the challenge doesn’t, and I think about the JHR the “floor reality” so to talk which allowed us to do that systematic side-by-side comparability utilizing Claude Code.
Having Claude Code assist me systematically evaluate these two datasets, generate figures, and pin down precisely the place and why they diverge was genuinely helpful. It’s the form of tedious comparability work that I’d have procrastinated on ceaselessly if I needed to do it manually.
I additionally needed to indicate Claude Code’s skill to seize stuff from the net. We pulled in PDFs, checked for replication packages on openICPSR, that form of factor. The CGBS steady DiD paper is sitting in my docs/references/ folder now. The JHR replication bundle exists on openICPSR (although you want to log in to obtain it, which Claude can’t do for me).
I preserve coming again to Beamer decks not as presentation supplies however as considering instruments. Right this moment we added slides displaying the geographic divergence between the 2 datasets — the place precisely are the thesis and JHR measures disagreeing? We constructed a TikZ graphic attempting for instance how county fastened results use the exogenous change in distance for identification. Identical girl, totally different distance. That’s the variation we’re exploiting. Right here’s the concept I described to Claude Code, and that is the slide he fabricated from it — which was mainly exactly what I had in thoughts, however by no means in one million years might I’ve had the persistence to determine how you can do it.
The deck is now 30 pages. You’ll be able to obtain it right here I believe. If this doesn’t work, I could have to start out migrating it to a greater location as generally with dropbox it’s a must to ask for permissions, however hopefully this works. It’s mainly a file of my evolving understanding of this challenge. Future me will thank current me.
I’m additionally attempting to determine how you can hand off extra of the challenge administration to the AI agent with out dropping my thoughts. We’ve bought:
-
CLAUDE.md with the foundations (don’t delete knowledge, don’t delete code, use the legacy folder, and so forth.)
-
todo.md monitoring what must occur subsequent
-
log/ with timestamped entries of what we did every session
The thought is that if a session dies or I come again to this in three months, there’s a paper path. The deck, the logs, the todo listing — they’re all methods of speaking with future me (and with future Claude classes that don’t have any reminiscence of what occurred earlier than). So on this you see extra of me attempting to make some organizational choices. For a few of you, that is going to be such a pure a part of your workflow as you’re simply by nature a really organized particular person in comparison with me I’m positive. However you may no less than see me attempting to get this spun up in markdowns that implement it repeatedly.
Right here’s the factor I’m wrestling with now, and it’s the rationale I haven’t began any precise evaluation but.
About 42% of Texas lives in 5 counties: Harris (Houston), Dallas, Tarrant (Fort Price), Bexar (San Antonio), and Travis (Austin). These city counties mainly by no means see any variation in distance. There’s at all times a clinic close by.
So once I run a diff-in-diff, who’s the counterfactual for some rural Panhandle county that simply misplaced its nearest clinic? Is it Austin? Austin by no means experiences any therapy variation. Is that actually who I need imputing the counterfactual for Lamb County?
That is the core identification query I have to work by earlier than touching any CGBS code. The methodology is barely nearly as good because the comparability group, and I’m not satisfied I’ve thought laborious sufficient about who the legitimate comparability models are. And so I left this for me to think about within the todo.md that I’m holding as a operating to do listing.
So yeah. That is me utilizing Claude Code on an actual challenge. It’s messy. The movies are lengthy. I’m considering out loud. Typically I am going down rabbit holes that don’t pan out.
However that’s form of the purpose. This isn’t a sophisticated tutorial. It’s documentation of how I really work — how I exploit AI brokers to handle tasks, audit code, visualize concepts, and slowly construct up understanding of what’s in my knowledge and what I can credibly estimate.
In case you’re eager about steady diff-in-diff, stick round. In case you’re eager about how AI brokers can match into empirical workflows, stick round. In case you simply need to watch somebody argue with Claude about whether or not a TikZ polygon seems sufficient like Texas, properly, that’s in there too.
The video is on the high. It’s like I stated 1.5 hours. Skip round as wanted. I discuss an excessive amount of. However that’s the gist of it! Thanks once more for all of your assist. Please think about changing into a paying subscriber! And thanks everybody who already is a supporter each paying but in addition being a cheerleader and constructive particular person in my life. That too is way appreciated.
