I need to share one thing I discovered not too long ago. I discussed it this week on my podcast, The Odd Couple, with Caitlin within the opening. And it’s this — I discovered final week throughout my MIT speak, from a grad pupil named Theo, that Claude Code is writing a diary of every part it did in your mission, together with the deserted work it did. And I believe these of us who’re beginning down the trail of learning the habits of AI brokers (which seems to be a subject I’m going to be pursuing for slightly bit anyway), that is precious for understanding mechanisms. Let me clarify.
There’s a file related together with your work that Claude Code is conserving that could be a JSONL, and it’s Claude’s diary. It’s its notes out of your work with it. It’s a central a part of the native folders the place Claude Code is recording its reasoning. This isn’t what it seems like. As a result of Claude Code is a reasoning mannequin, not simply an agent mannequin, its reasoning is being recorded. Why does this matter? Nicely, if we try to know the habits of agent-based manufacturing of analysis, that’s the place we’re going to study it. And for me, that’s one of many matters I believe I’m sorting into — what can we study their habits? How do they make choices? And are they influenced by the researcher?
Ordinarily we are able to solely estimate causal results beneath quasi or managed randomized eventualities. So we contain independence, conditional independence, parallel developments, and so forth. After which upon focusing on the corresponding inhabitants causal estimand, we construct the estimators that focus on these identification assumptions and pow — we get it. Below the regulation of enormous numbers, the estimator’s sampling distribution will middle precisely on high of that inhabitants causal impact, and beneath central restrict theorem, we get a traditional distribution that we are able to use for speculation exams. Win-win.
However that’s not itself one thing we are able to simply use for figuring out mechanisms. Fairly often all we have now are a bunch of variables and should roughly use the identical analysis design, cross our fingers that the assumptions that held for the unique goal parameter will proceed to carry for the “mechanism variables”, however after all that’s not assured. Parallel developments may maintain for mortality however not illnesses, as an illustration, regardless that you might suspect illnesses are the drivers of the mortality outcomes you discovered. And but if you happen to change out the analysis design with a purpose to get the illness channel, perhaps you aren’t longer on the earth of the diff-in-diff with its focused ATTs. Possibly to get the mechanism, it’s important to change over to an instrument, and consequently, you find yourself with one thing extra like a LATE — which isn’t solely a special parameter, however it’s a special inhabitants, and I don’t simply imply it’s the typical complier impact. I imply that the LATE will not be the LATT will not be the ATT.
And so you find yourself with having to invoke loads of homogenous therapy impact assumptions to determine the mechanism, which is identification by magic in some situations as a result of as I mentioned, there’s no implicit assure that an identification assumption for one consequence, Y, should subsequently maintain for one more consequence, name it Y2.
I believe that is really not a broadly famous drawback with trendy causal inference and the belief of unbounded heterogenous therapy results. Shopping for into unbounded heterogenous therapy results is definitely an enormous headache. It broke OLS twoway mounted results (RIP), it broke instrumental variables (additionally RIP), and it’ll preserve breaking issues. In reality, heterogeneous therapy results is so harsh that it even appears to step on the toes of creating Popperian falsification rules for testing scientific theories unattainable. Why? As a result of if therapy results actually may be something, and you’ve got a concept that claims some comparative static goes in some route, then you find yourself with out the flexibility to say that empirically. The speculation can say that, however as soon as you find yourself in a world of unbounded heterogenous therapy results, you let concept go. Possibly the general impact is meant to be detrimental, however does that subsequently imply the typical complier impact goes to be detrimental? Unbounded heterogenous therapy results could undermine Popperian rules of falsification making it laborious to really work out by deduction why you discovered what you discovered.
What does this should do with AI Brokers? Nicely — we even have the mechanism for his or her habits. How so? As a result of in that JSON, they are going to actually write it down. And because it’s all in textual content, the one factor you actually need to do is figure with pure language processing methods to attract it out.
In my minimal wage examine that I discussed the opposite day, I’ve these numerous brokers that I randomly assigned to the identical activity with barely completely different primes concerning the anticipated impact of the minimal wage. As soon as I get the R&R resubmitted, I’ll put up about it, however let me let you know this for the sake of everybody else engaged on these questions — I’m discovering proof of specification looking out in response to the assorted minimal wage primes. I can actually see in these diaries Claude Code operating a mannequin, wanting on the consequence, after which brazenly stating it’s going to strive a special mannequin as a result of the estimated impact will not be what it anticipated.
Which suggests just a few issues. First, Claude Code will not be going to robotically bind itself to design rules. It received’t take the place in different phrases that the result’s what the result’s. This, once more, is proof that we’re going to should develop into adept at verification. Even after I considerably handicapped the brokers (I compelled them to learn our JEL on difference-in-differences rife with warning about twoway mounted results, as an illustration), they are going to change issues round, all beneath the hood, leaving me with outcomes that I’m not personally concerned in.
Secondly, Claude Code doesn’t seem like focusing on a selected inhabitants estimand. Which signifies that the human is probably going going to should scaffold that, and that requires human capital, and proofing the work does too, and determining if it’s specification looking out goes to must be one thing that the researcher is on high of. I’m skeptical that the researcher can submit their JSONs within the mission, as a result of frankly, that’s pretty straightforward to clip. However I believe it will require at minimal loads of consciousness collectively that the human can simply lay again and led the agent drive. Even when the agent has the comparative benefit in manufacturing doesn’t subsequently imply it has the comparative benefit in verification too. In reality, not often does any actor have the comparative benefit throughout all the resolution nodes. And if there are solely two resolution nodes, they received’t have the comparative benefit in each even when they’ve absolutely the benefit in each.
After all, a few of this does get slightly tenuous with brokers as a result of we aren’t coping with only one actor, however almost a doubtlessly enormous variety of swarming brokers.
Anyway, we’re at the start of all this. And I simply needed to place on the market, once more, that you’ve got extra information than you might know, and that that is in all probability a type of “textual content as information” moments. I encourage you to start out tearing that JSON open, subsequently, and incorporate it into your analysis — for certain if you happen to’re learning the habits of brokers, however in all probability even if you happen to aren’t.
Additionally I’ve now written 50 essays about Claude Code. My first one was December thirteenth, 2025. And immediately is Might thirteenth (or no less than, it’s in Zurich). Six months. What a journey six months has been for me. Going from first noticing, virtually randomly, in mid November the toggle on the desktop app with Claude Code, to now being real and honest that Claude Code has modified my life. I’m all in. I’m pushing all my chips onto this expertise. No turning again for me. All the issues that Claude Code creates for my very own workflow are merely issues to unravel. I’m undeterred. I discover it such a wierd sensation to get up feeling precise gratitude for software program. However I do.
