Stale code, deprecation, and the canon

I feel I first seen I wanted a structured workflow with utilizing Claude Code when throughout a protracted venture the analytical pattern went from having N observations to N+ok observations. It had no rationalization as I had been constantly verifying outcomes, but it surely was one thing concerning the inter-temporal outcomes, not the contemporaneous outcomes, the place I simply seen one thing was amiss. Why had the analytical pattern all of a sudden modified? I solely caught that, too, as a result of my workflow was to religiously produce “stunning tables” and “stunning figures” which I saved in “stunning decks” — none of which was meant to catch bizarre issues like that.

The opposite time I seen it, although, was when in a unique venture, I used to be working lengthy hours with Claude Code going backwards and forwards doing evaluation. In my thoughts, every time I used to be requesting one thing new to be executed, I thought it was merely enhancing an present code. I might test that code and it will be right, however then in a while I took a step again and located not Ok applications, however extra like 5K applications. It was like I used to be taking a look at a really previous drawback that used to hang-out me after I was first beginning out the place I had copies of the identical code with bizarre naming conventions, solely this time Claude Code was doing it, placing much more distance between me and the work.

In a number of posts, I alluded to the truth that I’ve been constructing my very own customized dashboard that capabilities as a manifestation of a inflexible guidelines for panel knowledge. I have already got a inflexible guidelines I observe based mostly partly on Don Rubin’s article “Design Trumps Evaluation”, in addition to a diff-in-diff guidelines by Pedro Sant’Anna that I’ve talked about on right here earlier than, in addition to tweaked (in addition to included in my new ebook that comes out subsequent month). Plus issues by Guido Imbens that he at all times appears to incorporate in his survey articles (see right here for an instance, and flip extra in direction of the top). And so the backbone of the dashboard is the guidelines, and as soon as I’ve it executed, I’ll share it, with an outline and a stroll by means of.

I believe that there’s some form of excessive stage common conceptual framework of a harness-workflow-for-empirical-research-using-AI-agents, and that when we heard it articulated, it’ll be apparent that that’s the manner that work must be organized when utilizing AI brokers for empirical analysis. After which we’ll pivot our workflows in direction of it. However my hunch is that even when that easy mannequin is articulated, we’ll nonetheless be iterating on that to make our personal dashboards and workflows if solely as a result of comparative benefit in all probability will drive it in direction of that equilibrium.

This morning, although, I used to be fascinated with my very own. I’ve received a guidelines with a collection of inflexible steps. It’s clocking in at round 12 to 14 steps that I observe when engaged on a causal inference venture, and it roughly runs like this:

0. PKG: Write down all of the software program packages I might be utilizing for the venture

1. TARGET: What’s the causal parameter, expressed utilizing potential outcomes, populations and weights? And if there’s a couple of causal parameter, title them clearly.

2. BITE: Illustrate the therapy’s chunk. That is the place I merely doc information concerning the intervention. Often the objective is to create round 2-3 displays (tables and figures) per stage, too. So I’d present a time collection of meth use over time. I’d present policing variations utilizing a warmth map laid over a county form file. One thing, although, to actually assist me perceive the therapy working on the earth.

3. COVARIATES. That is the stage the place I choose covariates wanted to fulfill unconfoundedness, conditional parallel tendencies, or presumably the an element mannequin of the untreated potential consequence. I’ve a ability I created, additionally, name /covariates that I’ll clarify one other time which helps pull from my very own private beliefs candidate covariates based mostly on a specific modeling philosophy I imagine in. Once more for an additional day. However in covariates, I produce tables measuring imbalance, plot propensity scores, and so forth. Once more — 2-3 displays.

4. ROLLOUT. That is the place I doc the rollout of the therapy, be it two teams of greater than two teams. I make a desk counting the variety of models by cohort, named if attainable, in addition to pattern shares (variety of handled models divided by the entire variety of handled models). However I additionally then make a determine exhibiting the rollout utilizing Yiqing Xu’s “panelview” command.

5. OUTCOME. That is the place I first current displays (tables and figures) of the end result. If panel knowledge, then it’s the end result’s evolution over time. If not, then another manner. The objective is a presentation, although, that matches the evaluation that might be executed.

6. POWER. On this stage, I do energy calculations. What’s the minimal detectable impact for dimension 0.05 and energy 0.8? And am I powered for it in my very own pattern? I create energy curves for this usually, in addition to attempt to quantify over-rejection issues I discovered in my Monte Carlos in order that I can resolve if I’m going to even select a specific inference technique or a given estimator within the first place.

7. ESTIMATOR. After which there’s the number of the estimator. And that is solely flowing from the TARGET stage, the COVARIATES stage, and my interpretation of BITE. As a result of if I feel I see indicators of randomized therapy task in that BITE stage, I’ll lean in a single course, but when I don’t, I’ll lean extra in direction of one thing like parallel tendencies or an element mannequin of the untreated potential consequence. Whichever, although, that is the place the estimator is chosen, and when it’s estimated.

8. EVENT. Simple however I’ve for now separated out the manufacturing of occasion research graphics from the ESTIMATOR stage, although I’m leaning in direction of combining them.

9. FALSIFY. Now I begin working down attainable falsifications, like outcomes that shouldn’t be affected by the therapy versus teams that shouldn’t be affected by the therapy.

10. MECHANISM. Right here I start to work by means of mechanisms. Are there sure mechanisms that I feel I could possibly be finding out as attainable explanations for the issues I discovered within the ESTIMATOR and EVENT and FALSIFY levels?

11. PIPELINE. Then I rerun every little thing to make sure it replicates. I additionally run /referee2 right here in addition to /blindspot, although frankly these have additionally been being ran at every stage as effectively. I simply am compulsively double checking.

12. DELIVER. If I’m executed, then that is the stage the place I’m going to take what I’ve produced and starting storyboarding the findings. That is outlining, that is brief little papers, that is no matter course of I exploit however the objective of the stage is to start out producing deliverables based mostly on every little thing that got here earlier than it.

13. SIGN. After which as soon as I’m executed, I log out.

I needed to develop an insanely inflexible guidelines as a result of I stored experiencing that “drift” drawback I famous, so this guidelines is each design ideas, but it surely’s additionally a response to AI Agent drift.

So, inside every of these levels, what I’ve been doing is arising with a collection of rituals that should be executed every time to make sure that I’ve accomplished the stage. I exploit a wide range of issues to do that. For example, the dashboard has a visible going left to proper of circles which are shade coded inexperienced (closed), crimson (open) and amber (“you’re right here”). Every circle goes with a kind of 13 levels. And if it’s inexperienced, then it means I’ve gone by means of a collection of steps that has given me a stage of confidence that that work is accomplished and proper. But when it’s crimson, it means it’s not but closed, and as no venture can ever have a crimson circle, meaning the venture can also be not closed. After which amber, which pulses visually by means of numerous hues of yellow-amber, is the place I do know every day I’m after I begin the venture again up.

Effectively, sooner or later, I’m going to stroll you thru a venture utilizing diff-in-diff so as to see how I’m doing this, however I do know that that is simply going to be how I do it, as mature researchers all have their very own type, their very own preferences, their very own wants. However I do suppose “AI Agent drift” is an actual drawback, and what I need to go away with is a metaphor.

The early church fathers had their very own drift within the sense that there have been loads of letters and books floating round. These letters and books have been like previous information. That they had totally different names, and so they had comparable sounding names, and so they advised comparable tales, however not an identical tales. And sooner or later, the early church received collectively and selected the canon. What was the canon? The canon was the pipeline of code that may produce the displays (tables and figures) that may assist the ultimate manuscript that the creator offered as their very own.

That they had some means of choice and curation. That means of choice and curation did two issues. It selected what can be within the pipeline, and it selected what can be deprecated. They didn’t use the phrase “deprecated” although. They used extra harmful sounding phrases like “heresy”, or if that’s too harsh, then “apocryphal.” These have been numerous gradations as to what can be tolerated.

Effectively, right here’s the factor. AI Brokers are going to provide, when used for empirical analysis, an abundance of information, figures, tables and code. There’s something concerning the means of a prolonged empirical research whereby and not using a very carefully managed harness, you’ll find yourself with a multiplicity of these objects. They are going to have comparable sounding names like “The Gospel of Thomas” versus “The Gospel of Mark”. They are going to inform comparable tales — however not the very same story, and their variations could also be benign such that they’ll each be included (e.g., The Gospel of Mark versus the Gospel of Matthew). However generally their variations might be an excessive amount of (e.g., The Gospel of Mark versus The Gospel of Thomas) and a tough choice should be made.

You and I are going to want a standards and a course of and a rule that selects the canon and that course of is prone to be an unfamiliar one. It’s prone to be an unfamiliar course of as a result of previous to AI Brokers used for empirical analysis, I feel we have been unconsciously curating our work with out actually understanding it.

I imply, I take that again. Not on a regular basis. It’s a really quite common factor to teach younger folks into coding etiquette and warn of assorted pitfalls that evolve naturally when sure issues are ignored. So it’s quite common to see whole recommendation dedicated to what you’re going to call information, how you’ll arrange your directories and model management (see Gentzkow and Shapiro’s traditional guide on this as an example). We develop guidelines and greatest in practices in response to the applied sciences and the issues they create for us and the alternatives they create for us.

Effectively, quickly somebody goes to write down that for the usage of AI Brokers for empirical analysis, and when it comes, it’s going to bear some resemblance to these previous methods however I feel it’s going to very fairly weird at first to us. However one of many issues that it’ll most positively have because of the pure tendency of code drift from brokers is that it should have an intentional course of following a detailed set of steps that might be used to canonize code right into a replicable pipeline. Mine is a guidelines, utilizing metaphors like locks and colours and maps and practices and hooks, to make sure that not simply my future self, but additionally future Claudes can at all times decide up the place left off and know exactly what’s heresy, what’s apocryphal and what’s canon. That’s the objective. The objective is that the displays we find yourself with are displays we selected and that they imply what we expect they imply, and the pattern doesn’t simply drift and alter over the course of the venture unknowingly due to a forgetful Claude who improvises on a regular basis with out consciousness.