That is a part of my ongoing Claude Code sequence, that are substack posts discussing what I’m studying about Claude Code because it pertains to quantitative social scientists whose work lives inside folders and directories on their native machines. My declare continues to be that in the meanwhile, there’s a surplus of writings about Claude Code by engineers for engineers, and a paucity of writings about Claude Code by social scientists for social scientists. So I’m simply documenting what I’m noticing, typically doing video walkthroughs, typically writing essays, and this one is extra essays about coping with the necessity to discover verification programs now that productiveness is legitimately enhanced with Claude Code. All Claude Code posts stay free once they come out, not like different posts are randomly paywalled. Every little thing goes behind a paywall after a couple of days, although. For those who discover this sequence invaluable, I encourage you to help it at $5/month or $50/12 months!
As with so lots of Claude Code posts, they’re pretty stream of consciousness. And this one isn’t any completely different. The fabric on this substack is kind of an concept I’m understanding which is printed on this deck which is that even with sustaining the identical quantity of human time on analysis, I believe there are such a lot of new issues with utilizing Claude Code for analysis that we might very nicely be in a really laborious place the place we have now to spend way more time on non-research associated actions making an attempt to resolve these new issues that we aren’t used to encountering.
On this deck, I’ve been understanding the issues I’m creating for myself with a lot new, larger high quality analysis output, the place I’m inadvertently creating too many actions. And within the technique of seeing productiveness features, however all the time with diminishing marginal returns, these new prices are exploding round me in methods I’m not anticipating. I name them all through “inventory pollution”, virtually like litter, and I’m making an attempt to determine which ones are simply tolerable, and which of them are completely not tolerable.
Hyper Systematic Group Interacting with Unusual Consideration
So again to the issue. Over the previous few weeks, Claude Code had helped me generate an enormous quantity of fabric for this undertaking that I had revived after months of sitting on it and procrastinating on the revision, making an attempt to faux I didn’t keep in mind the deadline was approaching. So I had used Claude Code to interrupt down particularly what the referees and editor wished achieved. The revised analyses, the robustness checks, new specs, figures, and so on. I used Claude Code to interrupt down exactly what they had been asking me to do, set up these into aichecklist, like a map of duties, in order that I’d not inadvertently skip something. After which we had one after the other began doing them.
And I might really feel my productiveness exploding as a result of it was — there have been some issues, many issues, that I used to be finishing immediately, and there have been issues that I additionally felt like I used to be getting achieved that was more durable to clarify. However it needed to do with how tousled I get due to my ADHD “primarily inattentive” stuff. How I can’t fairly keep in mind the place I’m in a course of, or how I get fascinated with essentially the most trivial particulars, going deeper and deeper, till I mainly nowhere close to the place I began. And to me this isn’t simply how I’m, but when I’m sincere, it’s how I wish to be too. I really like that hyper fixation, that stream state, when it occurs. I’m profoundly curious, to a fault, and in ways in which I believe annoy coauthors, however so usually they lead me to the enhancements. It’s simply that in addition they spew off lots of air pollution too, and so in my analysis initiatives, my coauthors usually must tolerate lots of it within the hopes that on common we’re getting someplace that can enhance issues. And my finest coauthor experiences don’t thoughts it, see the purpose of it, and are prepared to trip it with me.
So then in that sense the analysis with Claude Code has that very same function. It’s simply that my productiveness is ramped up by 5x, and since these issues are nonetheless there, it implies that the externalities are additionally generated at 5x. And I’m not likely positive that they’re the truth is linear within the work. I believe typically that the externalities could even nonlinear within the productiveness, and since my velocity of labor is now quicker, and in a brand new setting with out the guardrails I had spent years perfecting after graduate college to maintain me targeted and on observe with minimal errors and maximized output, the prices related to the progress would possibly very nicely be convex, rising quicker than the features.
So let me then share a bit of of what I’m pondering. I believe that my fashion of interacting with the analysis by way of my “rhetoric of decks” philosophy, the place I hold fixed notes in a journal of an evolving scroll of “lovely decks”, largely including to them, may very well be creating some challenges. I can’t fairly put my finger on it, and haven’t but, however I believe the decks are obligatory for me, and but they’re additionally the supply of inventory pollution rising quick within the analysis course of, making discovering what I would like like discovering a needle in a haystack as I can’t appear to on the finish, when it’s time to complete this up, keep in mind the place issues are.
A few of it’s because as organized as Claude Code is, each single concept I give, he generates the code and shops it, however I’ve observed he could not all the time put it within the place I would like it. And never solely that, he’ll usually generate new code, quite than add to the present file I would like, which I believe could create these small random perturbations within the pipeline the place issues are branching off. This occurs most of all in outdated initiatives being revived I’ve observed because the outdated initiatives have legacy kinds of group that aren’t essentially what I’m doing now once I begin. As a result of once I begin now, I are inclined to have a a lot less complicated place to begin that appears like this.
That’s generated above utilizing my /newproject talent. It generates that listing construction for all new initiatives. However for outdated initiatives, like I stated, I can’t and don’t try this out of concern that I’ll overwrite issues, which is an actual fear I’ve with Claude Code — the inadvertent deleting of knowledge is one thing I explicitly inform Claude Code to not do in my static Claude.md markdown.
Which suggests although that my present R&Rs the place I’m brining Claude Code for AI help are messier than meant. Once I revive outdated initiatives, and attempt to carry it into the self-discipline of Claude Code, it seems a lot crazier as a result of it has this Frankenstein fashion hodge podge of the outdated and the brand new, and I discover I’m not prepared to only flip to the brand new fashion and as a substitute are inclined to grandfather within the outdated, which this undertaking was because it was an R&R, and which subsequently could or could not have contributed to the difficult-to-put-my-finger-on battle I used to be having preserving observe of simply what was occurring
Isoquants, Consideration, Misplaced Consideration
Recall that I gave this discuss to the Boston Fed again in mid-December, which now looks like I used to be making an attempt to carry lately found fireplace to them in mild of the fast explosion in Claude Code consciousness via the social sciences, however which on the time I used to be sort of apprehensive I used to be going to sound like a manic and considerably over-reacting seminar speaker stuffed with prophetic hopes and doomsday predictions. Nicely, each might be true. Anyway, right here was my primary framework when you didn’t learn earlier posts about this (which frankly are posts I used to write down way back to 2023).
Recall that my core conviction is that the isoquants from manufacturing capabilities for doing “inventive cognitive work” have flatted from being quasi-concave pre-AI — whereby it was unattainable to do any inventive cognitive work with out utilizing nontrivial quantities of human time — to having flattened, and for a lot of duties, truly linear which as economists know implies that if I’m proper, then machine time and human time are good substitutes, even for cognitively inventive duties.
Nicely if they’re the truth is good substitutes, then rational actors will use the cheaper of the 2 on the margin. We pay month-to-month costs for Claude Code at anyplace between $20 to $200. We don’t pay for tokens on a case foundation, however we do incur alternative price of human time on a case foundation (proxied by the worth we place on our subsequent finest various). And in order such there’s temptation when utilizing AI for analysis, virtually like we’re sporting heavy weights round our legs, for AI to tug us in direction of utilizing much less time on analysis. I don’t imply, although, doing much less analysis notice. I imply much less time. Much less human time on analysis, and if human time is a direct enter in consideration, as you can not take note of issues that you’re not actually focusing your time on, we are able to find yourself studying much less and doing extra on the identical time.
That is actually on the core of lots of the issues, other than ethics (although this certainly will get into ethics too as to what diploma are you the professional on issues you’re driving?), of utilizing AI for social scientific analysis. Lowered time resulting in lowered consideration, resulting in much less human capital, regardless of the completion of precise cognitively inventive duties is the place human researchers turn into roughly elective within the technique of doing analysis.
So I define three prospects, solely two of that are good for human researchers if our aim is to keep up a reference to the data we’re answerable for creating. And one among them — the primary one — is the one I personally maintain to which is that I preserve my time use dedicated to the analysis in order that I preserve my curiosity and studying, as a result of my curiosity is my energy, and if I stay a life the place I drift away from my love of studying, discovery and engagement with my curiosity, I would as nicely go discover a job some other place. I merely refuse to stay an inferior life the place I’m not engaged within the actions I really like, which is a connection to studying in all of the ways in which makes my coronary heart sing. That is partly what differentiates me from being purely somebody who cares about coverage for its personal sake — I’m a hedonist. I care about my passions and curiosity for its personal sake, and every part else will get swept together with it. I simply attempt to intention that short-term wants with long-term targets in order that I’m helped in ways in which contact on different values I’ve, like serving to folks.
And so this image is kind of what I see as my very own aspirational aim. Keep time use on analysis subjects utilizing AI in order that the productiveness features occur. That is represented to me as the perfect final result as a result of the output features are the most important on a per unit foundation. It’s the identical time, H*, it simply will inevitably be completely different time.
However the reality of the matter is that there’s a pull, like a gravity power, that pulls the researcher down and away from H*. And one among them is arguably welfare enhancing from the angle of elevated data for oneself, and the opposite shouldn’t be. The one on the left represents gained data with lowered time use, and the on the correct represents excessive automation the place time use fell an excessive amount of such that the human grew to become actually nothing greater than what I simply typically name the “button pusher”, the place analysis turns into manufacturing unit work.
And so what I used to be experiencing within the R&R was that particular manifestation of how during which utilizing Claude Code to help me within the analysis course of was mixing me concurrently amongst all three of those states. It was creating some sort of inner coordination drawback that I couldn’t fairly put my finger on, however I wished to now simply describe what I believe is going on.
The Drawback of Too A lot Higher Work
So a part of the issues I believe I’m having is that as I’m going so quick, growing my work by 5-10x, and utilizing “lovely decks” to keep up my connection to the progress, like a operating diary, I’m someway creating too many decks, with out of order progress. This occurs particularly for the really advanced initiatives, too. The place there could also be 5 methods of doing one thing and the place ex ante there isn’t a clear cause to favor one over the opposite, and so I do all 5 after which must determine how they are often reconciled, if they need to be reconciled, and how you can go about positioning these reveals. Do they go within the manuscript? In that case the place? In that case how will they be displayed? 5 tables? 5 figures? 5 panels? One panel? So I could attempt all choices for aesthetic functions, however I could too iterate sequentially as I do it, realizing that the correct strategy to do it’s to do XYZ, not realizing that that perception got here to me after some earlier step of ABC.
The issue in utilizing decks this strategy to preserve my reference to the work is small, delicate particulars. For one, Claude Code could virtually randomly laborious code the code output into the decks except I say in any other case. And if I’m not utilizing /talent instructions for repeated work, and if these /talent instructions haven’t been completely perfected to keep away from laborious coding into decks — one thing so particular it might be missed — chances are you’ll not notice that randomly all through the deck are non-replicable work.
See, if the work is tough coded into the deck, regardless that the output ./tex exists, then chances are you’ll very nicely have TWO copies of the identical factor — you could have the outdated copy that’s utilizing at-that-time output, and you could have a brand new copy in .tex generated from estout or outreg2.
So this has been a problem for me to resolve. How do I preserve a brand new diary of progress, sustaining my consideration, however now coping with the inventory pollution, let’s name it, of stuff surrounding me? If the truth is the manufacturing of extra waste is convex in time use, then maybe I’ve two issues taking place without delay — I’ve elevated productiveness, however diminishing marginal returns because the one legislation of economics, even moreso than demand sloping downward itself (however which is the truth is answerable for demand sloping downward when it does), is the legislation of diminishing marginal returns to human time. And I’ve convex price capabilities such that every further use of time will increase at an exponential price rising marginal prices alongside some dimension that I could circuitously perceive, however which via repeated interactions on this new setting I completely hold encountering.
Sustaining Consideration, Lowered Congestion and Human Verification Is The brand new Ability
I noticed Andrew Karpathy say lately, I’ll must dig up the quote, that the brand new talent is in human verification. It’s not in ‘vibe coding’ on this age of Claude Code as there’s mainly no obstacles to entry to telling Claude Code “do that and that”. There isn’t any talent in any respect in dictating “do that difficult partial identification factor I’ve all the time wished achieved”. That takes no talent, and since Claude Code is mainly a genius, compliant, and cussed like an obedient canine to do something and every part you ask of it, it’s going to do it.
The actual talent going ahead shouldn’t be subsequently within the doing. We’ll all be sitting with jet packs on our again, and that after we determine to elevate off, we’ll elevate off — simply not slowly. If we aren’t cautious, we’ll rip via our environment at mild velocity, and whereas it’s true we’ll get someplace quicker, we’ll shatter home windows and homes in our means too.
I’m targeted now on simply the extreme litter I’m creating in my decks with my new workflow that I can’t fairly get a bead on. And I’ve latched onto my “rhetoric of decks” as a result of I’m utilizing it lots to assist me hold observe of labor over time. However I’m subsequently coping with idiosyncratic issues too from that being an imperfect answer.
So to Kaparthy’s level, the brand new talent shouldn’t be within the doing. Quite it’s in a single space he identifies, and two extra that I’m targeted on.
-
Human verification. We’re answerable for every part. We should subsequently discover a strategy to insert 100% correct verification programs into the analysis course of. There might be no errors. And admittedly, given the issue of figuring out errors, I believe in an virtually Beckerian like means, the stigma and punishments aimed toward even the smallest AI-related errors going ahead are in all probability going to be draconian. Similar to in a footnote in “Crime and Punishment”, Becker’s basic 1968 JPE on the economics of crime, the place he notes that Vietnamese rice speculators had their arms minimize off for his or her crimes as a result of low possibilities of detection, I believe we’ll see much more of that going ahead. Science is many issues, however scientific communities have a tendency to manage their very own via sanctions and rewards.
-
Excessive Stage of Consideration. So we should be vigilant and even obsessive about zero error philosophies now greater than ever. And it’s actually unclear on these time use curves I drew simply the place that’s, and the place we have to be concerned and the place we don’t have to be concerned, and the way we are able to automate even the verification, and which elements can’t be automated in any respect. All I do know is that the ultimate product should be one thing all of us perceive simply as a lot as we ever did, which finest I can inform requires a excessive degree of consideration. I believe this positively means for many of us preserving human time use on the analysis undertaking as excessive as humanly potential and resist, and even refuse, automation of analysis. Not a lot as a result of we’re in precept dedicated to human work, however as a result of I don’t assume we’re even near a world the place robots have the comparative benefit in automating scientific discoveries. I doubt the isoquants are straight strains — but.
-
Congestion. However sustaining the identical degree of time use with out addressing these convex prices coming from the inventory pollution related to the identical sort of time use is I believe going to be its personal drawback to be solved. It’s associated, clearly, to the opposite two, however I believe it’s nonetheless useful to separate it out.
Which brings me again to all my “lovely decks”. I’m not saying that the fault is in my deck philosophy — of utilizing decks to maintain me connected. A few of what I outlined, in spite of everything, is completely fixable via new workflows the place I all the time use exported .tex recordsdata it doesn’t matter what.
However I nonetheless assume I see the issue a bit extra clearly from these overflowing decks as a result of sooner or later, for any typical analysis undertaking, I will find yourself with too many slides, and regardless of how “lovely” these slides are, I’ll find yourself with congestion, and I’ll have a tough time pin pointing precisely the place that congestion is happening.
So I’ll finish there. Among the ongoing video stroll throughs I believe might be much less about me doing as it’s about me coping with the issues of my doing. I’ll clearly be doing issues. I’ve a cool new video sequence that I wish to announce however am ready a bit longer to take action. However I believe what you will note is me stumbling round, in actual time, making an attempt to doc the character of those rising marginal prices, after which making stabs at making an attempt to shift them down.
However that’s it for at the moment. Have an excellent day! Let’s hope we are able to hold going with none accidents!
