All Courses - Analytics Campus

U.Ok.’s lethal meningitis outbreak exhibits significance of vaccination

Science

March 22, 2026

U.Ok.’s lethal meningitis outbreak exhibits significance of vaccination

Well being officers within the U.Ok. are speeding to include a lethal outbreak of a type of bacterial meningitis known as by meningococcal meningitis launching an enormous antibiotic and vaccination marketing campaign for the 1000’s of individuals affected, a lot of whom are college students.

The contagious an infection was first detected amongst younger folks on the College of Kent in England. The illness has since shortly unfold to different college students and folks within the area: as of March 19, 1000’s of individuals had been contacted by well being authorities due to their potential publicity to the illness, a minimum of 29 folks had confirmed or suspected infections, and two had died, in keeping with the U.Ok. Well being Safety Company.

Treating uncovered folks with antibiotics earlier than signs present up may help shortly squash bacterial meningitis outbreaks. However specialists say that routine meningitis vaccination for high-risk teams, together with younger adults who congregate in settings akin to school campuses, is vital to stopping such harmful transmission within the first place.

On supporting science journalism

In case you’re having fun with this text, take into account supporting our award-winning journalism by subscribing. By buying a subscription you might be serving to to make sure the way forward for impactful tales in regards to the discoveries and concepts shaping our world at this time.

“This outbreak is a really uncommon occasion,” says William Schaffner, an infectious illness doctor at Vanderbilt College Medical Heart. For context, the U.Ok. had 378 confirmed instances of the an infection that causes meningococcal meningitis from July 2024 by June 2025; the U.S. reported 503 confirmed and possible instances in 2024. “It’s very spectacular that our complete vaccination packages in the USA and within the U.Ok. have decreased the prevalence of this very scary illness very considerably in our populations,” he says.

But the vaccines designed to curb this type of bacterial meningitis have not too long ago been forged into doubt within the U.S. by the Trump administration. In January the Facilities for Illness Management and Prevention rolled again a advice that every one kids get a meningococcal meningitis vaccine course as a part of their routine childhood vaccines. A latest U.S. district courtroom ruling has quickly blocked that call, nevertheless. The Division of Well being and Human Companies, which oversees the CDC, didn’t instantly reply to Scientific American’squestions relating to the present standing of vaccine suggestions for meningitis.

Scientific American spoke with infectious illness specialists about what the risks of bacterial meningitis are and the way vaccination may help shield in opposition to the illness and outbreaks.

What’s bacterial meningitis?

Meningitis is irritation of membranes known as meninges that encompass the mind and spinal twine. It may be attributable to micro organism, fungi, viruses and autoimmune illnesses, explains Nicholas Van Sickels, a medical director for the an infection prevention and management program at College of Kentucky HealthCare.

“In an outbreak setting, although, what we’re usually speaking about is bacterial meningitis,” he says. “You usually hear about it in conditions like what’s happening in England, the place you may have a university campus and sometimes very wholesome people current as very, very sick in a brief time period—and a few die.”

Bacterial meningitis is usually attributable to meningococcal illness, an an infection with the bacterium Neisseria meningitidis, which will be transmitted by aerosolized droplets and shut private contact. It’s extremely contagious in enclosed settings akin to school dorms, navy bunks or detention facilities. Preliminary signs can embody fever, headache, speedy respiratory and chills. Contaminated folks can also develop a pinprick purple or purple rash that doesn’t disappear beneath stress. The sickness can progress very quickly, spreading into cerebral spinal fluid and into the mind. “The affected person can get drowsy and even delirious after which lapse right into a coma,” Schaffner says.

The bacterium may also infiltrate the bloodstream, inflaming blood vessels, which can depart lasting injury even in individuals who recuperate from the preliminary an infection.

“Even when the an infection is properly handled, there’s a lot irritation that they’ll have long-term disabilities as a consequence,” Schaffner says. “Some folks have listening to difficulties. Generally you get gangrene, and amputations turn into vital.”

How is bacterial meningitis handled or prevented?

Antibiotics can deal with bacterial meningitis, they usually’re handiest when taken very early in an an infection, Schaffner says. At the moment, U.Ok. well being officers are giving out 1000’s of doses of antibiotics—a first-line intervention measure—to college students who’ve been uncovered or who’re at excessive danger of the illness. “That ought to convey this outbreak to a detailed,” Schaffner says. However vaccines are key to decreasing the danger of those outbreaks in future, he provides.

Two several types of meningococcal vaccines can be found in each the U.Ok. and the U.S.: MenACWY and MenB. The letters stand for varied strains of N. meningitidis every vaccine covers. Strains A, C, W and Y are usually extra frequent than pressure B.

Within the U.Ok. MenACWY vaccines are really useful to youngsters round age 14. Well being officers additionally advocate a primary dose of the MenB vaccine for infants at eight weeks of age, a second dose at 12 weeks and a booster at age one. The outbreak in Kent was pushed by the rarer B pressure. That’s why, along with distributing antibiotics, U.Ok. well being officers have pushed a focused MenB vaccine marketing campaign for college kids within the Kent space to assist forestall future unfold.

Within the U.S., instances of meningococcal illness have been trending upward since 2021, with a specific improve in pressure Y instances, in keeping with the CDC. At the moment, the company recommends that immunocompromised kids aged 16 and older obtain a shot of the MenB vaccine and that different youngsters on this age group solely obtain it on a shared-clinical choice foundation.

Given the rare presence of B strains, the shot is non-compulsory however obtainable to those that determine with a clinician that their baby ought to get it, Schaffner says. “Increasingly dad and mom are having their kids vaccinated in opposition to meningitis B, notably earlier than they go to varsity, as a result of they don’t need to be in a circumstance the place there may be straightforward transmission of this bug.”

The CDC beforehand really useful that every one youngsters obtain two doses of the MenACWY—the primary dose between age 11 and 12 and a booster at age 16. However in January the company moved to alter its advice to make the MenACWY vaccine non-compulsory beneath shared clinical-decision making for most youngsters. The way forward for that change is unclear as a authorized problem in opposition to it continues.

Each the MenB and MenACWY vaccines are very secure and efficient at stopping illness, Van Sickels says, with the highest ranges of safety occurring inside about 5 years after MenACWY vaccination and about one to 2 years after MenB vaccination. For this reason the timing of preliminary vaccination and subsequent boosters is necessary in reaching “the candy spot,” so immunity is best when kids are at greater danger, Van Sickels says. Some U.S. schools require college students to obtain each vaccines if they are going to be dwelling in dorms. MenACWY meningococcal vaccine necessities exist for the U.S. navy.

“Our nationwide vaccination marketing campaign with the [four-strain MenACWY vaccine] has been an excellent success,” Schaffner says. “It’s one other instance of how routine, complete vaccination has had a profound impact on actually dramatically decreasing the prevalence of what was as soon as an excessive, ordinarily feared an infection.”

Utilizing “Dispatch” on my cellphone with Claude Code to revisit the hashish paper, plus different stuff about CS

Econometrics

Dr. Mike

March 22, 2026

Utilizing “Dispatch” on my cellphone with Claude Code to revisit the hashish paper, plus different stuff about CS

Properly it is a large submit. It began with me simply wanting for example the brand new replace to Claude Code — “Dispatch”. You’ll need to watch me use it within the video. However principally it is a fairly huge replace. Why? As a result of with “Dispatch” you’ll be able to solely run your Claude Code stuff from your cellphone. So it’s essential:

After which comply with the directions by clicking on the Dispatch tab from inside your cellphone. I present you the way it goes down within the video. However actually, it is a fairly large deal. Why? Properly, for vibe coding fashion work within the first place, all you’re doing is chatting anyway. And you realize the place you chat rather a lot anyway? Your textual content message communication which is 80% to 100% of the time out of your cellphone anyway.

So, “Dispatch” goes to completely change your workflow. You’re going to be simply sitting on the sofa, texting every little thing. As long as your laptop computer or desktop is open, I believe it runs. Now we positively need to get a Mac mini or Mac Studio. Set that factor up, get it hooked as much as wifi, let it do all of your work — when you’re operating on a treadmill, from church, throughout your seminars, sitting on the sofa ingesting your morning espresso like I did. No matter. I’m undecided it’s higher per se, but it surely’s 100% going to be the primary manner we work. I’ve little question about that.

So it makes me suppose that it’s solely doable that Claude Code “Dispatch” might have a nontrivial impact on what machine we get. Perhaps I solely want an enormous machine for house, headless like a Mini or Studio, after which I simply have a standard laptop computer the remainder of the time. I imply you may dispatch from the desktop app too. So why not have the blazing quick factor however not essentially two blazing quick issues should you’re on the price range constraint?

However let me stroll you thru it. I’m going to point out you a few movies of me loading up dispatch and revisiting the “milled hashish paper” from the opposite day. That was the one the place I principally simply instructed Claude Code to select a subject for me, do all of the evaluation, and write it up, and it wrote a manuscript in 3.5 hours.

Thanks once more everybody for all of your help on this substack! I actually recognize all of the help, as I mentioned yesterday, each for these Claude Code posts, but additionally for your entire historical past of the substack. All of the emails, all of the feedback — it’s very cool. I like doing this, so I recognize it. It’s a true labor of affection. For those who actually get worth from it such that the marginal advantage of the substack every month exceeds $5, then think about turning into a paying subscriber! It’s solely $5/month! 🙂

Establishing “Dispatch” in your cellphone and desktop app.

First, let me present you the place you need to look. Dispatch just isn’t within the “Code” a part of the desktop app. Reasonably it’s within “Cowork”. And spot, it’s essential pull down “Claude” menu prime left (on the Mac desktop app anyway), and replace it. I’m already having to replace once more apparently.

When you do this, you then must go to the App Retailer, and replace your Claude cellphone app. Then return to the desktop app, to “Dispatch” and I believe you’ll must do some steps that I’ve already forgotten. However it’s possible you’ll both must scan a QR code, or you might have to simply reinitiate your Claude connection on the web site. However the level is, there’s a bit of setup, and when you do the setup, you’re able to go. So do this first.

Now let me present you the video stroll throughs. They’re, as soon as once more, round >1 hours however I believe it’s possible you’ll discover it attention-grabbing. First, you may watch me navigate the Dispatch stuff. However secondly, it’s within the context of “doing sensible empirical analysis”, which as you realize is form of the best way I write all these Claude Code posts. It’s much less abstractly about “right here is that this Claude Code idea known as /hooks!” However fairly it’s extra like “right here I’m utilizing a /hook for this mission I’m engaged on” sort of deal. Which I believe for the individuals who subscribe actually is the best way to do it. I do know it’s how I need to do it anyway.

However the second factor is me actually operating all of the coding from the cellphone. Working R, merging, even having Claude Code freaking internet crawl to get extra knowledge! Insane stuff. I’m actually not even certain if I even will primarily use the cellphone or the pc. I’m such a compulsive texter within the first place that for all I do know, I received’t even use my laptop computer once more. However right here, you watch and see what you suppose.

Video 1

Video 2

When Your Estimator Returns a Quantity That Isn’t an Estimate

That is frankly an enormous submit. It’s about Dispatch, it’s about logit, it’s about difference-in-differences, it’s about Callaway and Sant’Anna, it’s about conditional parallel tendencies. And whereas I want I might’ve made this shorter, I’m not going to. So frankly, possibly simply copy and paste it into Claude and interrogate the substack submit that manner should you don’t need to learn it. I’ll attempt to be fast although.

So earlier this week, I dug into that hashish reform paper that Claude “milled” up for me the opposite day. I couldn’t actually fake I didn’t see these occasion research (you may see it within the above substack if you wish to se it because it’s the picture on the quilt of the substack). The pre-trends appeared flat and the post-treatment estimates had been optimistic. It was round a 2% improve in wages as a consequence of leisure hashish reform and I simply couldn’t cease seeing it in my head.

So what I did was learn the paper. I learn it just a few occasions. And recall — I didn’t write this. Claude got here up with the concept (I didn’t even give the hashish thought to him), collected the information utilizing his “internet crawl” bash command, selected the estimator (Callaway and Sant’Anna for staggered therapy difference-in-differences), after which chosen did all of the evaluation together with the occasion research. After which he wrote it up “within the voice of Martin Weitzman”, who as many very long time readers know is my favourite author within the historical past of economics. That’s for an additional day although. Level is, he did it and that is the paper. The entire thing took 3.5 hours, and by now, most individuals studying this have already seen any such “paper mill on the supply” experiment just a few occasions, or accomplished it themselves, and due to this fact perceive why I wrote this submit saying I predict a smashing of milled papers colliding with our subject journals, editor desk, and loading up requests for referees by multiples of possibly 5-10x fold.

Anyway, I learn the paper and I seen that the CS estimator was utilizing “log state inhabitants” as a covariate. So I wished to consider that extra rigorously. So what I did was I requested Claude Code to gather “a ton of covariates” from the net that in his opinion had been applicable for a conditional parallel tendencies assumption. What’s that? Properly to me, the conditional parallel tendencies assumption is framed finest within the Heckman, Ichimura and Todd (1997, Restud) arrange the place you kind of regress the primary totally different (development) final result onto a bunch of baseline covariates for the management group solely, then use these fitted values to impute the primary totally different predicted final result for the therapy group. Which implies that I kind of consider the variables which are extremely predictive of the primary distinction final result as candidates to fulfill parallel tendencies since that’s what HIT97 does of their two step imputation process.

Anyway, Claude Code got here up with a candidate checklist of covariates and internet crawled till he discovered them, then pulled them in. After which what he did was put them in one by one, which I’ll present you now.

You’ll be able to see that none of those easy ATTs are statistically important although — the usual error is twice as massive because the ATT once I embody inhabitants, however had been even bigger with out something. So meaning the occasion research, in the event that they had been accomplished appropriately, had been most likely issues I actually need to dig into, and can.

One of many issues he present in doing this was that the state unemployment price he pulled by way of the BLS API.

BTW, can we simply paused for a second. We now apparently dwell in a world the place “synthetic intelligence” doesn’t simply exist, however that Claude Code will autonomously pull knowledge into my native folder on my precise laptop utilizing the API, then simply begin merging information and operating evaluation? I nonetheless can’t. I’m undecided when I’ll cease being amazed by that one reality.

Okay however again to the work. We discovered collectively that the unemployment price had tons of NAs as a result of it was month-to-month knowledge not annual knowledge. I had him determine that out by calculating NxT (N=50 states; T was 2003 to 2024) after which counting all covariates and outcomes that weren’t that quantity. By the best way, strongly encourage you add this into your workflow as an early step. Why? As a result of a lot is now occurring behind the scenes, and we’re very doubtless to be working with imbalance panels unknowingly if we actually are ten steps faraway from even seeing the output.

Anyway, he did all that, and we grew to become all in favour of covariates, which led to this.

Propensity scores, CS and “small handled models”

So it’s possible you’ll recall the opposite day that I did that many analyst design the place I estimated the variation in ATT estimates throughout three programming languages (python, R and Stata) every of which had two packages, giving me six packages per CS specification. The outcomes had been miserable.

Mainly loads of what I discovered needed to do with how the totally different packages dealt with “large numbers” from that early, largely hidden first stage, in CS of estimating a logit, cohort by cohort, modeling the therapy project (a dummy variable therapy indicator) in opposition to baseline covariates. And should you recall the best way that every bundle was rounding and utilizing floating numbers was creating all these issues, however the packages dealt with the issue in a different way which precipitated the ATT estimates to swing round, however just for circumstances the place you had covariates in any respect, and had been utilizing double strong and the IPW.

So logits have ever since been in my head, however actually, they’ve been in my head for years with CS. However right here’s the gist of it — there’s an early paper from the Nineties by Peduzzi, et al. (1996) that kind of discovered that you just want round “10 occasions” per covariate to keep away from good separation with one thing like logit. Yow will discover the paper right here. They use the phrase “occasions per variable” or EPV. And principally, with logistic regressions, they did a Monte Carlo and located that when the EPV values had been 10+, you had been superb. Which means “10 handled states” for every covariate in our context. however while you had lower than 10, the regression coefficients get wonky and are biased in each optimistic and detrimental path. You’ll be able to see the summary right here, however I extremely encourage you to learn this paper in case you are operating CS with the double strong or IPW specification.

Within the case of diff-in-diff, an “occasion” is the variety of handled models on the cohort stage. It’s a left hand aspect variable idea — it’s the variety of handled models. And when you might imagine “properly I’m superb. I’ve 20 handled states”. Sure, you do have 20 handled states. However bear in mind — CS estimates every propensity rating, and every 2×2, per cohort-year. So, don’t ask what number of handled states you could have. Ask what number of handled states do I’ve in a given 12 months? Do you could have 10? Do you could have 5? Do you could have 1? The reply is “Sure”. You almost certainly have state of affairs the place 10 states get handled, the place 5 get handled, and actually, you most likely just a few singletons in several years.

This can be a large deal, I believe. If you’re actually solely having a single handled state in your CS state of affairs, and are estimating a logit below the hood since you did double strong, or IPW, then you definately actually shouldn’t even doing it within the first place. You will get the unsuitable propensity scores if it even does it. It’s all below the hood bear in mind. CS did, csdid, diff-diff, variations, all of them — they don’t report the coefficients from the logit. Typically they don’t even maintain the coefficients. They will not even retailer the propensity scores so you may test this out. So I believe this isn’t a trivial deal.

Why? Why is that this not trivial. As a result of in the US, we regularly are working with state stage knowledge. And we’re working with staggered adoption. And we regularly covariates. And an increasing number of we’re estimating Callaway and Sant’Anna. Which implies we’re working with conditions which typically have fewer than 10 occasions per variable. Which implies the logit is biased. Ugh.

You’ll be able to watch me and Claude Code in “Dispatch” speak about all this, however I extremely encourage you interrogate Claude Code by yourself. He is aware of that Peduzzi, et al. (1996) paper too, however go forward and provides him this hyperlink when you’re at it, after which have him examine conditions the place you could have fewer than 10 occasions per covariate utilizing a logit on your state-level dataset and inform me what you discover.

What I’m Pondering

So I’m going to be excited about this for some time. This “occasions per variable” deal just isn’t trivial. See, in our JEL paper, we studied Medicaid reform on mortality utilizing county-level knowledge. Not state-level knowledge. Why does that matter?

Properly, that issues as a result of you could have loads of occasions per variable should you drop all the way down to the county stage. A single state like Texas has 254 counties. So if Texas will get handled, then you definately don’t get one handled unit — you get 254 handled models.

It would really feel prefer it’s dishonest, but it surely isn’t. Verify the JEL paper the place we talk about weighting by county inhabitants and the way that modifications the goal parameter from the “common county” to the “common particular person”. I believe really that should you inhabitants weighted in state stage knowledge, and inhabitants weighted in county stage knowledge, the goal ATT parameter is similar both manner and so is the parallel tendencies assumption. Each the ATT and the parallel tendencies assumption will probably be expressed as weighted means. So I believe it’s going to be similar.

Besides in a single state of affairs. And that’s covariates and the logit estimation. You probably have just one handled state versus 254 handled counties, and also you estimate a logit with 1 or 2 covariates, I believe that can mess up stuff within the former, however not a lot within the latter. So in case your aim is the inhabitants stage ATT within the first place, I believe it’s essential transfer all the way down to the county stage simply to outlive this complete factor! I’m going to dig into it extra with a simulation myself, however frankly, I believe you must too. Begin excited about it — you don’t want my substack for that.

However I don’t suppose it’s trivial in any respect, and I don’t suppose it’s one thing extensively appreciated with CS. We spend a lot time targeted on staggering and so little time excited about covariates as of late. And there may be an informality we how we speak in regards to the parallel tendencies assumption, focusing solely on the occasion research, and due to this fact the inclusion of covariates is form of one in every of these “ehhh” sort issues. However I’ve seen the signal flip in an occasion research from one covariate. I’ll write one other submit about it one other time, although. For now, study these occasions per variable, however I don’t suppose we’re speaking about this sufficient, in any respect. So I could go on a little bit of a rabbit gap on this. So buckle up.

Think about regression adjustment over the propensity rating

However there may be a simple repair should you can’t get your arms on county stage knowledge — don’t specify double strong and don’t specify IPW. Use regression adjustment as a substitute. Why?

As a result of, initially, regression adjustment just isn’t logistic regression, which is what that Peduzzi, et al. (1996) article is about. In order that’s one factor.

Two, there are not any “occasions per variable” while you use regression adjustment. Why? As a result of in regression adjustment, you might be regressing the primary distinction within the final result in opposition to the baseline covariates utilizing OLS. There aren’t any occasions. There’s no dummy. You aren’t modeling the therapy with regression adjustment — you might be modeling the primary distinction final result. I believe it’s possible you’ll can get away with it there, and actually you aren’t sacrificing a ton. You’re nonetheless interesting to conditional parallel tendencies. And also you additionally don’t want widespread help with OLS. The unique HIT 1997 I believe did want that because it was nonparametric, however I believe below the hood in CS it’s vanilla OLS, so it’ll simply extrapolate with funky traces.

In order that may even be one thing I’m going to be attending to the underside of, which suggests Claude Code will come to the mountain prime with me to be taught extra about logits within the context of difference-in-differences and CS.

Dispatch and Conclusion

There’s extra however I’m going to cease there, however within the movies you’ll discover much more issues we talked about. However right here’s the factor I need to let you know.

I actually dug this Dispatch. Large time. I believe I’m going to get a Mini or a Studio too. I believe it’s time. This iMac desktop I’ve sucks within the age of Claude Code. My MacBook Professional is okay. I don’t plan on updating it for most likely a 12 months or two as I actually maxed it out again in 2022. And to this point it’s not giving me bother.

However that iMac I bought may as properly be a calculator at this level within the age of Claude Code. A headless mini or simply linked to my exterior monitor might be superb. I solely get round $250 for the commerce in on this factor, however no matter. I believe having a house workplace arrange is value it given the form of work I now do. So, that’s the place I’m heading. Hope this was useful. Try Dispatch. I believe it’s going to blow you away.

Safely Deploying ML Fashions to Manufacturing: 4 Managed Methods (A/B, Canary, Interleaved, Shadow Testing)

Artificial Intelligence

Dr. Mike

March 22, 2026

Safely Deploying ML Fashions to Manufacturing: 4 Managed Methods (A/B, Canary, Interleaved, Shadow Testing)

Deploying a brand new machine studying mannequin to manufacturing is likely one of the most crucial levels of the ML lifecycle. Even when a mannequin performs effectively on validation and take a look at datasets, straight changing the present manufacturing mannequin may be dangerous. Offline analysis not often captures the total complexity of real-world environments—knowledge distributions could shift, person conduct can change, and system constraints in manufacturing could differ from these in managed experiments.

Because of this, a mannequin that seems superior throughout improvement may nonetheless degrade efficiency or negatively affect person expertise as soon as deployed. To mitigate these dangers, ML groups undertake managed rollout methods that enable them to guage new fashions underneath actual manufacturing circumstances whereas minimizing potential disruptions.

On this article, we discover 4 broadly used methods—A/B testing, Canary testing, Interleaved testing, and Shadow testing—that assist organizations safely deploy and validate new machine studying fashions in manufacturing environments.

A/B Testing

A/B testing is likely one of the most generally used methods for safely introducing a brand new machine studying mannequin in manufacturing. On this strategy, incoming site visitors is cut up between two variations of a system: the present legacy mannequin (management) and the candidate mannequin (variation). The distribution is usually non-uniform to restrict threat—for instance, 90% of requests could proceed to be served by the legacy mannequin, whereas solely 10% are routed to the candidate mannequin.

By exposing each fashions to real-world site visitors, groups can examine downstream efficiency metrics comparable to click-through price, conversions, engagement, or income. This managed experiment permits organizations to guage whether or not the candidate mannequin genuinely improves outcomes earlier than steadily rising its site visitors share or totally changing the legacy mannequin.

Canary Testing

Canary testing is a managed rollout technique the place a brand new mannequin is first deployed to a small subset of customers earlier than being steadily launched to all the person base. The title comes from an previous mining apply the place miners carried canary birds into coal mines to detect poisonous gases—the birds would react first, warning miners of hazard. Equally, in machine studying deployments, the candidate mannequin is initially uncovered to a restricted group of customers whereas the bulk proceed to be served by the legacy mannequin.

Not like A/B testing, which randomly splits site visitors throughout all customers, canary testing targets a particular subset and progressively will increase publicity if efficiency metrics point out success. This gradual rollout helps groups detect points early and roll again rapidly if essential, lowering the chance of widespread affect.

Interleaved Testing

Interleaved testing evaluates a number of fashions by mixing their outputs inside the identical response proven to customers. As a substitute of routing a complete request to both the legacy or candidate mannequin, the system combines predictions from each fashions in actual time. For instance, in a advice system, some gadgets within the advice listing could come from the legacy mannequin, whereas others are generated by the candidate mannequin.

The system then logs downstream engagement indicators—comparable to click-through price, watch time, or unfavorable suggestions—for every advice. As a result of each fashions are evaluated inside the identical person interplay, interleaved testing permits groups to match efficiency extra straight and effectively whereas minimizing biases attributable to variations in person teams or site visitors distribution.

Shadow Testing

Shadow testing, also referred to as shadow deployment or darkish launch, permits groups to guage a brand new machine studying mannequin in an actual manufacturing surroundings with out affecting the person expertise. On this strategy, the candidate mannequin runs in parallel with the legacy mannequin and receives the identical reside requests because the manufacturing system. Nonetheless, solely the legacy mannequin’s predictions are returned to customers, whereas the candidate mannequin’s outputs are merely logged for evaluation.

This setup helps groups assess how the brand new mannequin behaves underneath real-world site visitors and infrastructure circumstances, which are sometimes troublesome to copy in offline experiments. Shadow testing gives a low-risk strategy to benchmark the candidate mannequin in opposition to the legacy mannequin, though it can not seize true person engagement metrics—comparable to clicks, watch time, or conversions—since its predictions are by no means proven to customers.

Simulating ML Mannequin Deployment Methods

Setting Up

Earlier than simulating any technique, we want two issues: a strategy to signify incoming requests, and a stand-in for every mannequin.

Every mannequin is solely a perform that takes a request and returns a rating — a quantity that loosely represents how good that mannequin’s advice is. The legacy mannequin’s rating is capped at 0.35, whereas the candidate mannequin’s is capped at 0.55, making the candidate deliberately higher so we will confirm that every technique truly detects the development.

make_requests() generates 200 requests unfold throughout 40 customers, which supplies us sufficient site visitors to see significant variations between methods whereas maintaining the simulation light-weight.

import random
import hashlib
 
random.seed(42)


def legacy_model(request):
    return {"mannequin": "legacy",    "rating": random.random() * 0.35}
 
def candidate_model(request):
    return {"mannequin": "candidate", "rating": random.random() * 0.55}
 
def make_requests(n=200):
    customers = [f"user_{i}" for i in range(40)]
    return [{"id": f"req_{i}", "user": random.choice(users)} for i in range(n)]
 
requests = make_requests()

A/B Testing

ab_route() is the core of this technique — for each incoming request, it attracts a random quantity and routes to the candidate mannequin provided that that quantity falls under 0.10, in any other case the request goes to legacy. This offers the candidate roughly 10% of site visitors.

We then accumulate the prediction scores from every mannequin individually and compute the common on the finish. In an actual system, these scores would get replaced by precise engagement metrics like click-through price or watch time — right here the rating simply stands in for “how good was this advice.”

print("── 1. A/B Testing ──────────────────────────────────────────")
 
CANDIDATE_TRAFFIC = 0.10   # 10 % of requests go to candidate
 
def ab_route(request):
    return candidate_model if random.random() < CANDIDATE_TRAFFIC else legacy_model
 
outcomes = {"legacy": [], "candidate": []}
for req in requests:
    mannequin  = ab_route(req)
    pred   = mannequin(req)
    outcomes[pred["model"]].append(pred["score"])
 
for title, scores in outcomes.gadgets():
    print(f"  {title:12s} | requests: {len(scores):3d} | avg rating: {sum(scores)/len(scores):.3f}")

Canary Testing

The important thing perform right here is get_canary_users(), which makes use of an MD5 hash to deterministically assign customers to the canary group. The vital phrase is deterministic — sorting customers by their hash means the identical customers all the time find yourself within the canary group throughout runs, which mirrors how actual canary deployments work the place a particular person constantly sees the identical mannequin.

We then simulate three phases by merely increasing the fraction of canary customers — 5%, 20%, and 50%. For every request, routing is set by whether or not the person belongs to the canary group, not by a random coin flip like in A/B testing. That is the basic distinction between the 2 methods: A/B testing splits by request, canary testing splits by person.

print("n── 2. Canary Testing ───────────────────────────────────────")
 
def get_canary_users(all_users, fraction):
    """Deterministic person task through hash -- secure throughout restarts."""
    n = max(1, int(len(all_users) * fraction))
    ranked = sorted(all_users, key=lambda u: hashlib.md5(u.encode()).hexdigest())
    return set(ranked[:n])
 
all_users = listing(set(r["user"] for r in requests))
 
for section, fraction in [("Phase 1 (5%)", 0.05), ("Phase 2 (20%)", 0.20), ("Phase 3 (50%)", 0.50)]:
    canary_users = get_canary_users(all_users, fraction)
    scores = {"legacy": [], "candidate": []}
    for req in requests:
        mannequin = candidate_model if req["user"] in canary_users else legacy_model
        pred  = mannequin(req)
        scores[pred["model"]].append(pred["score"])
    print(f"  {section} | canary customers: {len(canary_users):2nd} "
          f"| legacy avg: {sum(scores['legacy'])/max(1,len(scores['legacy'])):.3f} "
          f"| candidate avg: {sum(scores['candidate'])/max(1,len(scores['candidate'])):.3f}")

Interleaved Testing

Each fashions run on each request, and interleave() merges their outputs by alternating gadgets — one from legacy, one from candidate, one from legacy, and so forth. Every merchandise is tagged with its supply mannequin, so when a person clicks one thing, we all know precisely which mannequin to credit score.

The small random.uniform(-0.05, 0.05) noise added to every merchandise’s rating simulates the pure variation you’d see in actual suggestions — two gadgets from the identical mannequin gained’t have equivalent high quality.

On the finish, we compute CTR individually for every mannequin’s gadgets. As a result of each fashions competed on the identical requests in opposition to the identical customers on the identical time, there is no such thing as a confounding issue — any distinction in CTR is solely right down to mannequin high quality. That is what makes interleaved testing essentially the most statistically clear comparability of the 4 methods.

print("n── 3. Interleaved Testing ──────────────────────────────────")
 
def interleave(pred_a, pred_b):
    """Alternate gadgets: A, B, A, B ... tagged with their supply mannequin."""
    items_a = [("legacy",    pred_a["score"] + random.uniform(-0.05, 0.05)) for _ in vary(3)]
    items_b = [("candidate", pred_b["score"] + random.uniform(-0.05, 0.05)) for _ in vary(3)]
    merged  = []
    for a, b in zip(items_a, items_b):
        merged += [a, b]
    return merged
 
clicks = {"legacy": 0, "candidate": 0}
proven  = {"legacy": 0, "candidate": 0}
 
for req in requests:
    pred_l = legacy_model(req)
    pred_c = candidate_model(req)
    for supply, rating in interleave(pred_l, pred_c):
        proven[source]  += 1
        clicks[source] += int(random.random() < rating)   # click on ~ rating
 
for title in ["legacy", "candidate"]:
    print(f"  {title:12s} | impressions: {proven[name]:4d} "
          f"| clicks: {clicks[name]:3d} "
          f"| CTR: {clicks[name]/proven[name]:.3f}")

Shadow Testing

Each fashions run on each request, however the loop makes a transparent distinction — live_pred is what the person will get, shadow_pred goes straight into the log and nothing extra. The candidate’s output is rarely returned, by no means proven, by no means acted on. The log listing is all the level of shadow testing. In an actual system this could be written to a database or an information warehouse, and engineers would later question it to match latency distributions, output patterns, or rating distributions in opposition to the legacy mannequin — all and not using a single person being affected.

print("n── 4. Shadow Testing ───────────────────────────────────────")
 
log = []   # candidate's shadow log
 
for req in requests:
    # What the person sees
    live_pred   = legacy_model(req)
 
    # Shadow run -- by no means proven to person
    shadow_pred = candidate_model(req)
 
    log.append({
        "request_id":       req["id"],
        "legacy_score":     live_pred["score"],
        "candidate_score":  shadow_pred["score"],    # logged, not served
    })
 
avg_legacy    = sum(r["legacy_score"]    for r in log) / len(log)
avg_candidate = sum(r["candidate_score"] for r in log) / len(log)
 
print(f"  Legacy    avg rating (served):  {avg_legacy:.3f}")
print(f"  Candidate avg rating (logged):  {avg_candidate:.3f}")
print(f"  Be aware: candidate rating has no click on validation -- shadow solely.")

Take a look at the FULL Pocket book Right here. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 120k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you may be a part of us on telegram as effectively.

I’m a Civil Engineering Graduate (2022) from Jamia Millia Islamia, New Delhi, and I’ve a eager curiosity in Knowledge Science, particularly Neural Networks and their utility in varied areas.

Historical Fragments Might Be The World’s Oldest Recognized Geometry : ScienceAlert

Science

Dr. Mike

March 21, 2026

Historical Fragments Might Be The World’s Oldest Recognized Geometry : ScienceAlert

Historical people have been surprisingly inventive, structured, and geometrical of their considering some 60,000 years in the past, in response to some intricately engraved ostrich eggshells discovered throughout southern Africa.

The surviving fragments from these adorned shells, first described in 2010, aren’t randomly scribbled, like the perimeters of a daydreamer’s pocket book, however etched with mindfully geometric options, similar to grids and diamonds.

These designs, analyzed in a brand new research from researchers in Italy, present a bridge between eons and a glimpse into the minds of our predecessors.

“These indicators reveal a surprisingly structured, geometric mind-set,” says Silvia Ferrara, an archaeologist on the College of Bologna and senior creator of the research.

“We’re speaking about individuals who didn’t merely draw strains, however organised them in response to recurring ideas – parallelisms, grids, rotations and systematic repetitions: a visible grammar in embryo.”

The adorned ostrich eggshell fragments recovered in southern Africa. (Texier et al., J. Archaeol. Sci., 2013)

To decipher the ‘geometric grammar’ utilized by the shells’ decorators, the researchers analyzed the spatial association of marki(TExier et al. ngs on 112 engraved ostrich eggshell fragments. Found at two shelter websites in South Africa and a cave website in southern Namibia, these intact shells might have been used of their heyday as water containers, as they nonetheless are by some foragers within the area.

However have been their designs deliberately complicated? The researchers examined practically 1,300 strains etched on the shell fragments and concluded that their makers demonstrated a hanging stage of cognitive group.

Greater than 80 p.c of the etchings show coherent spatial regularities, with designs wealthy in parallelism, proper angles, and the repetition of strains and patterns. The extra complicated creations function hatched bands, grids, and diamond motifs.

These designs present proof that early people had not solely a regular hand but additionally a inventive thoughts able to cognitive operations similar to rotation, translation, and embedding, which rework easy strains into various creations and hierarchical designs.

What’s extra, that is tangible proof of the cognitive basis needed for the emergence of summary thought. The flexibility to conceptualize issues outdoors of 1’s private expertise, together with issues that may’t be seen or might not even exist, is a defining human trait and the supply of all our good (and dangerous) fiction.

The research builds on earlier findings suggesting that people possess an intuitive sense of geometric options, similar to line, level, parallelism, and proper angles, and that these ‘geometric intuitions’ are a part of humanity’s core data.

“These engravings are organised and constant, and present mastery of geometric relationships,” says Ferrara.

“There may be not solely a technique of repeating indicators: there may be actual visuo-spatial planning, as if the authors already had an total picture of the determine in thoughts earlier than engraving it.”

Associated: 10,000-Yr-Outdated Symbols And Artwork Present in Egyptian Rock Formation

In fact, it is unclear if these designs maintain some deeper esoteric which means – and that is past the scope of the work. However they spotlight an important development within the evolution of human considering, one which laid the muse for a future filled with artwork and invention.

“Our evaluation reveals that Homo sapiens 60,000 years in the past already possessed a outstanding skill to organise visible area in response to summary ideas,” says Valentina Decembrini, a PhD scholar on the College of Bologna and the research’s first creator.

“Reworking easy varieties into complicated programs by following outlined guidelines is a deeply human trait that has characterised our historical past over millennia, from the creation of decorations to the event of symbolic programs and, finally, writing.”

This analysis was revealed in PLOS One.

Decreasing GPU Reminiscence and Accelerating Transformers

Artificial Intelligence

Dr. Mike

March 21, 2026

Decreasing GPU Reminiscence and Accelerating Transformers

Introduction

The transformer revolution is now deep into its lengthy‑context period. Fashions like GPT‑4 (32 okay tokens), MosaicML’s MPT (65 okay), and Claude (100 okay) can course of total chapters or codebases. But as context grows, the consideration mechanism turns into the bottleneck: calculating the similarity matrix S = Q·Ok^T and the likelihood matrix P = softmax(S) produces N×N information buildings. These matrices have to be moved between the GPU’s tiny on‑chip SRAM and its bigger however slower excessive‑bandwidth reminiscence (HBM), consuming bandwidth and limiting throughput. In a world the place compute FLOPs proceed to climb, the actual constraint has change into reminiscence.

FlashAttention, launched in 2022, addressed this downside by tiling the computation to keep away from ever storing the complete S or P matrices, delivering 2–4× speedups and as much as 10–20× reminiscence financial savings. FlashAttention‑2 (FA2) goes additional: it reduces expensive non‑matmul operations, parallelizes throughout sequence size, and partitions work to reduce shared‑reminiscence visitors. Benchmarks present FA2 is about twice as quick as its predecessor and as much as 9 instances quicker than customary consideration implementations, hitting 225 TFLOPs/s on NVIDIA A100 GPUs. This information explains how FA2 works, when to make use of it, combine it into your stack, and the place its limits lie.

Fast Digest

FA2 solves a reminiscence‑sure downside. Consideration’s N² reminiscence footprint stalls GPUs; tiling and kernel fusion deliver it right down to linear reminiscence value.
Key improvements: fewer non‑matmul FLOPs, further parallelism alongside sequence size, and slicing the question matrix throughout warps.
Adoption: Helps Ampere/Ada/Hopper GPUs and FP16/BF16 datatypes. Set up by way of pip and flip a flag in PyTorch or Hugging Face to allow.
Who advantages: Anybody coaching or serving lengthy‑context fashions (8 okay–16 okay tokens) or utilizing giant head dimensions; value financial savings are substantial.
Caveats: Solely consideration is accelerated; feed‑ahead layers stay unchanged. FP32 precision and older GPUs are unsupported.

The Reminiscence Bottleneck in Transformers

Why reminiscence—not compute—issues

Every token attends to each different token, so naïve consideration materializes N×N matrices. With 4 okay tokens and 96 heads, the similarity and likelihood matrices alone eat a number of gigabytes. On trendy GPUs, information motion between the tiny on‑chip SRAM (≈20 MB) and HBM (≈40–80 GB) dominates runtime. Extra compute doesn’t assist if the algorithm shuttles giant intermediate outcomes backwards and forwards.

To determine whether or not you want FA2, carry out the MEMS Verify:

Reminiscence – Estimate your consideration matrix measurement. If it could possibly’t slot in SRAM and triggers out‑of‑reminiscence errors, you’re reminiscence‑sure.
Effectivity – Use profilers (Nsight or PyTorch) to see if kernels saturate compute or stall on reminiscence transfers.
Mannequin measurement – Many heads or giant embeddings enhance reminiscence overhead.
Sequence size – Past ~2 okay tokens, customary consideration’s O(N²) reminiscence explodes.

If two or extra components flag crimson, FA2 may help. Nonetheless, duties with brief sequences (≤512 tokens) stay compute‑sure and received’t profit from tiling; the overhead of customized kernels might even sluggish them down.

Professional perception

“FlashAttention exploits the uneven GPU reminiscence hierarchy to deliver vital reminiscence saving and a pair of–4× speedups with out approximation.” – Dao et al.

Understanding that reminiscence—not computation—limits consideration is essential to appreciating FA2’s worth.

Fast abstract

Why does reminiscence restrict consideration? As a result of consideration creates big N² matrices that have to be moved between sluggish and quick reminiscence. Profilers assist decide in case your workload is reminiscence‑sure.

FlashAttention Fundamentals—Tiling and Recomputing

Tiling and kernel fusion

FlashAttention reorders computation to keep away from ever materializing the complete N×N matrices. It divides queries (Q), keys (Ok), and values (V) into blocks that slot in SRAM, performs matrix multiplications and softmax operations on these blocks, and accumulates partial sums till the ultimate output is produced. As a result of all intermediate work stays on‑chip, reminiscence visitors drops dramatically.

Kernel fusion performs a vital position: as an alternative of launching separate CUDA kernels for matmul, scaling, softmax, masking, dropout, and worth projection, FlashAttention performs them inside a single kernel. This ensures that information isn’t written again to HBM between steps.

Recomputation within the backward move

Throughout backpropagation, naïve consideration should retailer your complete consideration matrix to compute gradients. FlashAttention saves reminiscence by recomputing the required native softmax values on the fly. The small value of additional computation is outweighed by eliminating gigabytes of storage.

Unfavorable data

FlashAttention doesn’t alter the mathematical method for consideration; any deviations in output sometimes come up from utilizing decrease precision (FP16/BF16). Early variations lacked dropout assist, so guarantee your library model accommodates dropout if wanted.

Fast abstract

How does FlashAttention scale back reminiscence? By tiling Q/Ok/V into blocks, fusing operations right into a single kernel, and recomputing softmax values throughout backprop.

What’s New in FlashAttention‑2

FA2 refines FlashAttention in three main methods:

Fewer non‑matmul operations: GPUs obtain huge throughput on matrix multiplication however decelerate on normal FP32 operations. FA2 rewrites rescaling and masking code to reduce these non‑matmul FLOPs.
Parallelism alongside the sequence dimension: When batch measurement × head rely is small, the unique FlashAttention can’t saturate all GPU streaming multiprocessors. FA2 parallelizes throughout lengthy sequences, boosting occupancy.
Question slicing: As an alternative of slicing keys and values throughout warps (requiring synchronization), FA2 slices the question matrix, permitting warps to compute their output independently. This eliminates shared‑reminiscence writes and delivers extra pace.

FA2 additionally helps head dimensions as much as 256, in addition to multi‑question (MQA) and grouped‑question (GQA) consideration. Head dimension assist issues for code‑oriented fashions like CodeGen or GPT‑J.

Determination steering

Use this fast determination tree:

If you run on Turing GPUs (e.g., T4) –> stick with FlashAttention 1 or customary kernels.
Else if your head dimension >128 –> select FA2.
Else if (batch_size × num_heads) is small and sequence is lengthy –> FA2’s further parallelism pays off.
Else benchmark FA1 and FA2; the less complicated implementation might suffice.

Caveats

FA2 requires Ampere, Ada, or Hopper GPUs and presently helps solely FP16/BF16 datatypes. Compilation is extra advanced, and unsupported GPUs will fall again to FA1 or customary consideration.

Professional perception

“FlashAttention‑2 is about 2× quicker than FlashAttention and reaches as much as 230 TFLOPs/s on A100 GPUs.” – Tri Dao

FA2 closes a lot of the hole between consideration kernels and optimized matrix multiplications.

Fast abstract

What distinguishes FA2? It cuts non‑matmul operations, parallelizes over sequence size, slices queries as an alternative of keys/values, and helps bigger head sizes and MQA/GQA.

Putting in and Integrating FlashAttention‑2

Necessities and set up

FA2 helps A100, H100, RTX 3090/4090, and AMD MI200/MI300 GPUs and requires FP16/BF16 precision. Set up by way of:

pip set up flash-attn --no-build-isolation

Guarantee CUDA ≥12.0 (or ROCm ≥6.0) and PyTorch ≥2.2. Set up the ninja construct system to shorten compile instances; in case your machine has restricted RAM, cap parallel jobs utilizing MAX_JOBS=4.

Enabling FA2 in frameworks

In Hugging Face Transformers, set the use_flash_attn_2=True flag when instantiating your mannequin. For customized code, import and name the kernel:

from flash_attn_interface import flash_attn_func
output = flash_attn_func(q, okay, v, causal=True)

Enter tensors ought to be formed [batch, seq_len, num_heads, head_dim] or as required by the library. For unsupported {hardware}, implement a attempt/besides block to fall again to plain consideration.

Operational recommendation

GPU orchestration: Platforms like Clarifai’s compute orchestration make it simple to run FA2 on clusters. Choose A100 or H100 GPUs, and use the constructed‑in profiling instruments to observe tokens per second. For those who want turnkey {hardware}, Clarifai’s GPU internet hosting offers managed A100/H100 cases that combine with native runners and distant orchestration.
Combined precision: Mix FA2 with computerized combined precision (AMP) to maximise throughput.
Benchmarking: After integration, measure tokens per second, GPU reminiscence utilization, and wall‑clock time with and with out FA2. Use these numbers to regulate batch sizes and sequence lengths.

Fast abstract

How do I take advantage of FA2? Set up the package deal, guarantee you will have suitable GPUs and drivers, allow FA2 in your framework, and benchmark. Use Clarifai’s orchestration and mannequin inference instruments for scalable deployment.

Efficiency Benchmarks and Value Financial savings

Speedups on A100 and H100

Public benchmarks report that FA2 delivers round 2× speedup over FA1 and as much as 9× over customary PyTorch consideration. When coaching GPT‑fashion fashions finish‑to‑finish, FA2 achieves 225 TFLOPs/s on A100 GPUs and even increased throughput on H100 resulting from newer tensor cores.

An analysis by Lambda Labs exhibits that FA2 will increase the inexpensive batch measurement from 1 to 4 whereas protecting GPU reminiscence fixed; tokens per second soar from 3,717 to 10,650 on A100 and from 6,267 to 22,282 on H100.

Config	Tokens/sec	Batch measurement	Notes
A100 baseline	3,717	1	Normal consideration
A100 FA2	10,650	4	2.9× throughput enhance
H100 baseline	6,267	1	Normal consideration
H100 FA2	22,282	4	3.5× throughput enhance

Scaling to multi‑GPU clusters yields close to‑linear efficiency when excessive‑bandwidth interconnects (NVLink/NVSwitch) can be found.

Value influence

As a result of FA2 permits bigger batch sizes and better throughput, it reduces coaching time and compute value. For instance, replicating GPT3‑175B coaching with FA2 on 1,024 H100 GPUs is estimated to value round $458 okay, a 90 % discount in contrast with conventional kernels. On cloud platforms like Clarifai, fewer GPU hours translate instantly into value financial savings.

Caveats

Iter/sec might drop barely as a result of every batch is bigger. Precise tokens/sec is the significant metric; make sure you measure the appropriate amount. Multi‑GPU positive aspects rely upon interconnect bandwidth; low‑bandwidth clusters might not understand full speedups.

Fast abstract

How a lot quicker is FA2? Roughly twice as quick as FA1 and as much as 9 instances quicker than customary consideration. It will increase batch measurement and reduces coaching prices dramatically.

Sensible Use Circumstances and Determination Information

Lengthy‑context language fashions

FA2 shines when you want to course of lengthy paperwork, tales, or transcripts. With its linear reminiscence value, you’ll be able to prepare or fantastic‑tune fashions on 16 okay–64 okay tokens with out approximations. Authorized doc evaluate, novel writing, and analysis paper summarization all profit. Clarifai’s mannequin inference pipeline makes it simple to deploy these giant fashions and serve predictions at scale.

Code and multimodal technology

Fashions like CodeGen or Secure Diffusion 1.x use giant head dimensions (as much as 256), which FA2 helps. This enables for deeper code context or increased decision photos with out working out of reminiscence.

Excessive‑throughput inference with MQA/GQA

FA2’s assist for multi‑question and grouped‑question consideration reduces KV cache measurement and accelerates inference. That is very best for chatbots and actual‑time assistants serving hundreds of customers concurrently.

Determination matrix

State of affairs	Sequence size	Head dim	GPU	Suggestion
Quick textual content classification	≤2 okay	≤64	Any	Normal/FA1
Lengthy doc summarization	8 okay–16 okay	≤128	A100/H100	FA2
Code technology	4 okay–8 okay	256	A100/H100	FA2
Actual‑time inference	≤4 okay	≤128	A100/H100	FA2 with MQA/GQA
Extremely‑lengthy context (≥64 okay)	>64 okay	any	Combined GPU/CPU	Sparse/approximate

Frequent errors and ideas

Don’t assume that larger batches at all times enhance coaching; chances are you’ll have to retune studying charges. Multi‑GPU speedups rely upon interconnect bandwidth; examine whether or not your cluster makes use of NVLink. Lastly, keep in mind that FA2 accelerates self‑consideration solely—feed‑ahead layers should dominate runtime.

Fast abstract

Who ought to use FA2? Practitioners working with lengthy contexts, giant head sizes, or excessive‑throughput inference. Quick sequences or unsupported GPUs might not profit.

Limitations and Alternate options

Precision and {hardware} constraints

FA2 runs solely on Ampere/Ada/Hopper GPUs and AMD’s MI200/MI300 collection and helps FP16/BF16 datatypes. FP32 precision and older GPUs require falling again to FA1 or customary consideration. Edge units and cell GPUs are usually unsupported.

The place FA2 received’t assist

In case your sequences are brief (≤512 tokens) or your mannequin has few heads, the overhead of FA2 might outweigh its advantages. It doesn’t speed up feed‑ahead layers, convolutional operations, or embedding lookups; for these, think about different optimizations.

Alternate options

For very lengthy sequences (>64 okay tokens) or {hardware} with out FA2 assist, think about Performer, Linformer, Longformer, or Paged Consideration. These strategies approximate consideration through the use of low‑rank projections or native sparsity. They might sacrifice some accuracy however can deal with contexts that FA2 can’t.

Fast abstract

When do you have to keep away from FA2? When precision have to be FP32, when working on unsupported GPUs, when contexts are brief, or when approximations suffice for excessive lengths.

Trying Forward

Rising kernels

FlashAttention‑3 (FA3) targets the H100 GPU, provides FP8 assist, and leverages Tensor Reminiscence Accelerator {hardware}, pushing throughput even increased. FlashAttention‑4 (FA4) is being rewritten in CuTeDSL for Hopper and Blackwell GPUs, with plans for unified kernels and full FP8 assist. These kernels are in beta; adoption will rely upon {hardware} availability.

New consideration variants

Researchers are combining {hardware}‑conscious kernels like FA2 with algorithmic improvements. Flash‑Decoding accelerates autoregressive inference by caching partial outcomes. Paged Consideration breaks sequences into pages for reminiscence‑environment friendly inference, enabling 64 okay contexts and past. FastAttention adapts FA kernels to NPUs and low‑useful resource GPUs. Anticipate hybrid methods that unify tiling, sparsity, and new precisions.

Getting ready for the long run

To remain forward, observe these steps: subscribe to flash-attn launch notes, take a look at FP8 workflows in case your fashions tolerate decrease precision, plan for A100/H100/B200 upgrades, and discover combining FA kernels with sparse consideration for extremely‑lengthy contexts. Clarifai’s roadmap contains assist for brand spanking new GPUs and FP8, serving to groups undertake these improvements with out overhauling infrastructure.

Fast abstract

What’s subsequent? FA3 and FA4 goal new GPUs and FP8, whereas variants like Flash‑Decoding and Paged Consideration deal with inference and very lengthy contexts. Hybrid strategies will proceed to push transformer effectivity.

FAQs

Q: Does FlashAttention‑2 change the eye computation?
A: No. FA2 preserves the precise softmax consideration method. Variations in output come up from decrease precision; use FP16/BF16 accordingly.

Q: Does FA2 assist dropout and cross‑consideration?
A: Current variations assist dropout and are being prolonged to cross‑consideration. Verify your library’s documentation for specifics.

Q: Can I take advantage of FA2 with LoRA or quantization?
A: Sure. FA2 operates on the kernel stage and is suitable with methods like LoRA and quantization, making it complement to different reminiscence‑saving strategies.

Q: What about JAX or TensorFlow?
A: Official FA2 kernels can be found for PyTorch. Third‑social gathering ports exist for different frameworks however might lag behind in efficiency and options.

Conclusion

As transformer fashions stretch into the tens of hundreds of tokens, reminiscence, not compute, is the bottleneck. FlashAttention‑2 offers a well timed resolution: by tiling computations, fusing kernels, decreasing non‑matmul operations, and parallelizing throughout sequence size, it brings consideration efficiency nearer to the effectivity of optimized matrix multiplication. It doubles the pace of its predecessor and dramatically cuts reminiscence use. Actual‑world benchmarks verify that FA2 presents substantial throughput positive aspects and value financial savings.

FA2 will not be common; it requires trendy GPUs and helps solely FP16/BF16. For extremely‑lengthy sequences or unsupported {hardware}, approximate consideration strategies stay vital options. But for almost all of lengthy‑context workloads right this moment, FA2 is essentially the most environment friendly precise consideration kernel obtainable.

Implementing FA2 is easy: set up the library, allow it in your framework, and profile efficiency. Platforms like Clarifai’s compute orchestration and mannequin inference simplify deployment throughout clusters, permitting you to deal with mannequin design and software logic. For those who don’t have GPU {hardware}, Clarifai’s GPU internet hosting presents prepared‑to‑run clusters. And to check these capabilities threat‑free, begin without spending a dime and declare credit by way of Clarifai’s signal‑up. Use our MEMS Verify to determine whether or not your workload is reminiscence‑sure, and regulate rising kernels like FA3/4 and Paged Consideration.

In 2026 and past, transformer effectivity will hinge on pairing algorithmic improvements with {hardware}‑conscious kernels. FA2 presents a glimpse into that future—one the place reminiscence bottlenecks now not constrain the horizons of our fashions.

12 3...411 Page 1 of 411

On supporting science journalism

What’s bacterial meningitis?

How is bacterial meningitis handled or prevented?

A/B Testing

Canary Testing

Interleaved Testing

Shadow Testing

Simulating ML Mannequin Deployment Methods

Setting Up

A/B Testing

Canary Testing

Interleaved Testing

Shadow Testing

What do you wish to see most from the rumored reMarkable Paper Pure?

1. Value it for the lots

2. Introduce a quicker, extra responsive expertise

3. Follow grayscale

4. Spherical out the software program

5. Lose the subscription

6. And at last, add a frontlight

The Assaults from Half 1 Have been Identification Failures

The Machine Identification Explosion

What Workload Identification Will get Proper — And The place It Falls Brief

The place It Breaks Down: AI Brokers

The Delegation Downside No One Has Solved

Identification Makes Runtime Safety Actionable

The Path Ahead: Zero Belief Should Prolong to Brokers

Introduction

Fast Digest

The Reminiscence Bottleneck in Transformers

Why reminiscence—not compute—issues

Professional perception

Fast abstract

FlashAttention Fundamentals—Tiling and Recomputing

Tiling and kernel fusion

Recomputation within the backward move

Unfavorable data

Fast abstract

What’s New in FlashAttention‑2

Determination steering

Caveats

Professional perception

Fast abstract

Putting in and Integrating FlashAttention‑2

Necessities and set up

Enabling FA2 in frameworks

Operational recommendation

Fast abstract

Efficiency Benchmarks and Value Financial savings

Speedups on A100 and H100

Value influence

Caveats

Fast abstract

Sensible Use Circumstances and Determination Information

Lengthy‑context language fashions

Code and multimodal technology

Excessive‑throughput inference with MQA/GQA

Determination matrix

Frequent errors and ideas

Fast abstract

Limitations and Alternate options

Precision and {hardware} constraints

The place FA2 received’t assist

Alternate options

Fast abstract

Trying Forward

Rising kernels

New consideration variants

Getting ready for the long run

Fast abstract

FAQs

Conclusion

The ebook that went off like a bomb

When the unsuitable traces go up

The guess that explains the world

The harmful attraction of doomsaying

Instance: Evaluating parameters from two fashions