is simple. You load the weights, load the info, and watch for it to complete. For many fashions, that’s wonderful, and the one actual price is time. When the wait will get too lengthy, the same old repair is so as to add GPUs. Each trains on a special slice of the info in parallel, and the work finishes sooner. Nothing in regards to the mannequin state adjustments; you’ve simply thrown extra fingers at it.
That covers the widespread case, the place the issue is time. Giant fashions add a second downside that extra fingers can’t resolve: house. Over the previous yr, a significant a part of my work has concerned coaching and fine-tuning massive basis fashions throughout completely different domains, together with single-cell fashions and vision-and-language transformer fashions. When you get previous a couple of billion parameters, the mannequin, its gradients, and its optimizer states now not match on a single GPU. You may’t simply replicate the work throughout GPUs as a result of a replica doesn’t match on any of them. It’s important to cut up the mannequin itself.
These two issues map onto two methods. DDP (Distributed Information Parallel) assaults time. Each GPU retains a full copy of the mannequin and trains on a special slice of the batch. FSDP (Totally Sharded Information Parallel) assaults house. The mannequin is cut up into items in order that no GPU has to carry your complete mannequin.
There aren’t actually simply two choices, although. These are simply the 2 ends of 1 dial. In between these sit the ZeRO phases from DeepSpeed (Microsoft’s coaching library), which allow you to slide from one finish to the opposite, step-by-step, buying and selling slightly reminiscence for slightly communication at every cease somewhat than leaping straight from “all people holds all the pieces” to “no one does.”
The place to set that dial and what it really controls is roofed within the first a part of the article. However there’s a second factor that decides how nicely any of those methods work. Every considered one of them works by shifting knowledge between GPUs. And how briskly that knowledge strikes relies on the bodily wires connecting them. This was one thing I didn’t take into consideration till the identical code, on the identical type of GPUs, ran a number of instances slower on one node than on one other. So within the second half, we’ll talk about the {hardware} beneath it, and the way the wiring must also be one thing you take into account. (For people who are usually not a lot into {hardware}, belief me, it’s not that dangerous!)
One observe earlier than we begin. Every thing right here is measured, not simply concept. The technique comparisons ran on 4 A100 80GB GPUs; the {hardware} experiments ran on H200 machines, together with two which have the very same GPUs wired collectively in several methods, which seems to be the entire level.
Half 1: The Software program: What Every GPU Holds
Each distributed coaching technique is de facto a solution to 1 query: what does every GPU hold its personal copy of? For that to make sense, let’s take a look at an instance.
Take Mistral-7B and fine-tune it with Adam in BF16 blended precision. The parameters take 14 GB: 7 billion of them, two bytes every. The gradients take one other 14 GB. Then we have now the optimizer states, that are the costliest half. Adam retains two working averages for each parameter, momentum and variance, each saved in FP32, which works out to roughly 58 GB. Add it up, and we attain 87 GB earlier than the mannequin has seen a single batch or a single activation has been computed.
For an A100 GPU that has 80 GB of VRAM, the mannequin doesn’t match. So we distribute. And the query, as talked about originally of this part, is: what does every GPU maintain?
Distributed Information Parallel (DDP): Everybody Holds Every thing
The straightforward reply is that each GPU holds all of it. That is DDP, and it’s the one most individuals first encounter (which was the case for me as nicely). Every GPU retains a whole copy of the mannequin: all of the parameters, all of the gradients, all of the optimizer states. What will get divided is the info. Every GPU takes a special slice of the batch, runs its personal ahead and backward cross, and computes gradients from its personal slice.
The catch is that these gradients are all completely different as a result of every GPU noticed completely different knowledge. If each GPU simply stepped its optimizer now, the copies would drift aside, and also you’d now not have one mannequin, you’d have 4. So earlier than stepping, the GPUs common their gradients. Each GPU sends its gradients out, receives everybody else’s, and so they all find yourself with the identical averaged gradient. They then take an an identical optimizer step and the copies keep in sync. That averaging is the one piece of communication DDP wants, and it occurs as soon as per step. It’s known as all-reduce, and we’ll come again to it later after we begin counting what every technique prices the wire.
DDP is quick and easy exactly as a result of it communicates so little. One all-reduce per step, and PyTorch overlaps even that with the backward cross, so on good {hardware}, you barely discover it. However its limitation is constructed into the design. Each GPU holds all the pieces, which implies including GPUs does nothing for the reminiscence downside. If 87 GB doesn’t match on one A100, it doesn’t match on 4 both, as a result of every of the 4 continues to be holding the total 87 GB. Including GPUs to DDP buys you throughput, but it surely doesn’t scale back the reminiscence every GPU has to carry.
Totally Sharded Information Parallel (FSDP): No one Holds Every thing
If the issue is that each GPU holds all the pieces, the apparent repair is to verify no GPU does. That is FSDP. As a substitute of every GPU retaining a full copy, the mannequin is cut up into items, and every GPU owns only one piece of the parameters, one piece of the gradients, and one piece of the optimizer states. On 4 GPUs, each holds roughly 1 / 4 of the 87 GB, which permits the mannequin to suit.
However this raises an instantaneous downside. You may’t run a layer by way of a GPU that solely has 1 / 4 of that layer’s weights. So FSDP reassembles every layer simply earlier than it’s wanted and takes it aside once more proper after. When the ahead cross reaches a layer, the GPUs commerce their items so that each GPU briefly has that layer’s full weights, the layer runs, after which every GPU throws away the items it doesn’t personal. The subsequent layer comes up and the identical factor occurs once more. The total mannequin is rarely resident on any single GPU; just one layer is, and solely for the immediate it’s computing. The identical factor occurs in reverse through the backward cross.
That buying and selling has names, the identical method DDP’s averaging did. Gathering everybody’s items right into a full layer is an all-gather, and scattering the summed gradients again to their house owners is a reduce-scatter. Have in mind this distinction. DDP communicates as soon as per step. FSDP communicates continuously. An all-gather earlier than each layer within the ahead cross, one other earlier than each layer within the backward cross, and a reduce-scatter after. That’s principally the commerce that you just do when switching between them. FSDP buys reminiscence and pays for it in communication.
One clarification issues right here, as a result of it might be type of complicated. FSDP continues to be knowledge parallelism. Each GPU runs your complete mannequin by itself slice of the batch, precisely like DDP. The one distinction is that the weights are saved in items and reassembled on demand as a substitute of being saved as a full copy. Every GPU nonetheless computes the entire ahead and backward cross from begin to end.
That is completely different from mannequin parallelism, the place the mannequin itself is cut up throughout GPUs, and each is chargeable for solely a part of the computation, one set of layers, or one slice of each layer. On this paradigm, a single ahead cross has to hop between GPUs as a result of no single one can end it alone. So as to add to this, even inside mannequin parallelism, we have now layer and tensor parallelism, which dictate whether or not we divide the mannequin vertically or horizontally among the many layers. However earlier than we digress an excessive amount of, these are completely different strategies with completely different communication patterns, and it’s not what this text is about. Every thing right here, DDP, FSDP, and the ZeRO phases in between, is knowledge parallel. Similar computation on each GPU, simply with completely different knowledge. Solely the storage of the mannequin differs from one technique to the subsequent.
Coming again to it, now that we’ve mentioned the speculation, let’s see what occurs in follow. To see how these two methods examine, I ran experiments on two fashions. DINOv2 (a imaginative and prescient basis mannequin) and Mistral-7B (a big language mannequin). Each experiments ran on 4 A100 80GB GPUs on a single node, utilizing HuggingFace Speed up with BF16 blended precision. The one factor that modified between runs was the technique.
DINOv2 on Meals-101. The primary experiment fine-tunes DINOv2 on Meals-101, a classification dataset with 101 meals classes and about 101k photographs. The duty is to foretell a picture class given a picture. I examined each the bottom (86M parameters) and huge (304M parameters) backbones, utilizing a batch measurement of 32 per GPU for the bottom mannequin and 256 per GPU for the massive mannequin. This can be a case the place each methods work for the reason that mannequin matches comfortably in GPU reminiscence. However even right here, we are able to see the reminiscence distinction between the 2 approaches.
FSDP makes use of considerably much less reminiscence per GPU: 0.39 GB vs. 1.79 GB for the bottom mannequin, and 1.39 GB vs. 6.26 GB for the massive mannequin. That’s roughly 4.5x much less reminiscence with FSDP. This issues much less when you might have loads of headroom, but it surely reveals us why FSDP might be important for bigger fashions. So we noticed the reminiscence, what in regards to the pace?
We are able to see that DDP is constantly sooner, round 6% larger throughput for the bottom mannequin and 4% for the massive one. That is anticipated since DDP has much less communication overhead. It simply has one all-reduce per step, towards FSDP’s gather-and-discard at each layer. And after we see the coaching time:
The takeaway right here is that when the mannequin matches in reminiscence, DDP is the higher alternative. It’s less complicated and sooner. However discover the reminiscence hole. What occurs after we scale as much as a mannequin the place that hole really issues?
Mistral-7B on Alpaca. For the second half, I attempted fine-tuning the Mistral mannequin on the Alpaca instruction-following dataset. With DDP, the job crashed instantly with an out-of-memory error. With FSDP, it ran with out a problem.
DDP would wish an estimated 87 GB per GPU for the mannequin state (parameters, gradients, optimizers), which exceeds the 80 GB out there on the A100. FSDP shards this throughout 4 GPUs, bringing the estimated requirement right down to about 22 GB per GPU. The precise peak through the run was about 29 GB, which incorporates activation and different overhead. This is the reason FSDP exists. It makes coaching doable when DDP merely can’t match the mannequin.
In order that’s the rule I used to cease at. Does the mannequin, with its gradients and optimizer states, match on one GPU? If sure, use DDP. If no, use FSDP. It’s a very good rule. However there are two assumptions hiding inside it. The primary is that DDP and FSDP are the one two settings. At first, I already hinted that they aren’t. There’s a dial between them, and within the subsequent half, we’ll talk about these phases, the place you shard a number of the mannequin state however not all of it.
The second assumption is that we’re treating all machines the identical method. These methods don’t behave the identical in all places. The identical technique on the identical mannequin can run a number of instances sooner or slower relying on how the GPUs are wired collectively. That’s what the second half of the article is about. First, although, the dial.
ZeRO: The Dial Between DDP and FSDP
To see the stops on the dial, begin with a small reality in regards to the operations from Half 1. DDP’s all-reduce, the one which averages gradients throughout GPUs, is definitely two less complicated operations glued collectively. First is reduce-scatter, which sums everybody’s gradients however provides every GPU solely its slice of the consequence as a substitute of giving everybody the total consequence. Then an all-gather, which collects these slices again right into a full copy on each GPU. So all-reduce is principally reduce-scatter plus all-gather.
These are the identical two operations that FSDP makes use of. Now that we all know that the operations are shared, it is smart that the quantity of sharding might be in phases. The dial is then only a query of how a lot you’re keen to maintain sharded between steps as a substitute of reassembling.
To make that concrete, take a look at what every GPU shops per parameter. In mixed-precision Adam, there are three issues: the parameters, the gradients, and the optimizer states, with the optimizer states being by far the heaviest. DDP retains full copies of all three on each GPU. The ZeRO phases, from DeepSpeed, peel them off separately, heaviest first.
ZeRO-1 shards solely the optimizer states. Every GPU nonetheless holds the total parameters and full gradients, but it surely retains solely its slice of the optimizer states. Since these are the heaviest gadgets, this offers the largest reminiscence win, and it prices little or no in additional communication, as a result of nothing you want through the ahead and backward cross has moved.
ZeRO-2 provides the gradients to the shards. Now every GPU retains solely its slice of the gradients and its slice of the optimizer states, however nonetheless a full copy of the parameters. Communication continues to be near DDP’s, for the reason that parameters (the factor you really compute with) are all nonetheless native.
ZeRO-3 provides the parameters as nicely. Now, no GPU holds the total mannequin, so earlier than a layer can run, its parameters need to be gathered on the fly, used, and discarded. ZeRO-3 and FSDP implement the identical fundamental thought. One is constructed by DeepSpeed, and the opposite is native to PyTorch. So, since that is the place you cease retaining the parameters native, additionally, you will have the additional communication we mentioned earlier.
The next diagram reveals the dial from full replication to full sharding:
Two issues within the determine are price pausing on. The primary is the reminiscence column on the correct. These are the measured peaks for Mistral-7B on 4 GPUs, and so they fall in a clear sample as you progress down the dial. 87 GB at DDP, then 55, 51, 40, and 37 GB. However discover it by no means drops to 1 / 4 of 87 GB, which is what you’d count on if 4 GPUs every held 1 / 4 of all the pieces. The ground is generally activations and runtime overhead. Activations are nonetheless produced independently on every GPU, so sharding the mannequin state doesn’t make them disappear.
Second, each step down that dial buys reminiscence by spending communication. ZeRO-1 spends virtually none. ZeRO-3 and FSDP spend probably the most. That’s the software program view. DDP, ZeRO, and FSDP are completely different solutions to the identical query: what ought to every GPU hold regionally, and what ought to it fetch from the others? Shifting towards sharding buys reminiscence, but it surely additionally creates extra communication.
Thus far, we have now solely counted how a lot knowledge has to maneuver. We have now not requested how costly that motion is. That relies on the {hardware}. Extra particularly, it relies on the material connecting the GPUs.
Half 2: The {Hardware}: How Quick the GPUs Can Discuss
The Wires Between the GPUs
Each technique in Half 1 had the identical hidden price line: communication. DDP pays it as soon as per step. FSDP pays it at each layer. ZeRO permits you to select how a lot to pay. However communication just isn’t an summary price. It’s knowledge shifting between chips, and the trail it takes issues.
On this article, I’m solely communication inside a single node. As soon as coaching spans a number of servers, one other layer seems. GPUs now have to speak throughout InfiniBand or Ethernet, which brings a special set of bottlenecks. That issues loads as nicely, but it surely’s a separate article.
Inside one node, when GPUs have to change knowledge, how do they do it? Properly, there are completely different roads they’ll take, and the pace between them can differ by as much as 10×. The identical FSDP code can run at full pace on one machine and crawl on one other with the very same GPUs, purely due to how these GPUs are related to one another.
There are two fundamental paths GPU site visitors can take inside a server: PCIe and NVLink. PCIe is the default system path by way of the motherboard. NVLink is NVIDIA’s direct GPU-to-GPU interconnect.
However NVLink is simply the hyperlink. The topology issues simply as a lot. On the machines I examined, NVLink appeared in two completely different preparations. NVL bridge teams and NVSwitch. That offers us 4 phrases to maintain straight. PCIe, NVLink, NVL, and NVSwitch. They’re associated, however they don’t seem to be the identical type of factor. PCIe and NVLink are the paths. NVL and NVSwitch are methods of arranging the quick NVLink path throughout a number of GPUs.
PCIe: the default connection
PCIe (PCI Categorical) is the usual high-speed bus that connects elements to a pc’s motherboard, the slots your graphics card, SSD, and community all plug into. Each GPU sits on it by default.
When two GPUs have to change knowledge over PCIe, it goes by way of the system: out of 1 GPU, throughout the shared bus, typically by way of the CPU and host reminiscence, and into the opposite. It really works between any two gadgets, but it surely’s the slowest possibility, and each system on the bus is contending for a similar lanes. A PCIe Gen5 x16 slot strikes someplace round 64 GB/s in a single route.
NVLink: a direct GPU-to-GPU connection
NVLink is NVIDIA’s devoted interconnect. Hyperlinks that run from one GPU to a different with out going by way of the CPU or system reminiscence. On an H100 or H200, every GPU has 18 NVLink connections, which collectively come to roughly 450 GB/s in every route, about 7x what a single PCIe slot gives.
However NVLink is simply the hyperlink. What issues simply as a lot is how these hyperlinks are organized throughout the GPUs in a server. Two machines can each have H200 GPUs and nonetheless behave very otherwise if one makes use of NVLink by way of bridges and the opposite makes use of NVSwitch.
NVL: hyperlinks inside teams, PCIe between them
The cheaper association connects NVLink solely inside small teams of GPUs, utilizing bodily bridges between playing cards. NVIDIA sells these as “NVL” playing cards, just like the H200 NVL. On the machines I examined, eight playing cards had been bridged into two teams of 4. Inside a gaggle, each pair of GPUs has a quick NVLink bridge. Between teams, there’s no NVLink in any respect, and site visitors falls again to PCIe.
So the material is uneven. Some pairs of GPUs discuss over NVLink, others discuss over PCIe. And so which pair you get relies on which bodily GPUs your job lands on, which is determined by the cluster scheduler. On an NVL machine, placement is vital as it could possibly have an effect on efficiency.
NVSwitch: each GPU related to each different
The premium association places a swap within the center. Devoted NVSwitch chips join each GPU to each different GPU at full NVLink pace, with no quick pairs and no gradual pairs. Each GPU is one hop away from each different. That is what’s on NVIDIA’s HGX baseboards, the SXM type issue, and it’s the format that enormous coaching runs use.
To get a way of how far this concept scales, at Computex 2025, Jensen Huang confirmed off what NVIDIA calls an NVLink Backbone, a column of round 5000 cables wiring 72 GPUs right into a single all-to-all cloth, and claimed it strikes about 130 TB/s. That is greater than the height site visitors of your complete web. Take the comparability as you’ll, however the underlying level is actual. An NVSwitch cloth is a crossbar the place each GPU reaches each different at full pace, and it scales to an entire rack. The machine I examined is a a lot smaller model of the identical thought. It has 8 GPUs as a substitute of the 72, however wired on the identical precept.
Similar GPU, two completely different machines
It’s price clearing up a confusion that may occur. After we say “H200,” we’re principally naming the GPU. H200 techniques will be constructed in several type elements and materials: as SXM GPUs on an NVSwitch baseboard, or as NVL playing cards related by bridges. Each are H200-class GPUs with 141 GB of reminiscence, however they don’t seem to be the identical system. Energy limits, board design, and cooling can differ. However the level right here is that you just can’t infer the GPU-to-GPU cloth from the GPU mannequin identify alone. It’s important to take a look at the topology. Listed here are the 2 machines used for the experiments beneath. You may examine the wiring in your node with nvidia-smi topo -m.
On the SXM node, each pair reads NV18. That is uniform, full pace, each GPU is equal. On the NVL node, the identical H200 playing cards cut up into two teams of 4, quick NVLink inside a gaggle and a plain PCIe throughout. A job that lands on 4 GPUs inside one group runs close to full pace. A job scattered throughout teams drops right down to PCIe. The subsequent part is about precisely what that prices.
Measuring the Wires
We have now two sorts of measurements right here. The primary is a microbenchmark that instances the precise collective operations the methods rely on. All-reduce for DDP, all-gather and reduce-scatter for FSDP, and experiences the bandwidth each achieves. The second is end-to-end coaching, the identical Mistral-7B fine-tune from Half 1, now timed for steady-state throughput, so we are able to see whether or not the bandwidth variations really present up in coaching. Every thing beneath ran on the 2 machines described above.
Let’s begin with the cleanest doable comparability. Take two H200 SXM GPUs and run the identical collectives twice. For the primary run, allow them to use NVLink, and for the second, pressure them onto PCIe as a substitute. Similar GPUs, similar code, solely the wire adjustments.
The wire alone is price about 10-11x. This is identical silicon and the identical operation, the place the one distinction was the bodily connection the info travels over.
Subsequent, I measured how all-reduce bandwidth adjustments as you add GPUs to the job. For this, I needed to management which bodily GPUs every run used, so I might hold a job inside a single NVL group or pressure it to cross between teams.
We are able to see three behaviors right here. NVSwitch stays flat and quick. 330 GB/s at two GPUs, rising gently to 467 at eight, and it by no means issues which bodily GPUs the job will get. Each pair is NV18, so the material appears to be like the identical regardless of how the scheduler fills it. That is the habits massive coaching runs assume.
The NVL quad is a shock, and it comes right down to how NVLink hyperlinks are shared. Every of those GPUs has 18 NVLink hyperlinks to distribute amongst its neighbors. Inside a quad of 4, each GPU has three neighbors and provides six hyperlinks to every one, which is the NV6 you’d see within the topology matrix. Six hyperlinks per pair, eighteen hyperlinks complete per GPU.
That sharing is why job measurement issues a lot. A job utilizing solely two GPUs of the quad talks throughout simply the six hyperlinks connecting that one pair. The opposite 12 hyperlinks on every card go to GPUs that aren’t within the job, so that they sit idle, and the pair runs at about 114 GB/s, which is a 3rd of the cardboard’s NVLink capability. Add the opposite two GPUs, and each card now talks to all three of its neighbors without delay, lighting up all 18 hyperlinks. That’s the identical variety of hyperlinks an SXM module makes use of, which is why a full quad reaches about 322 GB/s, practically matching NVSwitch.
So the NVL quad was by no means gradual. It was simply underused. A small job leaves most of its hyperlinks idle. After we fill the quad, we fill the hyperlinks.
The third line is the cross quad case, and it by no means leaves PCIe territory. 28 GB/s at two GPUs, 35 at 4, 39 at eight. Including GPUs doesn’t assist as a result of the issue isn’t what number of hyperlinks are lively; it’s that the job has to cross between quads, and there’s no NVLink there in any respect. A collective is simply as quick as its slowest required hop, so a single PCIe hyperlink between the quads drags your complete all-reduce right down to PCIe pace, regardless of how briskly each different hyperlink runs. And since a quad is simply 4 GPUs huge, any job bigger than 4 on this machine is pressured to cross. Eight GPUs can’t keep away from it.
The clearest view is the 4-GPU level on the determine. The identical 4 playing cards, in the identical node, run at 322 GB/s inside one quad and about 35 GB/s throughout two, a roughly 9× distinction determined by nothing however which GPUs the job landed on. On an NVL machine, placement actually does make a distinction.
What it does to actual coaching
The bandwidth numbers show the wire issues in a microbenchmark. I additionally needed to see if it reveals up in actual coaching. So I ran the identical Mistral-7B fine-tune from Half 1 on 4 GPUs, 3 ways: on NVSwitch, on a full NVL quad, and on an NVL job intentionally spanning two quads.
Take a look at the 2 quick materials first, NVSwitch and the total NVL quad. They’re practically an identical for each technique. The playing cards had been by no means the bottleneck, it was the position. (If FSDP appears to be like prefer it edges out DDP contained in the quad, don’t learn an excessive amount of into it. Repeat runs put the noise at a couple of %, so these are a tie. The trustworthy abstract is that, on both quick cloth, DDP and FSDP are even.)
The across-quads case is the price of getting placement mistaken. Throughput drops by 3 to 5x. That’s a smaller hit than ~10x bandwidth drop from determine 9, and the hole is price understanding. Coaching isn’t pure communication. Every GPU continues to be doing actual compute that doesn’t care in regards to the wire, and that compute partly hides the gradual communication behind it. So the wire’s full penalty will get considerably diluted. However 3 to 5x continues to be a really significant distinction, particularly for lengthy jobs.
And the slowdown isn’t even throughout the methods. The sample is precisely what Half 1 predicts. FSDP falls the toughest, as a result of it communicates each layer, and on a gradual wire, these per-layer transfers can’t cover behind compute. DDP holds up greatest as a result of it communicates simply as soon as per step. When the wire is quick, sharding is sort of free, so FSDP retains tempo with DDP. When the wire is gradual, each additional little bit of communication is uncovered, so the extra a method shards, the extra it loses.
Put all of it collectively. The entire dial, on three wires
Earlier, we mentioned that DDP and FSDP are two ends of 1 dial, with ZeRO-1, -2, and -3 because the stops between them. That was a diagram, and now I can measure it. Right here’s the identical Mistral-7B job on 4 GPUs, run at each cease on the dial, on a quick wire and on gradual ones.
Begin with the reminiscence panel on the correct as a result of it does precisely what the speculation promised. Shifting down the dial, peak reminiscence falls in a clear sample. 87 GB at DDP, then 55 at ZeRO-1, 51 at ZeRO-2, and 40 at ZeRO-3. Every cease shards another factor, and each buys headroom. FSDP sits on the backside at 37 GB, however discover it’s not a giant step down from ZeRO-3’s 40, as a result of FSDP and ZeRO-3 shard precisely the identical issues. As I discussed above, they shard the identical issues, however the frameworks implement and schedule issues otherwise, which explains the small hole that we see. This staircase is the half that at all times works, no matter {hardware}. In case your downside is becoming the mannequin, the dial delivers, predictably.
The throughput panel is fascinating as a result of the dial behaves like two various things relying on the wire.
On the quick wire, the center of the dial loses. ZeRO-1 and ZeRO-2 surrender round 20% of DDP’s throughput, and ZeRO-3 provides up extra. However FSDP, sharding the identical issues as ZeRO-3, claws practically all of that throughput again whereas utilizing about the identical reminiscence. That’s fascinating as a result of the final two are implementations of the an identical algorithm, and land far aside on a quick wire.
The distinction is implementation, not methodology. The 2 frameworks schedule and overlap their communication otherwise, and on a quick wire, the place communication is reasonable, that overhead is the primary factor you’re measuring. (That is Hugging Face speed up’s default DeepSpeed config, with no tuning. A tuned setup would seemingly slender the hole; the purpose is what the defaults provide you with, not that one framework is inherently slower.)
On a gradual wire, the dial flattens. Every thing collapses towards the identical low throughput, and the variations between phases principally wash out. The costly factor on a gradual wire is communication itself, and as soon as that dominates, the query of which precise tensors you shard barely strikes the quantity. On this case, the wire is setting the tempo, not the technique.
Placing all the pieces above collectively, we are able to now make a extra knowledgeable determination.
When you’re on NVSwitch (SXM), cease enthusiastic about the wire. Each GPU is equally quick, and any GPUs the scheduler provides you might be nearly as good as every other. Use DDP if the mannequin matches, FSDP if it doesn’t. And since FSDP matches DDP’s pace right here whereas utilizing far much less reminiscence, it’s an affordable default even when the mannequin does match.
When you’re on NVL and your job matches inside one bridged group, pin it there, and also you get near-NVSwitch pace. On the machines I examined, which means 4 GPUs, and also you management the position with CUDA_VISIBLE_DEVICES. FSDP is a powerful default on this case. It will get you virtually DDP-class pace at roughly half the reminiscence.
When you’re on NVL and your job has to span teams, count on a substantial slowdown, worst for the communication-heavy methods. Don’t count on a milder ZeRO stage to win the pace again both, as a result of the dial flattens on a gradual wire. If the mannequin matches replicated, plain DDP is the least-bad possibility. If it doesn’t, decide the stage that matches in reminiscence and simply settle for that you just’re going to pay the wire’s worth.
Conclusion
We’ve coated a number of floor. Within the first half, we regarded on the completely different methods. DDP, ZeRO, and FSDP. Every of them solutions the identical query another way: what ought to every GPU retailer regionally, and what ought to it fetch from the others? DDP retains all the pieces replicated. FSDP shards all the pieces. ZeRO provides you the stops in between.
Within the second half, we regarded on the {hardware} layer, the material these requests journey over. On a quick cloth, sharding will be low cost sufficient that FSDP retains tempo with DDP whereas utilizing a lot much less reminiscence. On a gradual cloth, the identical additional communication turns into seen, and may gradual issues down considerably.
That’s the major lesson. The technique decides how a lot knowledge strikes, however the wire decides how costly that motion is. So earlier than launching a distributed job, run one command:
nvidia-smi topo -m
It tells you whether or not your GPUs are related by NVLink, NVSwitch, or a slower host path.
Thanks for studying, and I hope you discovered this beneficial!
References and Additional Studying
