you stroll throughout the campus of the Polytechnic College of Catalonia in Barcelona, you would possibly come across the Torre Girona chapel on a ravishing park. Constructed within the nineteenth century, it encompasses a large cross, excessive arches, and stained glass. However inside the principle corridor, encased in an unlimited illuminated glass field, sits a distinct form of structure.
That is the historic dwelling of MareNostrum. Whereas the unique 2004 racks stay on show within the chapel as a museum piece, the most recent iteration, MareNostrum V, one of many fifteen strongest supercomputers on this planet, spans a devoted, closely cooled facility proper subsequent door.
Most information scientists are used to spinning up a heavy EC2 occasion on AWS or using distributed frameworks like Spark or Ray. Excessive-Efficiency Computing (HPC) on the supercomputer degree is a distinct beast totally. It operates on totally different architectural guidelines, totally different schedulers, and a scale that’s tough to fathom till you utilize it.
I lately had the prospect to make use of MareNostrum V to generate large quantities of artificial information for a machine studying surrogate mannequin. What follows is a glance below the hood of a 200M€ machine: what it’s, why its structure appears the best way it does, and the way you really work together with it.
The Structure: Why You Ought to Care In regards to the Wiring
The psychological mannequin that causes essentially the most confusion when approaching HPC is that this: you aren’t renting time on a single, impossibly highly effective laptop. You might be submitting work to be distributed throughout 1000’s of unbiased computer systems that occur to share a particularly quick community.
Why ought to a knowledge scientist care concerning the bodily networking? As a result of in the event you’ve ever tried to coach a large neural community throughout a number of AWS cases and watched your costly GPUs idle whereas ready for a knowledge batch to switch, you realize that in distributed computing, the community is the pc.
To forestall bottlenecks, MareNostrum V makes use of an InfiniBand NDR200 cloth organized in a fat-tree topology. In an ordinary workplace community, as a number of computer systems attempt to speak throughout the identical most important swap, bandwidth will get congested. A fat-tree topology solves this by rising the bandwidth of the hyperlinks as you progress up the community hierarchy, actually making the “branches” thicker close to the “trunk.” This ensures non-blocking bandwidth: any of the 8,000 nodes can speak to another node at precisely the identical minimal latency.
The machine itself represents a joint funding from the EuroHPC Joint Enterprise, Spain, Portugal, and Turkey, break up into two most important computational partitions:
Common Objective Partition (GPP):
It’s designed for extremely parallel CPU duties. It accommodates 6,408 nodes, every packing 112 Intel Sapphire Rapids cores, with a mixed peak efficiency of 45.9 PFlops. That is the one you’ll be utilizing most frequently for the “basic” computing duties.
Accelerated Partition (ACC):
This one is extra specialised, designed with AI coaching, molecular dynamics and such in thoughts. It accommodates 1,120 nodes, every with 4 NVIDIA H100 SXM GPUs. Contemplating a single H100 retails for roughly $25,000, the GPU price alone exceeds $110 million.
The GPUs give it a a lot greater peak efficiency than that of the GPP, reaching as much as 260 PFlops.
There are additionally a particular kind of nodes known as the Login Nodes. These act because the entrance door to the supercomputer. Whenever you SSH into Mare Nostrum, that is the place you land. Login nodes are strictly for light-weight duties: shifting recordsdata, compiling code, and submitting job scripts to the scheduler. They don’t seem to be for computing.
Quantum Infrastructure: Classical nodes are now not the one {hardware} contained in the glass field. As of lately, Mare Nostrum 5 has been bodily and logically built-in with Spain’s first quantum computer systems. This features a digital gate-based quantum system and the newly acquired MareNostrum-Ona, a state-of-the-art quantum annealer primarily based on superconducting qubits. Relatively than changing the classical supercomputer, these quantum processing models (QPUs) act as extremely specialised accelerators.
When the supercomputer encounters fiercely advanced optimization issues or quantum chemistry simulations that might choke even the H100 GPUs, it will probably offload these particular calculations to the quantum {hardware}, creating a large hybrid classical-quantum computing powerhouse.
Airgaps, Quotas, and the Actuality of HPC
Understanding the {hardware} is barely half the battle. The operational guidelines of a supercomputer are totally totally different from a business cloud supplier. Mare Nostrum V is a shared public useful resource, which suggests the setting is closely restricted to make sure safety and truthful play.
The Airgap: One of many largest shocks for information scientists transitioning to HPC is the community restriction. You may entry the supercomputer from the surface world by way of SSH, however the compute nodes completely can not entry the surface world. There is no such thing as a outbound web connection. You can’t pip set up a lacking library, wget a dataset, or hook up with an exterior HuggingFace repository as you see match. The whole lot your script wants should be pre-downloaded, compiled, and sitting in your storage listing earlier than you submit your job.
In actuality, it’s much less of a problem than it seems, because the Marenostrum directors present a lot of the libraries and software program it’s possible you’ll want by way of a module system.
Shifting Information: Due to this strict boundary, information ingress and egress occur by way of scp or rsync by the login nodes. You push your uncooked datasets in over SSH, look forward to the compute nodes to chew by the simulations, and pull the processed tensors again out to your native machine. One shocking side of this restriction is that, because the precise computation will be so extremely quick, the bottleneck turns into extracting the completed outcomes to your native machine for postprocessing and visualization.
Limits and Quotas: You can’t merely launch a thousand jobs and monopolize the machine. Your undertaking is assigned a selected CPU-hour finances. Moreover, there are exhausting limits on what number of concurrent jobs a single person can have operating or queuing at any given time.
You have to additionally specify a strict wall-time restrict for each single job you submit. Supercomputers don’t tolerate loitering, in the event you request two hours of compute time and your script wants two hours and one second, the scheduler will ruthlessly kill your course of mid-calculation to make room for the following researcher.
Logging within the Darkish: Since you submit these jobs to a scheduler and stroll away, there isn’t a dwell terminal output to stare at. As an alternative, all normal output (stdout) and normal error (stderr) are robotically redirected into log recordsdata (e.g., sim_12345.out and sim_12345.err). When your job completes, or if it crashes in a single day, it’s a must to comb by these generated textual content recordsdata to confirm the outcomes or debug your code. You do, nonetheless, have instruments to observe the standing of your submitted jobs, equivalent to squeue or doing the basic tail -f on the log recordsdata.
Understanding SLURM Workload Supervisor
Whenever you lastly get your analysis allocation accepted and log into MareNostrum V by way of SSH, your reward is… a totally normal Linux terminal immediate.
After months of writing proposals for entry to a 200M€ machine, it’s, frankly, a bit underwhelming. There are not any flashing lights, no holographic progress bars, nothing to sign simply how highly effective the engine behind the wheel is.
As a result of 1000’s of researchers are utilizing the machine concurrently, you can not simply execute a heavy python or C++ script instantly within the terminal. If you happen to do, it’s going to run on the “login node,” rapidly grinding it to a halt for everybody else and incomes you an extremely well mannered however quite agency and offended e mail from the system directors.
As an alternative, HPC depends on a workload supervisor known as SLURM. You write a bash script detailing precisely what {hardware} you want, what software program environments to load, and what code to execute. SLURM places your job in a queue, finds the {hardware} when it turns into out there, executes your code, and releases the nodes.
SLURM stands for Simple Linux Utility for Resource Management, and it’s a free and open supply software program that handles job-scheduling in lots of laptop clusters and supercomputers.
Earlier than a posh pipeline, it’s good to perceive learn how to talk with the scheduler. That is executed utilizing #SBATCH directives positioned on the prime of your submission script. These directives act as your purchasing checklist for sources:
--nodes: The variety of distinct bodily machines you want.--ntasks: The overall variety of separate MPI processes (duties) you wish to spawn. SLURM handles distributing these duties throughout your requested nodes.--time: The strict wall-clock time restrict on your job. Supercomputers don’t tolerate loitering; in case your script runs even one second over this restrict, SLURM ruthlessly kills the job.--account: The precise undertaking ID that might be billed on your CPU-hours.--qos: The “High quality of Service” or particular queue you might be concentrating on. As an example, utilizing a debug queue grants quicker entry however limits you to brief runtimes for testing.
A Sensible Instance: Orchestrating an OpenFOAM Sweep
To floor this in actuality, right here is how I really used the machine. I used to be constructing an ML surrogate mannequin to foretell aerodynamic downforce, which required ground-truth information from 50 high-fidelity computational fluid dynamics (CFD) simulations throughout 50 totally different 3D meshes.
Right here is the precise SLURM job script for a single OpenFOAM CFD case on the Common Objective Partition:
#!/bin/bash
#SBATCH --job-name=cfd_sweep
#SBATCH --output=logs/sim_percentj.out
#SBATCH --error=logs/sim_percentj.err
#SBATCH --qos=gp_debug
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=6
#SBATCH --account=nct_293
module purge
module load OpenFOAM/11-foss-2023a
supply $FOAM_BASH
# MPI launchers deal with core mapping robotically
srun --mpi=pmix surfaceFeatureExtract
srun --mpi=pmix blockMesh
srun --mpi=pmix decomposePar -force
srun --mpi=pmix snappyHexMesh -parallel -overwrite
srun --mpi=pmix potentialFoam -parallel
srun --mpi=pmix simpleFoam -parallel
srun --mpi=pmix reconstructPar
Relatively than manually submitting this 50 instances and flooding the scheduler, I used SLURM dependencies to chain every job behind the earlier one. This creates a clear, automated information pipeline:
#!/bin/bash
PREV_JOB_ID=""
for CASE_DIR in instances/case_*; do
cd $CASE_DIR
if [ -z "$PREV_JOB_ID" ]; then
OUT=$(sbatch run_all.sh)
else
OUT=$(sbatch --dependency=afterany:$PREV_JOB_ID run_all.sh)
fi
PREV_JOB_ID=$(echo $OUT | awk '{print $4}')
cd ../..
executed
This orchestrator drops a sequence of fifty jobs into the queue in seconds. I walked away, and by the following morning, my 50 aerodynamic evaluations had been processed, logged, and able to be formatted into tensors for ML coaching.
Parallelism Limits: Amdahl’s Regulation
A standard query from newcomers is: In case you have 112 cores per node, why did you solely request 6 duties (ntasks=6) on your CFD simulation?
The reply is Amdahl’s Regulation. Each program has a serial fraction that can’t be parallelized. It explicitly states that the theoretical speedup of executing a program throughout a number of processors is strictly restricted by the fraction of the code that should be executed serially. It’s a really intuitive regulation and, mathematically, it’s expressed as:
[
S=frac{1}{(1-p)+frac{p}{N}}
]
The place S is the general speedup, p is the proportion of the code that may be parallelized, 1−p is the strictly serial fraction, and N is the variety of processing cores.
Due to that (1−p) time period within the denominator, you face an insurmountable ceiling. If simply 5% of your program is basically sequential, the utmost theoretical speedup you’ll be able to obtain, even in the event you use each single core in MareNostrum V, is 20x.
Moreover, dividing a activity throughout too many cores will increase the communication overhead over that InfiniBand community we mentioned earlier. If the cores spend extra time passing boundary circumstances to one another than doing precise math, including extra {hardware} slows this system down.
As proven on this determine, when simulating a small system (N=100), runtime will increase after 16 threads. Solely at large scales (N=10k+) does the {hardware} turn into totally productive. Writing code for a supercomputer is an train in managing this compute-to-communication ratio.
The Entry to the Immediate
Regardless of the staggering price of the {hardware}, entry to MareNostrum V is free for researchers, as compute time is handled as a publicly funded scientific useful resource.
In case you are affiliated with a Spanish establishment, you’ll be able to apply by the Spanish Supercomputing Community (RES). For researchers throughout the remainder of Europe, the EuroHPC Joint Enterprise runs common entry calls. Their “Improvement Entry” observe is particularly designed for tasks porting code or benchmarking ML fashions, making it extremely accessible for information scientists.
Whenever you sit at your desk watching that utterly unremarkable SSH immediate, it’s straightforward to overlook what you might be really . What that blinking cursor doesn’t present is the 8,000 nodes it connects to, the fat-tree cloth routing messages between them at 200 Gb/s, or the scheduler coordinating lots of of concurrent jobs from researchers throughout six international locations.
The “single highly effective laptop” image persists in our heads as a result of it’s easier. However the distributed actuality is what makes fashionable computing potential, and it’s rather more accessible than most individuals understand.
References
[1] Barcelona Supercomputing Heart, MareNostrum 5 Technical Specs (2024), BSC Press Room. https://towardsdatascience.com/what-it-actually-takes-to-run-code-on-200me-supercomputer/
[2] EuroHPC Joint Enterprise, MareNostrum 5 Inauguration Particulars (2023), EuroHPC JU. [link]
