When Compute Sits and Waits: Fixing the Hidden Storage Bottleneck in EDA and HPC with Azure NetApp Recordsdata

Howdy People!

You probably have ever stared at an HPC pipeline and puzzled why the queue depth retains climbing whereas each CPU graph seems lazy, this session from the Microsoft Azure Infrastructure Summit 2026 goes to really feel very acquainted. Ron Hogue from the Azure Storage staff and Ranga Sankar from NetApp spend twenty-seven minutes diagnosing the issue that no person desires to confess out loud, that the bottleneck isn’t compute, not the scheduler, and never licenses. It’s storage. After which they stroll by way of precisely how Azure NetApp Recordsdata (ANF) solves it.

📺 Watch the session:

If you happen to assist a semiconductor, simulation, rendering, or scientific computing staff, you’ve gotten lived this story. Engineers ask for extra cores. You purchase them. Wall-clock occasions barely transfer. Managers then ask for extra software licenses, as a result of absolutely that have to be the constraint. Spoiler, it normally isn’t.

Right here is why this issues to anybody operating shared infrastructure on Azure:

• EDA and HPC pipelines hammer storage with thousands and thousands of tiny reads and writes, metadata operations like file creation, rename, and unlink, all taking place concurrently throughout hundreds of processes.
• That sample breaks generic cloud file storage that was tuned for giant sequential I/O.
• ANF brings the identical NetApp ONTAP information providers that EDA groups have used on-premises for many years, delivered as a local Azure service.
• You’ll be able to transfer design environments to Azure with out re-architecting instruments, schedulers, or scratch path conventions.
• The economics modified within the final twelve months with the Versatile service stage and Cool Entry, so the previous “ANF is simply too costly for scratch” argument deserves a recent look.

Briefly, in case your job entails maintaining costly engineers and costly licenses busy, the storage layer deserves your consideration.

ANF is a first-party Azure service operating NetApp ONTAP on bare-metal infrastructure inside Azure datacenters. It speaks NFS v3, NFS v4.1, and SMB, with dual-protocol choices, and it preserves the file semantics that EDA instruments assume. Issues like POSIX permissions, quick metadata operations, snapshots, and constant low latency.

Ron and Ranga framed the session in eight layers (Drawback, Basis, Accelerator, Scale, AI Prepared, Guardrails, Optimize, Proof). Three items did a lot of the heavy lifting.

Basis, Migration Assistant with SnapMirror. That is the way you get the information into Azure with out rewriting your stock. SnapMirror replicates from on-premises ONTAP or Cloud Volumes ONTAP into ANF whereas preserving metadata, permissions, snapshots, and listing construction, with steady synchronization and minimal downtime. For EDA flows the place a lacking ACL can invalidate a whole run, that constancy isn’t non-compulsory.

Accelerator, Cache Volumes. Constructed on NetApp FlexCache, these volumes entrance your authoritative dataset (whether or not it lives in on-prem ONTAP or in Cloud Volumes ONTAP) and pull scorching reads near your Azure compute. Instrument libraries, PDKs, shared reference information, all served at sub-millisecond latency with out copying petabytes round. There’s nonetheless one supply of reality, which retains your information governance story clear.

Scale, Massive Volumes with Breakthrough mode. A single ANF massive quantity in Breakthrough mode scales to 2 PiB and delivers throughput within the tens of GiB per second by fanning I/O throughout six storage endpoints. That permits you to collapse sharded namespaces (the basic /proj1, /proj2, /proj3 break up that no person loves) into fewer high-throughput volumes that behave predictably beneath competition.

Cache Volumes are the FlexCache sample most ONTAP prospects already know. You peer a cluster, level a cache at an origin, and the cache propagates information on demand. Ron demoed this reside: cluster peering with on-prem ONTAP, a cache quantity created in ANF, and reads served from cache whereas the authoritative copy stayed residence.

Massive Volumes in Breakthrough mode are the place the structure will get attention-grabbing. As a substitute of a single mount level pinned to a single storage endpoint, Breakthrough mode exposes six storage endpoints for one logical quantity. Shoppers can mount, stability I/O, and mixture throughput throughout all six. Microsoft printed Linux scale-out benchmarks exhibiting a single 50 TiB massive quantity in Breakthrough mode sustaining roughly 50,000 MiB/s of sequential reads and approaching 1.8 million 8 KiB random learn IOPS utilizing twelve VMs (see the Microsoft Be taught hyperlink in Sources).

For shared environments, ANF added consumer and group quotas with real-time consumption reporting and exhausting limits. Ranga demoed quota guidelines that stopped a runaway simulation producing thousands and thousands of scratch recordsdata from ravenous the remainder of the staff. You probably have ever needed to ship the “who stuffed the scratch quantity” e-mail at 2am, this characteristic alone would possibly justify the journey.

On the associated fee aspect, the Versatile service stage decouples capability from throughput. You purchase a capability pool, you choose throughput independently, with 128 MiB/s of baseline throughput included and a ceiling of as much as 640 MiB/s per provisioned TiB (which is roughly 5 occasions the Extremely service stage). Cool Entry transparently tiers chilly blocks to Azure Blob behind the identical file mount level.

Briefly, you cease paying for throughput you don’t want on archival volumes, and also you cease overprovisioning capability to chase throughput on small scratch volumes.

The Proof part closed with SPEC Storage 2020 EDA Blended outcomes which can be price studying rigorously:

• A single massive quantity in Breakthrough mode sustained 2,880 EDA jobsets at about 0.51 ms general response time.
• Six volumes scaled linearly to 17,280 jobsets at about 0.60 ms.
• FIO measurements approached 2 million IOPS.

The sincere tradeoff: these numbers come from a benchmark, not your setting, and the SPEC tables present latency climbing on the highest load factors. So deal with the consequence as proof that the platform behaves predictably beneath EDA-style concurrency, not as a assure for each workflow. That predictability is the half that issues. EDA leads don’t lose sleep over peak throughput, they lose sleep over latency that drifts when concurrency rises.

Sensible situations the place this lands:

• Burst regression and verification runs to Azure throughout tape-out crunches, with Cache Volumes maintaining your on-prem golden tree authoritative.
• Full migration of EDA environments to Azure for groups whose datacenters are out of capability or out of lease.
• HPC simulation workloads (CFD, climate, seismic, life sciences) that share the identical metadata-heavy I/O profile.
• AI-adjacent pipelines that must learn design information with file semantics and floor the identical bytes as objects to Cloth, OneLake, or Databricks by way of the twin file and object entry sample Ron talked about within the AI Prepared part.

You don’t want a multi-quarter mission to get a helpful pilot transferring.

1. Register the Azure NetApp Recordsdata useful resource supplier in your goal subscription and request quota within the areas you care about.
2. Rise up a small capability pool, begin with the Versatile service stage so you possibly can dial throughput independently.
3. Create a check quantity, mount it from a consultant VM (HBv4 or Ev5 household are good beginning factors for HPC and EDA respectively).
4. Run your actual workload, not simply FIO. Use a consultant regression batch or simulation job and watch the metadata patterns within the ANF metrics blade.
5. You probably have on-premises ONTAP, peer a Cache Quantity towards it to check FlexCache conduct together with your precise datasets earlier than committing to a full migration.
6. Layer in consumer and group quotas earlier than you open the quantity to a wider staff. Belief me on this one.

Catch the complete Microsoft Azure Infra Summit 2026 session playlist right here.

Cheers!

Pierre Roman