GPU Cluster Networking: What Operators Need to Know
Summary
- GPU cluster networking is the set of networks that join GPUs inside a server and across servers, plus storage and management networks. NVIDIA's DGX SuperPOD B200 reference design uses four.
- InfiniBand and tuned RoCEv2 Ethernet both carry distributed training. Ethernet took about two-thirds of AI back-end switch sales in Q1 2026, according to Dell'Oro Group.
- The network is part of the deployment. It belongs on the equipment schedule, gets tested before revenue starts, and its performance terms belong in the offtake contract.
Why GPU Cluster Networking Is a Deployment Decision
Dell'Oro Group reports that AI back-end switch sales passed front-end switch sales for the first time in Q2 2026. The back-end network links GPUs to each other; the front-end network connects servers to everything else.
A GPU server can be specified down to the last DIMM and still underperform if the fabric between servers is the wrong shape. For an operator, that means poorly scaling multi-node jobs, failed acceptance runs and a weaker position against offtake performance terms.
Below are the four networks, the fabric decision and topology, then how the network enters the bill of materials, acceptance testing and the offtake contract. The scope stops at the server and rack boundary.
How GPU Cluster Networking Works: Four Networks Around Every Server
GPU cluster networking is the set of interconnects that let GPUs inside a server and across servers work as one pool of compute. It also includes the separate networks that feed those GPUs data and let operators manage them. NVIDIA's reference design for B200 systems runs four of these networks side by side.
In distributed training, GPUs exchange results through collective operations such as AllReduce, AllGather and ReduceScatter. These run on NCCL, NVIDIA's library for GPU-to-GPU communication. The NCCL documentation defines AllReduce as a reduction across devices whose result lands on every rank, meaning every GPU process. That traffic runs sideways between servers, known as east-west traffic, and a slow link leaves every GPU in the job waiting.
Scale-up inside the server, scale-out across servers
The GPU generation you finance sets each GPU's link speed to the cluster: 400 Gb/s on B200 servers and 800 Gb/s on B300. Inside an 8-GPU HGX server, NVLink joins the eight GPUs so they work as one unit, which is called scale-up. Our guide to HGX vs DGX systems compares the two builds. Rack-scale systems, covered in NVL72 rack-scale design, stretch NVLink across a whole rack.
Across servers, each GPU reaches the compute fabric through a network adapter, which is called scale-out. As of September 2026, HGX B200 servers pair with ConnectX-7 adapters at 400 Gb/s, as listed on the NVIDIA B200 hardware page. HGX B300 servers pair with ConnectX-8 at 800 Gb/s per GPU, as the NVIDIA B300 reference page shows.
The 800 Gb/s figure needs one qualification. NVIDIA's ConnectX-8 user manual lists XDR InfiniBand at 4 × 200 Gb/s, or 800 Gb/s, and Ethernet at 400, 200 or 100 Gb/s per port. Size an Ethernet design on B300 around ports of up to 400 Gb/s, and confirm the configuration with your OEM.
Compute, storage and management fabrics
NVIDIA's DGX SuperPOD reference architecture for B200 systems, updated September 2, 2026, runs four separate networks. Each one is its own line on your equipment schedule, with its own switches, adapters and cabling.
- Compute fabric. Carries east-west GPU traffic for training and inference. NVIDIA specifies a rail-optimized, non-blocking, full fat-tree with eight 400G connections per system.
- Storage fabric. Moves training data between servers and storage, independent of the compute fabric. NVIDIA designs it slightly oversubscribed, at a ratio near 4:3.
- In-band management network. Carries cluster services such as the Slurm scheduler, cluster management software, NFS home directories and access to external registries.
- Out-of-band management network. Connects the management ports of every device, including servers, switches and storage. NVIDIA keeps it on its own fabric because users never need those ports.
Two terms in that design shape what you can sell. Rail-optimized means the same-numbered network connection on every server lands on the same leaf switch, called a rail. Traffic between GPUs on the same rail within a scalable unit is one switch hop, while traffic across rails goes through the spine. Non-blocking means the switches carry enough uplink capacity for every port to send at full speed at once.
Hyperscalers draw the same lines. In August 2024, Meta's engineering team described a dedicated back-end training network, separate from a front-end network for data ingest, checkpointing and logging.
InfiniBand Vs Ethernet: Choosing the Compute Fabric
Both fabrics run large training clusters: Meta ran most of its AI capacity on RoCEv2 as of August 2024, and NVIDIA's SuperPOD design uses InfiniBand. Neither is universally faster, so the choice turns on your offtake customer's requirements, your operations team and how well the fabric is tuned.
Both rely on RDMA (remote direct memory access), which lets one server write into another's memory without involving either CPU. InfiniBand is a dedicated fabric built around RDMA, with congestion handling designed into the network. RoCEv2 (RDMA over Converged Ethernet, version 2) carries the same RDMA traffic over Ethernet switches, which widens vendor choice and adds tuning work.
What the market is buying
Ethernet now takes most AI back-end switch spending, with more than two-thirds of AI back-end switch sales in full-year 2025, according to Dell'Oro. Its share held at about two-thirds in Q1 2026, while InfiniBand sales more than tripled in the same quarter. These are shares of switch sales; they don't count deployed clusters.
On the InfiniBand side, NVIDIA's Quantum-X800 switch offers 144 ports at 800 Gb/s each, per its product page as of August 2026. It adds SHARP v4, in-network computing that performs part of a collective such as AllReduce inside the switch, so less data crosses the fabric.
Ethernet now has an open standard for AI and HPC: the Ultra Ethernet Consortium published version 1.1 of its specification on October 12, 2026. Product support trails a specification, so check what your switch and adapter vendors ship.
NVIDIA also sells its own tuned Ethernet platform. In an October 2024 release, NVIDIA says Spectrum-X maintained 95% data throughput on xAI's Colossus cluster through its congestion control, against 60% for standard Ethernet. Those figures are NVIDIA's own claim about one customer's cluster, and they haven't been independently tested.
What operator benchmarks show
Published results show tuning and routing move performance as much as fabric choice. At Meta, one routing choice degraded training performance by up to more than 30%. None of the public results below is a like-for-like test of a neocloud cluster, so treat them as inputs to your own testing.
| Source and date | Scale | Test | Result | Caveat |
|---|---|---|---|---|
| Elice's benchmark (Apr 2026) | 16–96 GPUs, 8 GB messages | NCCL AllReduce | RoCEv2 (PFC/ECN configured) matched or beat InfiniBand, +0.7% to +2.8% | Topologies differed: InfiniBand leaf-spine, RoCEv2 single-tier chassis |
| Elice (Apr 2026) | 16–96 GPUs, 8 GB messages | NCCL AllGather | RoCEv2 ahead, +14.8% to +16.7% | Collective-library results; application throughput wasn't measured |
| Elice (Apr 2026) | 16–96 GPUs, 8 GB messages | NCCL AlltoAll | InfiniBand ahead, 0.9% to 4.6% | Same topology difference applies |
| Meta (Aug 2024) | Meta's RoCEv2 AI capacity | Routing with path pinning | Training performance degraded by up to more than 30% | One hyperscaler, with in-house co-design |
| Meta (Aug 2024) | Meta's RoCEv2 AI capacity | Load-balancing change | AllReduce improved by up to 40% | One hyperscaler, with in-house co-design |
Elice held link speed, software versions and tuning variables identical, but its topologies differed, so part of each gap may come from switch layout. Meta co-designs its collective library, scheduler and network, which few operators can match. In our view, a well-tuned fabric of either type outperforms a poorly tuned one, so tuning belongs in the build plan.
On Ethernet, that tuning centers on PFC and ECN. PFC (priority flow control) pauses traffic on a link before switch buffers overflow, and ECN (explicit congestion notification) marks packets so senders slow down early. Setting both correctly on every switch is the ongoing operational cost of RoCEv2.
Topology, Oversubscription and Scaling Past One Pod
Scaling losses on a well-built compute fabric stay small but real. In October 2026, E2E Networks published results for a 1,024-GPU HGX B200 cluster built to NVIDIA's reference architecture, with eight 400G InfiniBand rails per node. Joining two 256-GPU units into one 512-GPU job lowered the NCCL sweep average by 6.5%. Peak bandwidth stayed essentially flat, at 384.89 GB/s against about 385.5 GB/s for a single unit.
Per-GPU throughput on Nemotron-4 340B training fell 3.9% between 128 and 1,024 GPUs. That figure reflects the whole stack, so it can't be pinned on the network alone, and both numbers come from one operator's cluster.
Oversubscription means a switch has more bandwidth facing servers than facing the rest of the network, so traffic queues when every server sends at once. NVIDIA's SuperPOD design keeps the compute fabric non-blocking and accepts a ratio near 4:3 on storage. Meta runs its AI zones non-blocking, oversubscribes the links between zones by design, and offsets that with topology-aware scheduling.
Topology-aware scheduling places a job on GPUs that sit close together. Slurm's topology guide describes placing a job under the lowest-level switch that can hold it, which keeps its traffic off higher switch tiers. For capacity you sell as one cluster, scheduler configuration is part of delivering the contracted performance.
Two signs point to the network as the constraint: multi-node jobs scale worse than single-node jobs, or collective results vary between identical runs. Trace either one before a customer workload lands.
Where the Network Shows up in Your Deployment: Bill of Materials, Acceptance Testing and Offtake Terms
The network touches three documents in every deployment: the bill of materials, the acceptance test plan and the offtake contract. They settle what gets bought and financed, when the cluster is ready for revenue, and what you promise the customer.
The bill of materials
Networking is a large line in AI equipment budgets, so list adapters, switches, optics and cabling on the equipment schedule with the servers. 650 Group put networking at 22% of 2025 data center equipment spend for the top five US cloud providers. AI servers took 55%, storage 15% and traditional servers 8%.
Those shares describe hyperscaler equipment budgets, and no public figure gives the networking share of a neocloud cluster. Optics need their own check. In March 2026, LightCounting forecast that switch-ASIC and XPU shortages would hold 2026 transceiver growth to about 60%. Capacity additions were easing optical shortages.
Treat that as a forecast, and confirm optics lead times with the purchase order. Financing through USD.AI treats the switches, optics and cabling installed with the servers, along with owned racks and PDUs, as eligible hardware on the purchase order; soft costs (installation and labor, shipping, taxes, software subscriptions, data-center build-out) aren't financed.
Acceptance testing before revenue
NVIDIA's Cluster Readiness Engine, published September 2026, splits validation into four stages: bring-up, burn-in, preproduction and production. Its test catalog covers five NCCL communication tests, DCGM (NVIDIA Data Center GPU Manager) level-4 diagnostics and NeMo training runs. NVIDIA states that "no thresholds ship by default," so the operator defines pass/fail criteria for each deployment.
E2E Networks ran its acceptance tests on its 1,024-GPU cluster before any customer workload. That is one operator's practice; no public standard sets a burn-in duration or numeric checklist. Agree those criteria with your customer and write them into the deployment documents, so both sides share one definition of "ready."
Where financing holds capital in escrow until servers are installed, burned in and independently verified, as financing through USD.AI does, network validation sits on the funding path. Fabric problems found at burn-in affect the release of capital as well as the start of revenue.
Interconnect terms in the offtake contract
Customers compare your cluster against published specifications such as CoreWeave's B200 instance specification. It lists eight B200 GPUs per instance on a 400G NDR non-blocking Quantum-2 InfiniBand fabric. The page is a service specification without contract terms, but it shows the level of detail buyers expect.
We found no public offtake contract clause requiring a specific fabric type, so treat the table below as items to check. Lenders read these terms closely because the offtake contract repays the loan.
| Item | What to check | Why it matters |
|---|---|---|
| Fabric type and speed | InfiniBand or RoCEv2 Ethernet, and the per-GPU link speed | Sets what the customer can expect from multi-node jobs |
| Non-blocking or stated oversubscription | Whether the compute fabric is non-blocking, or the stated ratio | Defines the bandwidth you promise under full load |
| Cluster size for a single job | The largest GPU count one job can use | Scaling losses grow with job size |
| Acceptance test and thresholds | Which tests run and the agreed pass/fail numbers | Thresholds don't come with the tools and must be agreed per deployment |
| Remedies if performance falls short | What happens when the cluster misses its stated terms | Gives both sides a defined outcome for performance risk |
Planning the Network into Your Deployment and Financing
GPU cluster networking comes down to four networks, a compute fabric chosen for your customer and team, and a non-blocking design. The network belongs on the bill of materials, gets tested before revenue and is specified in the offtake contract.
GPU financing through USD.AI covers the networking installed with the servers as part of the eligible hardware. Compute IQ scores the operator, the customer and the offtake contract on a 100-point scale. Deployments sit in Tier III or Tier IV data centers under a colocation lien waiver, in which the data center waives its lien on the hardware.
Capital is held in escrow, and escrow releases the loan capital once the servers are installed, burned in and independently verified. The USD.AI loan history includes a deployment of 16 B200 servers in Texas in October 2026. If you're planning a deployment and want the network inside the financed equipment, Talk to USD.AI.
FAQ
What is the best network for a GPU cluster: InfiniBand or RoCEv2?
No single network is best for every GPU cluster, so choose a non-blocking compute fabric on InfiniBand or tuned RoCEv2 Ethernet. Match it to your customer's workload and the team that runs it, knowing InfiniBand needs less congestion tuning. In Elice's April 2026 tests on 16–96 GPUs, RoCEv2 with PFC/ECN configured matched or beat InfiniBand on AllReduce and AllGather, while InfiniBand led AlltoAll.
Can Ethernet handle AI training?
Yes. Meta reported in August 2024 that it runs RoCEv2 Ethernet for most of its AI capacity, with congestion control tuned to its workloads. Operators running Ethernet should budget for the same PFC and ECN tuning.
Does GPU financing cover networking equipment?
Yes. Through USD.AI, the switches, optics, cabling, owned racks and PDUs installed with the servers are eligible hardware; soft costs such as installation aren't financed. Loans cover up to 80% of the deployment cost, with the operator contributing at least 20% as equity.
