The Wiring Behind Modern AI - (HPC)
When people talk about the compute powering large AI models, the conversation almost always lands on GPUs. How many chips a company owns, how fast each one runs, how much memory it carries.
That focus makes sense on the surface, but it hides where a lot of the real engineering effort has moved in the last two years: the connections between those chips.
A single GPU, no matter how powerful, cannot train a frontier model alone. Modern training runs stretch across tens of thousands of accelerators working on the same task at once, constantly exchanging data as they go. The system that links all of them together, carrying information between chips at speeds most people never think about, is what engineers call a high-performance compute fabric. It has quietly become one of the biggest bottlenecks, and one of the biggest opportunities, in AI infrastructure.
Training a large model is not one GPU doing one long calculation. It is thousands of GPUs each doing a small piece of the calculation, then comparing notes with every other GPU before moving to the next step. That comparison step, often called an all-reduce operation, happens constantly throughout training. If the connections between chips are slow, the GPUs sit idle waiting for data instead of computing, and all that expensive silicon goes to waste.
Industry researchers have started saying it plainly: cluster performance is now set by the interconnect rather than the processor. A rack full of the fastest accelerators on the market will still underperform if the fabric linking them cannot keep pace. This is the reason so much recent investment has gone into networking rather than chip design alone.
A compute fabric is the combined hardware and software layer that moves data between accelerators, storage, and switches inside a cluster. It includes the physical cabling, whether copper or optical, the switches that route traffic, and the protocols that decide how data gets packaged and prioritized.
There are two connection types worth separating. Scale up refers to the tight, extremely fast links between GPUs sitting inside the same server, often using proprietary technology like NVIDIA's NVLink. Scale out refers to the links between servers and racks, which typically rely on either InfiniBand or Ethernet, carrying traffic across the wider cluster.
For years, InfiniBand was the default choice for scale out networking because of its low latency and mature software stack. That is shifting. Ethernet has been catching up fast, helped by new specifications built specifically for AI workloads rather than general web traffic. The appeal is straightforward: Ethernet hardware is cheaper, more widely supported, and easier to source at the volumes hyperscalers now need.
For a long time, copper cabling was good enough. It is cheap, reliable, and simple to manufacture. But as data rates climb past roughly 25 gigabits per second per lane, copper starts fighting physics. Signal loss, reflections, and interference between lanes all get worse, forcing engineers to burn extra power on equalization just to keep the signal readable.
This is why data centers have been steadily shifting toward optical connections, first for the longer links between rows of racks, and now increasingly for shorter distances within a single rack. Fiber does not suffer the same attenuation problems as copper at these speeds, but it brings its own costs and complexity, particularly around the transceivers needed to convert electrical signals into light and back again.
One of the more interesting shifts happening right now is the move toward co-packaged optics, where the optical components sit physically next to the switch chip instead of in a separate pluggable module. This cuts the distance the electrical signal has to travel before conversion, which reduces both power draw and heat.
Given how much electricity a large AI cluster already consumes, even small efficiency gains at the interconnect level add up quickly across a data center running continuously. Several vendors have introduced high radix optical switching platforms this year that skip electrical switching layers entirely, using fiber level reconfiguration instead. The pitch is simple: less conversion, less heat, less wasted power, and more flexibility to reshape network topology as cluster demands change.
There is also early work on more unconventional approaches. Some companies are experimenting with dense arrays of low power optical channels instead of a handful of very fast ones, betting that spreading bandwidth across many simple channels can beat the diminishing returns of pushing a few channels ever faster. It is too early to say which approach wins, but it shows how much room is still left for architectural experimentation in this space.
Physical hardware only solves half the problem. The other half is topology, meaning the actual layout of how switches and nodes connect to one another.
A common design in modern AI clusters is the leaf spine layout. GPU nodes connect into leaf switches at the rack level, and those leaf switches then connect up to spine switches that tie the whole cluster together. This structure keeps the number of hops between any two GPUs relatively low and predictable, which matters enormously for workloads where thousands of chips need to synchronize on a tight schedule.
Other cluster designs use different topologies, such as fat tree or dragonfly layouts, each with its own tradeoffs around cost, latency, and how gracefully the network handles failures or congestion. Choosing the right topology is as much a business decision as an engineering one, since it directly affects how much a company spends per unit of usable bandwidth.
Here is something that rarely makes headlines but keeps infrastructure teams up at night: at the scale modern clusters operate, failures are not rare edge cases, they are a daily certainty. With tens of thousands of links running constantly, something is always degrading, disconnecting, or behaving unpredictably.
This has pushed reliability engineering to the center of fabric design rather than treating it as an afterthought. Systems now need to detect a failing link, reroute traffic around it, and keep a training run going without forcing an expensive restart from the last checkpoint. Congestion control matters just as much. When thousands of GPUs try to synchronize at once, uneven traffic loads can create bottlenecks that ripple through the entire cluster, and software has to actively balance the load to prevent one overloaded link from stalling everyone else.
The interconnect layer has quietly become one of the clearest competitive advantages a company can build in AI infrastructure. Two firms can buy the exact same GPUs from the same supplier and end up with very different real world training speed, simply because one built a better fabric around those chips.
This explains why chipmakers, networking companies, and cloud providers are all racing to own more of this layer rather than leaving it to commodity suppliers. It also explains why so many announcements this year, from new switching platforms to fresh interconnect standards, have come from companies most people outside the industry have never heard of. The GPU gets the headline. The fabric decides whether that GPU actually delivers on its promise.
The direction is fairly clear even if the exact winners are not. Expect optical technology to keep pushing further into the cluster, replacing copper in more places as speeds climb. Expect power efficiency to matter as much as raw bandwidth, since cooling and electricity costs are becoming a real constraint on how large a single cluster can practically grow. And expect more experimentation with unconventional approaches, since the industry has clearly accepted that scaling accelerator count alone is not enough if the wiring between them cannot keep up.
For anyone tracking the AI industry, the compute fabric is worth watching just as closely as the chips themselves. It is less visible, rarely gets its own keynote slide, and yet it increasingly decides who wins the race to train the next generation of models.
