AMD, Arista, Broadcom, Marvell, and Nvidia: The Future Of Scale-Up and Scale-Out Networks

When Advanced Micro Devices (AMD) launched its MI350 series accelerators recently, lack of clarity on Scale-Up and Scale-Out networking became a major point of focus for some investors. This is because rack scale systems have increasingly become the point of contrast between Nvidia (NVDA) and AMD as both companies field comparable GPUs with one company leapfrogging the other starting with AMD’s MI300 generation.

To wit, H100 reigned supreme since its launch to late 2023 when MI300 was launched. MI300, launched in December 2023 had better compute than H100; H200 launched in Q2 2024 surpassed MI300; MI325 launched in Q4 2024 surpassed H200; B200 launched in Q4 2024 surpassed MI325; and now MI355 launched in Q2 2025 surpassed B200; it now appears that B300 to be launched in Q3 2025 will split the honors with MI355 with H300 excelling in FP4 and MI355 excelling in FP6 and both chips being comparable when it comes to other data types.

With Nvidia and AMD GPUs having similar compute performance, how is that Nvidia has 90%+ of the commercial GPU market and AMD less than 10%? The answer is mainly two-fold. One is that the AMD software stack has been limited to hyperscaler inference whereas the Nvidia software stack is much more mature and enables not only hyperscaler and enterprise inference and training but many applications beyond that. The second, and the main topic of this article, is that AMD solutions have lacked the scale-up capabilities of Nvidia solutions. And this is the reason why there has been such investor focus on scale-up and scale-out networking.

A Primer on Scale-Up Vs Scale-Out

While most investors are aware of scale-up and scale-out networks, there seems to be uncertainty of what these terms mean and how they relate to technologies like NV Link, UALink, Infinity Fabric, Scale-out Ethernet, InfiniBand, etc. Recently, Broadcom (AVGO) complicated the issue further by first signing up to UALink consortium and then moving on to promote its own Scale-Up Ethernet. This section clarifies scale-up and scale-out and where each of these technologies play a role.

Let us take a step back and look at the concepts of Back-end and Front-end in AI networks. As can be seen from the image below from Arista Networks (ANET), the front-end network connects CPUs, storage, local area networks, and other elements of the data center. Given the universal nature of connectivity, Ethernet has been the de facto choice for this network. In an AI system, this network is used by CPUs to feed data required inference or training to the GPUs. Once fed, the GPUs go on to run the models to deliver the results. This network typically runs streaming RDMA data and traditional TCP data and tends to be less performance critical than the back-end network. The less intensive nature of the data movement means that these networks tend to be provisioned by “slower” 100G and 200G Ethernet ports (although the market is starting to move to faster ports).

The front-end network is dominated by Arista, Celestica, and Cisco (CSCO) with Broadcom and Marvell (MRVL) getting the lion share of the chip and NIC business. In the near term, the switch side of the business is unlikely to change much but AMD and Nvidia are likely to grab a chunk of the NIC business.

There will not be much additional discussion on front-end networks as this article focuses on the back-end network which is the primary growth driver.

The backend network connects the training or inference cluster of GPUs. This is where the GPUs talk to each other using very high high-speed RDMA streams. Low latency is also critical as many GPUs need to work together in a coordinated way to solve complex distributed computing problems – especially for training. The data bandwidth needs of this network have been exploding as GPU capabilities grow rapidly. These networks, which are mostly at 400G, are migrating to 800G (more on this later).

The back-end network is further subdivided into two parts – scale-up and scale-out.

Scale-up Networks

Scale-up network is typically a very high bandwidth network and is confined to a single rack. Within scale-up networks, GPUs are connected to each other with high-speed copper connections which enable each GPU to access the other GPUs and its resources with very low latency. There is a reason why these networks are restricted to a single rack. It is not possible to extend high speed chip-to-chip electrical connections using copper traces or wires beyond a rack. It is an engineering challenge to even make these connections work within a rack. Beyond the rack boundary, such high bandwidth connections are not possible without added latency and dramatically higher expense.

There are currently two competing methods that enable this scale-up connectivity – Nvidia’s NVLink and AMD’s Infinity Fabric. Until the Hopper generation, a typical scale-up network comprised of 8 GPUs (this is the reason we see the nomenclature x8 or 8x when people refer to H100/H200/MI300/325 systems). Some also refer to this as the “world size”. A world size of 8 implies that 8 accelerators are connected using extremely high bandwidth connections.

Nvidia was able to successfully increase the scale-up networking size to 72 GPUs with the Blackwell generation. Growing the world size from 8 to 72 CPUs results in significant performance benefits within the cluster and this has put Nvidia Blackwell generation in a class of its own. AMD’s MI355 is terrific product and beats Nvidia Blackwell on many metrics, but the lack of scaleup beyond x8 means that GB200 will be the highly preferred option for frontier model inference for hyperscalers and B200/GB200 will be the primary choices for frontier model training (MI355 will be highly attractive for hyperscaler non-frontier workloads and non-hyperscalers who typically do not deploy rack-scale systems).

AMD is aware of the scale-up challenge and has been working to improve scale-up with its Universal Accelerator Link, or UALink. Readers can refer to past Beyond The Hype articles for more detail on UALink and its industry implications. As can be seen from the image below, UALink and NVLink are comparable in terms of speeds and latency and are likely to have similar performance level in comparable systems. The main differences are that UALink, being more of a new clean slate approach, has more flexibility and scales to more nodes (1024 vs 576).

While the UALink specification is attractive, to enable a rack-level scale-up solution, UALink end points and switches should be readily available. AMD and other accelerator providers can enable end points by themselves but need a competent switch provider to form a complete system solution. Broadcom was an early supporter of UALink specification and was expected to field a UALink switch but has since decided to walk away from the effort (or at least deemphasized it). Astera Labs (ALAB) and Marvell have announced their interest, but it does not appear that either will have a switch solution until 2026 at the earliest. It is possible that AMD has its own UALink switch in the works, but the company has not made any such announcements.

Sensing the challenges in the UALink development, Nvidia has tried to redouble its efforts to attract the industry towards its proprietary NVLink Fusion solution but the effort has not gained much traction.

Recently, Broadcom announced that it is planning to offer its own scale-up option called Scale-up Ethernet or SUE. Broadcom considers SUE a good solution and deems it unnecessary to have proprietary options for scale-up.