Gaudi 3 Likely To Deliver Nvidia H100 Class Performance At ASP Below $10K By Q4 2024

Intel (INTC) announced its long awaited Gaudi 3 accelerators on Tuesday. Gaudi 3 is a Nvidia (NVDA) Hopper H100 and Advanced Micro Device (AMD) Instinct MI300 class accelerator. With Nvidia and AMD duking it out in the merchant silicon market and Broadcom (AVG), Marvell (MRVL), Alchip, Microsoft (MSFT) and others building proprietary solutions, the AI accelerator market has now become crowded. New players would have to offer considerable and sustainable advantages to win share from the incumbents – an extremely difficult task. With the investment necessary to build a competitive hardware and software solution climbing in billions of dollars, it will be increasingly difficult for new players to enter the merchant AI accelerator market and Gaudi 3 could very well be Intel’s last opportunity to enter the hot AI accelerator space.

With much at stake, and with a long recent history of poor execution, Intel needed to deliver a meaningful entry. This article discussed how Intel seems to have a respectable entry with Gaudi 3 but will have to bomb the pricing to make meaningful inroads.

Gaudi 3’s First Pass Review Is Decidedly Mixed

Gaudi’s specs are respectable although the use of HBM2e is a bit shocking. Given that Nvidia and AMD have already adopted HBM3 and are shipping products with that technology, Intel’s HBM2 spec suggests significant design methodology or execution issues. Without a doubt, Gaudi 3 would be a stronger product with HMB3 or HBM3e.

As with AMD and Nvidia, Gaudi comes in 3 variants – a basic PCIe form factor for enterprise or standard server use, in OAM form factor that is popular with data centers, and an 8-unit server base board that is a building unit for scaleup and scale out servers (image below).

Gaudi 3 being a third-generation accelerator means that Intel understands the market well and among other things understands the need for Gaudi 3 to scale. And Gaudi 3 is built to scale. Note is that Gaui3 uniquely sports 24 built-in 200Gb Ethernet ports. These ports can be used for scale-up or scale-out applications and give Intel the flexibility to build large Gaudi clusters.

OSFP in the image above stands for Octal Small Form-factor Pluggable and can be used to plug into copper or optical connectors as needed. Note that each Gaudi 3 connects to OSFPs using 3 200 Gb/s Ethernet links. So, what happens to the other 21 Ethernet links? (24 total)

The answer is that each Gaudi 3 connects to 7 of its neighbor Gaudi 3s on the 8-accelerator base board with 3 Ethernet links each. With 3 links per connection, there will be 600Gbps of connectivity between each Gaudi 3. These 75 GBps connections enable the 8 accelerators to work together to solve inference and training problems. While 75 GBps Ethernet connectivity between Gaudi 3’s may sound like a lot, this pales in comparison to AMD MI300X’s 128GBps Infinity Fabric Link and 128GBps Nvidia H100 Nvlink. (Note that Nvlink goes up to 256GBps for the Blackwell generation).

Intel uses standard Ethernet because the Company does not have a connectivity solution comparable to Infinity Fabric and Nvlink but Intel is a bit disadvantaged here. 75GBps is not only slow but Ethernet incurs higher latency than the competitive alternatives. What this means is that the Gaudi 3 performance will drop compared to Nvidia and AMD solutions as the system sizes get larger. Nevertheless, Ethernet is likely to be a good enough a solution for Gaudi 3 for several applications and will let Intel scale Gaudi 3 to large clusters (image below).

The scaling past the 8 Gaudi 3 baseboard can be seen in the images below.

The last image above shows how Gaudi 3 can scale to a 1024 node cluster offering 15 EF of FP8 compute, 1 PB of memory capacity, and 1,229 PB/s of networking bandwidth.