The Emerging Data Center Server Architecture

This article was originally published on 3/24/2023 and is one of the key/foundational articles being provided for reference.

This article is a follow-up to the earlier article “Major Refresh Is Coming” which discussed the current data center budget and power constraints and implications on hardware deployment.

A huge change has occurred in the computing world over the last few months with the arrival of ChatGPT. ChatGPT has captured the imagination of people with diverse needs and interests and this momentum is having and will have a great impact on data center architectures. This is because Generative Pre-Trained Transformer, or GPT, the technology behind ChatGPT, consumes large amounts of memory and computing resources. Not just GPT but many newly developing Machine Learning algorithms are set to use large models that consume memory and compute resources as we have never seen before.

Much of the software involved in processing this new genre of algorithms is “parallel code” or “vector code” and is ideally suited to be run on GPUs and accelerators as opposed to traditional CPUs. To be sure, CPU design companies like Intel (INTC), Advanced Micro Devices (AMD), ARM, and RISC-V have been racing to add instructions to make CPUs more adept at crunching this parallel code. As a result, a lot of “lightweight” parallel code, used in applications such as inference, is currently being run mostly on CPUs. However, with ChatGPT class algorithms, even inference becomes a relatively heavy-weight application for CPUs. For this class of applications, without a doubt, Nvidia’s (NVDA) GPUs have an edge and are now seen as de facto solutions. While we can expect CPU designers to give a stiff fight with their own newer generation products, right now the infrastructure inertia favors Nvidia.

There is one other subtle but important factor at play here: Intel, for nearly a decade now, has become a technology laggard and has not meaningfully improved the performance of its CPUs and, more importantly, their power efficiency. As a result, existing data centers are filled with power-inefficient Intel SkyLake-class products which are ripe to be replaced by superior solutions (some of this is covered in the “Major Refresh Is Coming” article).