some of the problems we face in implementing AI algorithms, we usually focus on the processors’ ability to handle them.
But now, NVIDIA’s GPUs power the latest large language models, and companies compete to build faster AI accelerators. These new chips promise more computational power, more cores, and more operations per second.
So, one would assume that the future of AI depends on building increasingly powerful processors! But there is a problem. Many of today’s AI systems are not limited by how quickly they can perform calculations. They are limited by how quickly they can access data. Data that is essential for them to perform their calculations.
In other words, the future of AI may depend less on computation and more on memory.
I know that may not seem very intuitive, but let’s take a step back and imagine something: imagine hiring the world’s most efficient and fastest chef.
This chef can prepare meals at incredible speed. However, every ingredient is stored in a warehouse several miles away. Before the chef can cook, someone needs to grab the ingredients and deliver them to the kitchen.
No matter how talented the chef is, there will be periods when they simply stand around, waiting for the ingredients they need to start cooking to arrive.
Modern AI systems face a similar challenge. The processors they use can perform calculations, but they cannot operate on data that has not yet arrived. If the processor can compute faster than memory can deliver information, performance becomes limited by data movement rather than computation.
This is what is known in computer science as the memory bottleneck. It is one of the most important and least discussed challenges in modern AI.
The Scale of Modern Models
To better understand why memory has become such a significant issue, we need to consider the size of today’s AI models. Early machine learning models might have contained thousands or millions of parameters, while modern foundation models contain billions or even trillions.
Each one of these parameters represents a numerical value that must be stored in memory and repeatedly accessed during training and inference.
Okay, that sounds logical, but let’s solidify it with some numbers! Suppose we have a model that contains 70 billion parameters. Before even performing a single calculation, the system needs a place to store all those parameters.
Now, I want you to imagine thousands of users interacting with the model simultaneously. The hardware must continuously move enormous amounts of information between memory and processors.
As you may conclude, the challenge is no longer simply performing calculations; rather, it is feeding data to the hardware quickly enough.
Moving data can be more expensive than computing on it, which is one of the most counterintuitive realities in computing. Over the decades, processor performance has improved dramatically, as engineers have become exceptionally good at designing chips that can perform calculations faster and faster. On the other hand, though, improvements of the memory system moved at a slower pace.
This created a growing imbalance that became more apparent as AI systems grew bigger. Modern processors can execute trillions of operations per second, but they often spend a lot of time waiting for data to arrive.
This data bottleneck appears in different ways within AI systems. We can see it while moving data between memory and processors, between GPUs, across servers, or between data centers.
These models will only continue to grow, and as they do, the data movement issue will play a major role in the system’s overall performance!
Understanding AI Memory
So far, I think I’ve written the word “memory” over 20 times (I didn’t count!). You might have wondered, what kind of memory is she talking about? Most people are familiar with RAM, the memory installed in laptops and desktop computers. AI systems use different types of memory for different purposes.
1- RAM: Random Access Memory stores data used by the CPU. It is relatively large but not particularly fast compared to specialized AI memory.
2- VRAM: Graphics Processing Units contain dedicated memory known as Video Random Access Memory (VRAM). Which is used during training and inference to store model parameters, training batches, activations, and intermediate calculations. The amount of available VRAM often determines whether a model can fit on a GPU.
3- High-Bandwidth Memory (HBM): Modern AI accelerators increasingly rely on High-Bandwidth Memory (HBM). HBM is designed specifically to move large amounts of data extremely quickly. Rather than simply increasing memory capacity, HBM focuses on increasing memory bandwidth, which is the rate at which information can be transferred.
Though the size of the memory (its capacity) is important, its bandwidth plays a bigger role. One way to think about those two concepts is a highway (stay with me here for a second). Capacity is the number of cars that can exist on the highway, while bandwidth is the number of lanes available.
You can have a huge parking lot, but if all vehicles must leave through a single lane, traffic becomes the limiting factor.
Now, as we said before, AI systems use different types of memory in different ways. The memory challenge appears differently during training and inference.
Training: Training requires storing model parameters, gradients, activations, and optimizer states. So, as a result, memory requirements become enormous. This would require distributing memory across many GPUs.
Inference: Inference generally requires less memory than training, but it introduces a different challenge. The model must continuously serve requests while retrieving parameters and generating outputs quickly. For interactive systems such as chatbots, latency is now a major issue!
The faster memory can deliver information, the faster the model can respond. This is one reason why memory technologies remain essential even after training is complete.
Some Final Thoughts
Most discussions about AI performance optimizations focus on larger models and faster processors. Yet, hardware engineers increasingly recognize a different reality.
Building smarter AI systems is not simply a matter of adding more computational power. It also requires solving the data movement problem. Luckily, many researchers are focusing on enhancing the data movement problem. Doing so, they are exploring different approaches:
- Improved memory architectures.
- Faster interconnects.
- Memory-efficient algorithms.
- Model compression techniques.
- Near-memory computing.
- Optical and photonic communication technologies.
Each of these approaches attempts to answer the same question: How do we move large amounts of information efficiently?
The answer may determine the future trajectory of AI.
Modern AI systems depend on both computation and communication. While processors perform calculations, memory systems determine how quickly data can reach them. As models continue to grow in size and complexity, memory capacity and bandwidth are becoming increasingly important factors in overall performance.
The next major breakthrough in AI hardware may not come from a processor with more cores or higher clock speeds. It may come from a better way of moving data.
So, which one of these approaches (if any) is the answer? Well, at the moment, we don’t know, but we are slowly getting there.
Some references
- Brown, T. B., et al. (2020). Language Models are Few-Shot Learners.
- Chowdhery, A., et al. (2022). PaLM: Scaling Language Modeling with Pathways.
- Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
- Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention.
- Zhao, W. X., et al. (2023). A Survey of Large Language Models.