SK hynix argues AI's real bottleneck is data movement, not raw compute

[DiamondDallas]

An SK hynix research piece published September 3rd argues that as AI workloads shift toward inference and agentic AI, the actual performance bottleneck has moved away from GPU compute power and toward how quickly data can move between memory and processors. The piece traces this back to a 1994 paper by computer scientists William Wulf and Sally McKee, who warned that processor performance was improving roughly 60 percent annually while DRAM access speed improved only about 7 percent a year, a gap they termed the memory wall

The article explains that large language models increasingly rely on a KV cache, which stores key information from every previous token a model generates so it can reference that context later, and that cache can grow large enough to exceed the memory needed to hold the weights of a 70 billion parameter model in long context conversations. When a user sends a query to a service like ChatGPT, more than half of the GPU's actual work can involve moving data around rather than performing new calculations, a structural problem that adding more GPUs alone can't solve

Software fixes like FlashAttention and quantization techniques such as TurboQuant, presented by Google at a recent conference, have helped by compressing data and reducing unnecessary transfers, but the piece argues these approaches are approaching their theoretical limits. The proposed path forward involves architectures that place memory physically closer to compute through technologies like High Bandwidth Memory, High Bandwidth Flash, and Compute Express Link. Curious what people think about this framing, does treating data movement rather than raw compute as the central bottleneck change how people should think about which AI hardware investments actually matter most

[AI Ecosystem] The real bottleneck: Data, not compute | SK hynix Newsroomnews.skhynix.com