SK hynix argues AI's real bottleneck is data movement, not raw compute

[DiamondDallas]

An SK hynix research piece published September 3rd argues that as AI workloads shift toward inference and agentic AI, the actual performance bottleneck has moved away from GPU compute power and toward how quickly data can move between memory and processors. The piece traces this back to a 1994 paper by computer scientists William Wulf and Sally McKee, who warned that processor performance was improving roughly 60 percent annually while DRAM access speed improved only about 7 percent a year, a gap they termed the memory wall

The article explains that large language models increasingly rely on a KV cache, which stores key information from every previous token a model generates so it can reference that context later, and that cache can grow large enough to exceed the memory needed to hold the weights of a 70 billion parameter model in long context conversations. When a user sends a query to a service like ChatGPT, more than half of the GPU's actual work can involve moving data around rather than performing new calculations, a structural problem that adding more GPUs alone can't solve

Software fixes like FlashAttention and quantization techniques such as TurboQuant, presented by Google at a recent conference, have helped by compressing data and reducing unnecessary transfers, but the piece argues these approaches are approaching their theoretical limits. The proposed path forward involves architectures that place memory physically closer to compute through technologies like High Bandwidth Memory, High Bandwidth Flash, and Compute Express Link. Curious what people think about this framing, does treating data movement rather than raw compute as the central bottleneck change how people should think about which AI hardware investments actually matter most

[AI Ecosystem] The real bottleneck: Data, not compute | SK hynix Newsroomnews.skhynix.com

[Ivory Cass]

This is coming from a memory manufacturer specifically, worth keeping in mind that SK hynix obviously has a direct commercial interest in convincing the industry memory architecture matters more than pure GPU compute

[Jan]

Compute Express Link turning memory into a shared system level resource rather than something dedicated to one specific GPU feels like the more architecturally significant shift buried in this piece, that's a real rethink of how systems get designed

[NinaVrina]

Curious how much this specific bottleneck actually shapes near term purchasing decisions at major AI labs versus staying mostly a research and roadmap level conversation for the next several years

[PlanckLimit36]

More than half of a GPU's actual work during a typical query being data movement rather than computation reframes a lot of the pure compute focused AI hardware coverage that usually dominates headlines and investor conversations

[RatedRStu10]

The KV cache genuinely exceeding a 70 billion parameter model's own weight size in long context scenarios is a striking way to actually visualize how much memory pressure modern reasoning and agentic AI workloads create

[KnotKnull]

A 30 year old warning finally becoming the industry's central bottleneck is a genuinely satisfying bit of long term technical foresight, Wulf and McKee clearly saw this coming decades before anyone needed to care about it practically

[StarLord67]

TurboQuant approaching a theoretical compression limit is the detail that stood out most to me, once software optimization genuinely runs out of room, the pressure shifts entirely onto actual physical hardware architecture to solve the rest