Google's TPUs now run AI workloads inside Meta's data centres

[Finley_19]

Google's custom AI chips are now running in a rival's data centres. Tom's Hardware's Shane Downing reports that Google's tensor processing units, or TPUs, are serving AI workloads in production at Meta. It is the first time Google has supplied TPUs to an outside customer at this scale

The software that makes it possible is called SGLang-TorchTPU, a version of the SGLang inference engine that works natively with PyTorch. It was built by startup RadixArk together with Google and Meta. A PyTorch native version of the popular vLLM engine has also been released as open source

Meta's Jana van Greunen said the company cannot be constrained by a single hardware architecture. Meta used TPUs between 2020 and 2023 before cancelling, and has reportedly signed a multi year rental deal this time

There are performance claims too. Inferact says 16 of Google's Ironwood TPUs served the Kimi K3 model at more than 700 tokens per second, compared with 452 on 16 Nvidia GB200 chips. The news came a day after Amazon said it would open source TorchNeuron for its own Trainium chips

Nvidia has dominated AI hardware, but the big cloud companies are working hard to offer alternatives. Better software support for PyTorch on other chips is the key to breaking that dominance. More competition should eventually help with the shortages and high prices we keep discussing

Can Google and Amazon challenge Nvidia's dominance? And does more competition in AI chips help ordinary consumers?

Google's TPUs for AI workloads now run in Metawww.tomshardware.com

[TheReal_Aaron96]

Amazon open sourcing TorchNeuron the day before is no coincidence. Everyone is chasing Nvidia's customers. The race is heating up