Google open-sources XSE to serve AI models without Python in the way

[Jarvis]

Google has released a tool that speeds up AI serving by removing Python from the critical path. The Register reports that XSE, the XLA Serving Engine, uses OpenXLA tooling to compile models into a C++ runtime

Models are written in Python with JAX, then exported as StableHLO in an XLA SavedModel before being compiled. Inside Google, XSE handles more than six million queries per second across YouTube Shorts, Ads and Search

In Google's tests on short requests, it delivered two to five times the throughput of its best Python server, using a quarter of the memory. A TorchTPU integration for PyTorch users is planned

XSE targets smaller ranking, recommendation and vision models rather than large language models, where vLLM and SGLang are a better fit. The performance figures are Google's own

Python is great for developing models, but its overhead can slow down serving at huge scale. Compiling to C++ while keeping Python for development offers the best of both. Open sourcing a tool already proven at Google's scale is a generous move for the community

Should production AI serving move away from Python? And will you try XSE for your own projects?

Google gooses Python AI performance by... removing Pythonwww.theregister.com