Apple is talking to a startup that shrank a 54GB AI model down to under 4GB

[Grover26]

Apple is in early talks with PrismML, a Caltech spinout backed by Khosla Ventures, about technology that compresses large AI models small enough to run entirely on an iPhone. PrismML CEO Babak Hassibi told CNBC that Apple and other companies have been evaluating the startup's models, measuring speed, energy efficiency and performance on real devices, though he described the discussions as very early with no clear sense yet of where they'll lead

The technical trick is aggressive compression, storing each value in the model using just one or three possible states instead of the usual 16-bit precision, similar in spirit to the chip industry's shift from 8-bit to 4-bit computing but taken further. Using this method, PrismML shrank Alibaba's open source Qwen model from roughly 54GB down to under 4GB while keeping all 27 billion parameters active, small enough to run on an iPhone 15 or newer

The company says the compressed models use 10 to 15 times less memory, respond 6 to 8 times faster, and consume 3 to 6 times less energy than standard versions on existing hardware. There is a real tradeoff though, Hassibi acknowledged the compressed models lose a few percentage points of overall performance, with factual recall weakening before reasoning, math or coding skills do

Analysts are cautious about taking the claims at face value outside controlled demos. Tarun Pathak of Counterpoint Research said the real test will be millions of queries across thousands of device combinations, and IDC's Phil Solis flagged power consumption as possibly the biggest open question, since a model capable enough to run continuously in the background for agent-like tasks could still drain a battery even using less memory. The release also lands amid broader industry debate over whether AI efficiency gains like this could eventually reduce demand for memory chips and expensive datacenter infrastructure

Apple in talks with startup that shrinks AI models to run on an iPhonewww.cnbc.com

[TinyCompass]

The tradeoff detail is the important part everyone glosses over, losing factual recall first before reasoning and coding is actually a reasonable order of priorities for a phone assistant

[Ava82]

10 to 15 times less memory for basically the same active parameter count is a genuinely huge efficiency jump if it holds up outside a controlled demo