Google Pixel deploys zero-copy MTP, Gemini Nano inference speeds up by over 50% while saving memory.

According to Beating's monitoring, Google has deployed a Multi-Token Prediction (MTP) architecture in its Pixel 9 and Pixel 10 series devices, directly accelerating the built-in Gemini Nano v3 model. By attaching a lightweight Transformer prediction head to the tail of the frozen main model, the new architecture improves on-device inference speed by more than 50% while fully preserving the original secure alignment and output quality. Traditional speculative decoding requires running a separate draft model to predict candidate tokens. This not only consumes additional memory on the phone but also limits prediction accuracy because the independent model cannot access the internal hidden state of the main model. The new architecture, by embedding the MTP head at the tail of the frozen main model, successfully reuses the feature activations already computed by the main model, significantly improving the prediction accuracy of candidate tokens. To avoid redundant memory overhead during autoregressive generation, Google designed a zero-copy mechanism. In traditional solutions, the draft model needs to maintain an independent key-value cache when generating candidate words. The zero-copy mechanism allows the external prediction head to directly access the main model's existing cache through cross-attention. This not only eliminates the startup latency of draft prediction but also saves approximately 130MB of RAM for the phone. In Pixel's real-world applications such as notification summarization and text proofreading, the MTP architecture enables the model to successfully predict nearly two more tokens on average per inference, reducing the frequency of the main processor being woken up for verification and thus saving system power. In highly structured text generation tasks such as smart replies, token acceptance rate is improved by up to 55%.