Google Open-Sources TorchTPU and Ships llm-d 1.0 for Kubernetes AI Inference

Google released TorchTPU, its native PyTorch stack for Google TPUs, as open source on GitHub on October 20, 2026, and shipped version 1.0 of llm-d, an open-source distributed inference stack that runs on Kubernetes. Both were announced at PyTorch Conference 2026, according to a Google Cloud blog post by Mark Lohmeyer, vice president and general manager of AI and computing infrastructure, and Bill Jia, vice president of engineering.

“With TorchTPU, TPUs join GPUs and CPUs as a native option for PyTorch,” the post said. Google said TorchTPU was co-developed with the PyTorch team at Meta and lets teams run existing PyTorch workloads on TPUs with minimal code changes, debugging in eager mode and then moving to torch.compile for production performance.

Meta said it is using TorchTPU on key production use cases. “We cannot be constrained by a single hardware architecture,” said Jana van Greunen, Director of PyTorch Engineering at Meta, in the Google post. She said TorchTPU supports major LLM, ads and recommendation workloads at Meta and lets the company deploy PyTorch workloads to Google TPUs.

What Runs on TorchTPU

Google said it partnered with Hugging Face so that open-weight models on the Hugging Face Hub run natively on TorchTPU. In Transformers v5, deploying a model to TPUs takes passing device_map=”tpu”, the post said. TorchTPU also works with the vLLM and SGLang serving engines, the TorchTitan training framework, the VeRL reinforcement learning framework and Ray.

Google’s one training figure: TorchTPU’s training paths through TorchTitan reached up to 52% Model Flops Utilization on Qwen 3 8B, with what Google called near-linear scaling across up to 8,192 Ironwood TPUs. Google also said its TPU agent suite, which helps automate onboarding, diagnostics and inference tuning for PyTorch models, is in preview on GitHub, and that LiteRT, its on-device inference runtime formerly called TensorFlow Lite, has joined the PyTorch ecosystem.

llm-d Reaches 1.0

Google said llm-d 1.0 is a Kubernetes-native stack for distributed LLM inference, co-founded by Google with Red Hat, IBM and others, and built by more than 500 contributors across 25 organizations. It is a Cloud Native Computing Foundation Sandbox project.

The llm-d router sends each request to the replica that already holds its cached context and splits compute-heavy prefill from memory-heavy decode. The 1.0 release adds multi-tenant flow control, router high availability, zero-downtime updates, stable v1 APIs and session-aware routing for agents. It also introduces llm-d-async-processor, which pulls latency-tolerant requests from queues such as Google Cloud Pub/Sub or Redis and runs them in spare capacity alongside real-time traffic.

In Google’s own benchmark on an agentic coding workload on Ironwood TPUs, prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4x compared with a plain Kubernetes Service, as of October 2026. The Gemini platform, formerly Vertex AI, doubled its prefix cache hit rate from 35% to 70% after serving traffic through GKE Inference Gateway, which runs the llm-d router as a managed service.

Snap is integrating llm-d into its production AI infrastructure, and Vinay Kola, a senior manager of software engineering at Snap, said in Google’s llm-d post that prefix-cache-aware routing produced “prefix cache hit rates ranging up to 75-80%.”

Red Hat, which helped launch the llm-d project, framed 1.0 as a maturity milestone. In a Red Hat blog post, Brian Stevens, the company’s senior vice president and CTO for AI, wrote that an IBM Research deployment on llm-d reached a 98% prefix-cache hit rate and reduced blended serving costs five to ten times compared with proprietary options. Red Hat said it ships the stack to customers through Red Hat AI Inference, on Red Hat OpenShift AI or third-party Kubernetes environments, while Google offers verified Helm charts for deploying llm-d 1.0 on GKE.