llm-d 1.0: LLM serving on Kubernetes with 85% lower time to first token

85% lower time to first token and 2.4x the throughput of a plain Kubernetes Service: that is what Google measured with llm-d on an agentic coding workload running on Ironwood TPUs. On October 20, 2026, at PyTorch Conference, the project shipped 1.0, with v1 APIs and the stability guarantee it needed to go into production without dreading every upgrade.

01What llm-d is and why a regular load balancer falls short

llm-d is an orchestration layer that sits above model servers (vLLM, SGLang) and turns a set of replicas into one coordinated pool. It is a CNCF Sandbox project under Apache 2.0, founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, with more than 500 contributors from 25 organizations according to Google.

The problem it solves is a very SRE one. Round-robin assumes requests are fast, uniform and cheap, and LLM requests are none of those: agents resend the same growing prefix over and over, a single response can take more than a minute, and sizes range from a thousand tokens to over a hundred thousand. When each agent step lands on a different replica, every replica recomputes the same prompt, accelerator memory fills with duplicate state, and one bursty agent loop starves interactive users.

The llm-d router builds on the Gateway API Inference Extension: it knows which replica has each prefix cached and also weighs load and predicted latency before picking a target.

02What 1.0 adds

Most of the building blocks already existed in 0.x releases (prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism). What is new is what you need to run it:

• Multi-tenant quality of service: flow control enforces priority and fairness so latency-sensitive traffic and bursty agents share a fleet without blowing each other's SLOs.
• v1 APIs and a hardened request path, validated across several clouds and accelerator types.
• Zero-downtime upgrades: router high availability plus DisaggregatedSet, built on LeaderWorkerSet, which rolls prefill and decode out together on matching versions without dropping the context of active sessions.
• Agent-aware serving: session-aware routing, turn-priority scheduling and native Responses API state.
• Async processing with llm-d-async-processor, which pulls requests from queues such as Pub/Sub or Redis and runs them in spare capacity, with priority lanes and quotas.

Tiered KV cache is what you feel most in long conversations: when accelerator memory fills up, llm-d moves the cache to host RAM and then to SSD or Managed Lustre (through vLLM's OffloadingConnector, Mooncake or SGLang's HiCache) instead of evicting it.

03The numbers and where they come from

Almost every figure in the announcement is Google measuring its own infrastructure, so read them for what they are: the ceiling in a heavily tuned environment.

The last row is the one I care about most: AMD hardware, no TPUs, no Google managed services, and the gain is still 3x. Most of the effect comes from not recomputing prefixes, and that has nothing to do with which cloud you run on.

04Deploying the baseline setup

The Optimized Baseline guide is the starting point the project recommends: two replicas of Qwen/Qwen3-32B on vLLM with NVIDIA GPUs (reference: 80 GB H100, TP=2, 4 GPUs in total) and the router in standalone mode, with Envoy bundled. You need kubectl, helm and a Hugging Face token in the llm-d-hf-token secret.

git clone https://github.com/llm-d/llm-d.git && cd llm-d
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh
export GUIDE_NAME="quickstart" NAMESPACE=llm-d-quickstart

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml
kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

helm install ${GUIDE_NAME} ${ROUTER_STANDALONE_CHART} \
  -f guides/recipes/router/base.values.yaml \
  -f guides/optimized-baseline/router/optimized-baseline.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

kubectl apply -n ${NAMESPACE} -k guides/optimized-baseline/modelserver/gpu/vllm/base/

By default guides/env.sh resolves versions to latest; for anything serious, pin the *_VERSION variables the script exposes so the chart doesn't change between two deploys. On GKE you can skip standalone mode: GKE Inference Gateway runs the llm-d router as a managed service and adds cross-region failover.

05Checklist before production

• Measure your peakPrefillThroughput with the repo's calibration recipe and set it on the affinity filter: the default is tuned for Qwen3-32B on H100, and it decides when to spread load instead of keeping affinity.
• Benchmark against your current load balancer with the same multi-turn workload; llm-d-benchmark and inference-perf ship reproducible recipes, and Prism lets you upload your runs next to the published ones.
• Test a DisaggregatedSet rollout under real traffic before trusting zero-downtime.
• Review autoscaling: KEDA or HPA on queue depth and saturation, not CPU.

If you serve your own models behind a Service today and your users are agents, 1.0 is the first release where it makes sense to treat llm-d as a stable part of the cluster rather than an experiment.

06Sources

• Announcing llm-d 1.0 — Google Cloud Blog, October 20, 2026.
• llm-d repository — README, license and performance figures.
• llm-d Quickstart — installation commands.
• Optimized Baseline guide — model, reference hardware and calibration.