A GKE cluster running Dataplane V2 can hold at most 65,536 network identities, and an agent workload that gives every pod its own label can get close to that ceiling. Google laid out the problem on October 28, 2026, in a post on scaling agent sandboxes on GKE. The bottleneck isn't bandwidth or IP space. It's something as mundane as pod labels.
How a label can slow down pod startup
Dataplane V2 is GKE's Cilium- and eBPF-based data plane. To enforce network policy it doesn't look at each pod's IP. It uses a numeric identity derived from the pod's set of labels: pods with the same labels share an identity, and the eBPF rules are written once for all of them.
Trouble starts when a custom controller stamps each pod with something unique, such as an agent ID, a timestamp or a claim UID. Every new combination means a new identity. According to Google, that allocation is serial and updates the eBPF maps, so under heavy churn it can pause pod creation. And churn in these workloads is extreme: Google mentions research labs asking for up to one million concurrent code-execution sandboxes, and reinforcement learning pipelines that spin thousands up and down every minute.
A regular Deployment doesn't run into this because Cilium already ignores the per-instance labels Kubernetes generates, such as pod-template-hash, controller-uid or statefulset.kubernetes.io/pod-name. What it can't guess is that agent-id or timestamp are just as irrelevant to security.
Google's two fixes
The first is a design rule: data that isn't used by a Service selector or for grouping resources shouldn't be a label at all, but an annotation. Identities are computed from labels only, so moving the agent ID or the timestamp there removes the per-pod identity without losing traceability.
The second is for when you don't control who sets the labels, for example when a framework adds them: exclusion filters in the cilium-config-emergency-override ConfigMap in kube-system, with rules such as !agents.x-k8s.io/sandbox-name-hash or !timestamp. With those in place, every sandbox of the same agent type goes back to sharing a single Cilium identity.
With both measures, Google puts the result at 300 sandboxes per second with sub-second latency, and cites a customer running hundreds of thousands of isolated environments across clusters, with spikes of more than 30,000 sandboxes on a single one.
Doing the same with self-managed Cilium
If you run Cilium yourself, the project docs describe the same mechanism through the labels key of the cilium-config ConfigMap (or the Helm option --set labels=...). Patterns are regular expressions anchored at the start of the label key, so dots need escaping, and !timestamp would also drop timestamp-utc:
apiVersion: v1
kind: ConfigMap
metadata:
name: cilium-config
namespace: kube-system
data:
# Appended to Cilium's default exclusions
labels: "!agents\\.x-k8s\\.io/sandbox-name-hash !timestamp$"Existing identities don't change when you save the ConfigMap: you need to restart the agents (and the operator too, if it is the one managing identities). The operator then garbage-collects the old identities once nothing uses them.
kubectl rollout restart -n kube-system ds/ciliumThe caveat that matters: an excluded label no longer counts towards identity, so a NetworkPolicy that selects pods by that label can no longer tell them apart. Before excluding anything, search your policies for it.
Takeaways
Labels are part of your network contract, not free metadata. In any data plane that enforces policy by identity, every label you add to a pod can change how its traffic is evaluated. If you write controllers or operators, keep labels for what gets selected and push everything that only exists for tracing into annotations.
It's the Prometheus cardinality lesson applied to networking: one unique value per instance multiplies the state the system has to keep. And just as with metrics, it's better fixed at the source, in the controller, than with filters after the fact.
What doesn't carry over
Most clusters never get near 65,536 identities: you need high churn and unique per-pod labels at the same time. Outside AI, the likeliest place to hit it is controllers that create short-lived pods with a per-instance label, such as CI runners or per-pull-request environments.
What Google offers for going further, Agent Substrate, is a GKE product that applies these recommendations and also sidesteps Kubernetes scheduler limits. Its numbers (200,000 agents, 500 activations per second, and suspend and resume in under 500 ms) aren't something you can reproduce with your own Cilium install.
Sources
- How GKE networking scales to support high densities of concurrent AI agents, Google Cloud Blog, October 28, 2026.
- Limiting Identity-Relevant Labels, Cilium documentation.