Around 800 developers spent weeks unable to ship to production because their pods could not open a TCP connection to their own API server. There was no single culprit: three bugs in three different layers lined up, and the final fix was a Linux kernel patch. Google published the write-up on October 30, 2026, and it is worth reading even if you have never touched Private Service Connect.
The symptom: timeouts against the API server VIP
The customer ran a large multi-cluster GKE estate in europe-west, with the control plane exposed privately through Private Service Connect (PSC). In January 2026, dial tcp ...:443: i/o timeout errors against the kube-apiserver's internal IP started showing up in a UAT cluster. By day 10 they were persistent; by day 15 a P0 bug was open, five clusters were affected and production releases were frozen.
The list of ruled-out suspects is the useful part, because it is the list anyone would have worked through: a custom kopf-based operator (errors began eight days before it was deployed), External Secrets Operator and the security agents (uninstalled, no change), conntrack exhaustion (at 50% of 1.3 million entries) and an overloaded control plane (the VM was scaled up; latencies were already fine). DaemonSets, Deployments, system pods and even external clients such as laptops and GitHub Actions runners were all failing. The problem was the path, not any workload.
What the packet captures showed
Simultaneous captures at the control plane VM's NIC and inside the API server container exposed the pattern: a pod sent a SYN from a source port that had been used minutes earlier and, instead of a SYN-ACK, got back an ACK belonging to the previous connection. The server was still holding that socket in FIN_WAIT_2 more than two minutes after closing it.
That ACK is the RFC 5961 challenge ACK: when a SYN matches a connection the kernel still remembers, it replies with the old sequence number so a legitimate client answers with a RST and clears the state. It works as long as the ACK gets through. A change in PSC's packet switch, rolled out to fix an unrelated security issue in proxy protocol handling, started dropping ACKs that arrive before the SYN-ACK. An internal "invalid ACK" drop counter that had been flat for months jumped to several drops per second on February 18, 2026, matching the rollout.
PSC's NAT made it worse: many pods on different nodes share the endpoint and end up reusing the same source port, so 5-tuple collisions, and therefore challenge ACKs, were far more frequent than on a direct connection.
Three bugs that only hurt together
Rolling back the PSC change in February had clusters healthy again by early March, but it left the real question open: why were sockets stuck in FIN_WAIT_2 at all?
- Packet reordering. The first packet of a new flow takes a slow path through PSC (NAT plus proxy header) and later packets take a fast one. Packets arrived in order at the host NIC but not at the VM's virtual NIC, so the client's FIN could land after later ACKs.
- Envoy's half-close. Envoy, in front of kube-apiserver, closed with
shutdown(SHUT_WR): it stops writing but keeps the socket open for reading. That is correct TCP, buttcp_fin_timeout(60 seconds on those VMs) only applies to orphaned sockets. With Envoy holding it, the timer never started. - A kernel 6.1 bug. When the reordered FIN arrived,
tcp_ack()returned 0 for an ACK that was old but acceptable under RFC 5961, and inFIN_WAIT_2that meanttcp_fin()was never called. The FIN was silently dropped.
Each one alone was survivable. Together, the FIN arrived out of order, the kernel ignored it, Envoy kept the timeout from firing, NAT reused the port and PSC dropped the challenge ACK that would have cleaned it all up.
How it was fixed
Beyond the rollback, PSC got a proper fix for the challenge ACK case, backed by a PacketDrill regression test. The root cause was fixed in the kernel by Eric Dumazet's commit "net: tcp: accept old ack during closing": it is in Linux 6.9.7, backported to the 6.6.y and 6.1.y LTS branches, and ships in GKE through the COS images used from 1.33.5 onwards.
What I take away
The lesson that travels best outside Google is the half-close one: tcp_fin_timeout does not save you from a proxy that keeps the socket open. If you run Envoy, HAProxy or your own service behind NAT and see sporadic connect timeouts, check how many sockets sit in FIN_WAIT_2 and whether a process owns them:
# FIN_WAIT_2 sockets and the process holding them (needs root)
ss -tanp state fin-wait-2
# How many there are
ss -tan state fin-wait-2 | tail -n +2 | wc -lIf the count keeps climbing and every socket has an owning process, tcp_fin_timeout will not close them. And if your nodes run your own 6.1 or 6.6 kernel, make sure your distribution ships the tcp_ack() fix.
The second: any middlebox that "cleans up" odd-looking packets can break TCP mechanisms nobody remembers until they fail. Here, one well-meant filter deciding the challenge ACK was invalid was enough.
The third is about method: the case only cracked with simultaneous captures at the host, at the virtual NIC and inside the container, correlated with network metrics. A capture at a single point would have shown the symptom, never the reordering.
What does not transfer
USPS, Andromeda and Google's virtual NIC are internal components: you cannot instrument or patch them, and on GKE Google upgrades the control plane for you. It is also a single anonymised customer, with a port-reuse pattern that depends on how PSC shares its endpoint. What does transfer is the mechanism: reordering plus half-close plus an unpatched kernel can leave sockets hanging in any proxy you run yourself.
Sources
- When the Kubernetes API went silent: A three-layer networking mystery, Google Cloud Blog, October 30, 2026.