Our service map looked fine on a test cluster. Then we looked at it on our public demo — Google’s Online Boutique, eleven services calling each other thousands of times a minute — and the frontend’s map showed two things: an edge to a peer named :, and an edge from a Kubernetes node.

The frontend calls the product catalog about 11,000 times every 15 minutes. It calls the currency service, the cart, recommendations, ads, shipping and checkout. None of that was on the map.

The map wasn’t broken in any obvious way. It was built from a reasonable source — TCP connections the kernel saw — and that source has three blind spots that add up to most of the traffic in a modern cluster.

The map was built from connection records, and a connection record was written when a connection closed: who connected to whom, for how long, how many bytes. That’s the natural moment — it’s when you know the totals — and it’s how plenty of flow-based tools work.

It’s also exactly wrong for how services talk to each other now. A gRPC client opens a channel to its server and keeps it for the life of the process. HTTP keep-alive holds connections open across thousands of requests. Database and cache clients keep pools. On our demo, in a 15-minute window, the Online Boutique services produced zero closed connections between them. Every connection record came from short-lived things: health checks, DNS, node agents.

So the busiest edges in the cluster were the ones least likely to appear.

When a pod connects to another service by name, it resolves the Service’s address — a virtual IP that the cluster’s networking forwards to a real pod. The connection’s peer, as the client’s socket records it, is usually that virtual IP, not the pod behind it.

Our map named peers by matching IP addresses against pods. A virtual Service IP doesn’t belong to any pod, so it couldn’t be named. And the : peer? Requests the sensor had parsed — HTTP and gRPC calls — were being counted with the connection records, and they carry no peer IP at all. An empty address and an empty port, joined by a colon.

Every upgrade of the collector restarts it, and a freshly started collector has never seen any of the connections already open on its node. If a connection is only recorded when it closes, and it closes next week, it’s invisible until next week.

On a cluster that upgrades its observability stack every couple of weeks, and whose gRPC channels last longer than that, the long-lived edges are effectively never seen.

What we changed

Use the requests, not just the connections

The sensor already parses application traffic — HTTP, gRPC, database protocols — and every request names what it was talking to: productcatalogservice:3550, cartservice:7070, checkoutservice.shop.svc.cluster.local. That name is the edge, and it doesn’t care whether the connection underneath has been open for a second or a month.

A name still has to be turned into a workload, and the cluster says how: the Service called cartservice in that namespace has a selector, the selector matches pods, and the pods belong to a Deployment. Following that chain names the edge correctly even when a Service isn’t named after its workload. Peers given as IP addresses resolve the same way: a pod’s IP to its workload, a Service’s virtual IP to whatever its selector picks.

There was one more trap. Online Boutique is also instrumented with OpenTelemetry, so each call is traced twice — by the application and by our sensor. Our pipeline marks the sensor’s copy as a duplicate so request counts don’t double, and the new map, reading only the sensor’s copies, skipped them as duplicates too. Every edge an instrumented application made vanished. For the map, the sensor’s copy is the edge, so it counts.

Report open connections, not just closed ones

Some traffic is never parsed — Redis, encrypted connections, protocols the sensor doesn’t speak — and for those the connection is all there is. So the sensor now reports open connections every minute: what moved on each one since the last report.

To see every open connection, including the ones opened before it started, the sensor walks every process’s open files in the kernel and follows each socket. (Our first attempt used the kernel’s TCP socket iterator, which turned out to walk only the network namespace of whoever runs it — the sensor’s own — and so saw no pods at all.)

Two details keep the numbers honest:

A connection that was already open when the sensor started is baselined the first time it’s seen. Its lifetime totals aren’t reported as this minute’s traffic.
When a connection finally closes, its close record carries only what moved since its last report, so bytes are never counted twice.

Make it fast

Reading every connection and request in a window to build one service’s map was slow — around 2.8 seconds and several hundred megabytes read, every time a service page loaded or an investigation looked at a service’s neighbours. Edges are now rolled up per minute as data arrives, and a map reads a few thousand rows.

The result

The frontend’s map on the demo now shows what it actually does: calls to the product catalog, currency, cart, recommendations, ads, shipping and checkout, with request counts, errors and latency on each, and the load generator calling it. The cart service shows its connection to Redis, through Redis’s Service address, with the bytes moving over it — an edge that exists only because the connection is now reported while it’s open.

Is your map telling the truth?

A few questions worth asking of any service map, including one you built:

Where do its edges come from? If it’s closed connections or flow records, long-lived connections may be missing. Check a service you know talks gRPC.
How does it name peers? If it matches pod IPs, connections to Service addresses may show up unnamed — or not at all.
What happens after a restart? Restart the collector and see whether established connections come back, or wait until they reconnect.
What does it do with traffic that’s traced twice? If duplicates are dropped before the map is built, instrumented services may lose their edges.

The uncomfortable part is that none of these failures look like failures. The map still renders, with plausible-looking edges. It’s just showing you the traffic that’s easiest to see, rather than the traffic that matters.

Why your service map is lying to you

Blind spot 1: connections that never close

Blind spot 2: peers that are virtual

Blind spot 3: everything opened before you started looking