It’s been a while since part 11 of my Kubernetes Intro, but new things keep coming. When you have a nice little Kubernetes cluster with dozens of interacting microservices running, you quickly come to the point where you need to debug the system. Today, and particularly in the telecoms networking world, tracing on network interfaces and analyzing the result with Wireshark is one of the most important debugging tools. But that’s a bit of a challenge when your signaling path flows through a Kubernetes cluster.
In a modern system architecture, a service is split up into many microservices running in many container instances across many worker nodes. As things are very dynamic and as there are so many network interfaces, tracing all of them and combining the output into a single Wireshark trace is hard. On top, communication between microservices is often encrypted (see my post on Istio), so an on-wire tap, even if it is virtual, helps little.
So there’s three challenges: The many network interfaces, dynamic communication between many microservices on many nodes, and encrypted network communication. The solution: A standardized way for microservices to output debug and status information. One popular way to do this is with a piece of software called ‘Jaeger’.
Jaeger offers an API that container applications can call to output debug information. The Jaeger solution grabs this output by running a ‘sidecar’ in Kubernetes pods alongside the actual microservices. The sidecars receive the debug information and forward it to a centralized Jaeger service (running in a Kubernetes cluster of course). There, debug information is stored and correlated. Correlation is important, because this way it is possible to see what kind of microservice interaction a http request triggers inside the Kubernetes cluster on physical and virtual network interfaces. There’s quite a bit of terminology around this correlation and Jaeger in general, about which you can find out more in the Jaeger documentation!