Building DevSecOps solutions using AWS, Terraform and Kubernetes

Debug Istio: From Pod to Gateway

  • 24th May 2025
Istio Diagram

Introduction

In this tutorial, we’ll create a very basic Istio setup that includes:

  • Kubernetes Namespace (gw-test)
  • Istio Gateway
  • Istio VirtualService
  • Kubernetes Service
  • Kubernetes Pod (via a deployment)

We’ll then explore ways to debug the setup using the CLI.

Understanding how Istio fits together like this is essential, but in real-world scenarios, you may find the Kiali Console more useful for debugging Istio configurations.

Prerequisites

  • You have a local kubernetes development cluster
  • You have kubectl installed and working
  • You have istioctl installed and working
  • You have istio configured
  • You have istio ingressgateway configured

This example code is intended for lightweight tutorial purposes only and is not suitable for production use.

Setup

Create nginx-gateway.yaml:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  selector:
    istio: ingressgateway # This is taken from the istio ingress label
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - istio.nginx.local # The hostname that comes into the cluster
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: nginx-vs
spec:
  hosts:
  - istio.nginx.local # This must exist in the Gateway
  gateways:
  - mesh # Allow VirtualService to be accessed from within the mesh too
  - nginx-gateway # The name of our Gateway that we created above
  http:
  - route:
    - destination:
        host: nginx-svc # The kubernetes service we want to connect to
        port:
          number: 80
    match: # Match on /nginx path only
      - uri:
          exact: /nginx
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  type: NodePort
  ports:
  - name: http-web
    port: 80
    targetPort: 80
    nodePort: 30001
  selector:
    app: nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: '
events {
}
http {
   server {
       listen 80;
       location / {
           return 200 "Hello world!";
       }
   }
}
'
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
          - name: config-vol
            mountPath: /etc/nginx/
      volumes:
        - name: config-vol
          configMap:
            name: nginx-config
            items:
              - key: nginx.conf
                path: nginx.conf

Create the namespace, enable Istio sidecar injection, and apply the resources defined earlier:

kubectl create namespace gw-test
kubectl label namespace gw-test istio-injection=enabled --overwrite
kubectl apply -f nginx-gateway.yaml -n gw-test

From Pod to Gateway

Now we’ll verify each component step by step, starting from the Pod and working our way out to the Gateway.

Use istioctl to describe the Pod and check its Istio configuration:

istioctl x describe pod nginx-56856bc749-86r6n.gw-test

Make sure to replace nginx-56856bc749-86r6n with the actual name of your pod, followed by the gw-test namespace.

You should see output similar to this:

Pod: nginx-56856bc749-86r6n.gw-test
Pod Revision: default
Pod Ports: 80 (nginx), 15090 (istio-proxy)
--------------------
Service: nginx-svc.gw-test
Port: http-web 80/HTTP targets pod port 80
VirtualService: nginx-vs.gw-test
Match: /nginx
--------------------
Effective PeerAuthentication:
Workload mTLS mode: PERMISSIVE
--------------------
Exposed on Ingress Gateway http://192.168.0.113
VirtualService: nginx-vs.gw-test
Match: /nginx

If the setup is correct, you can expect to see the full flow:

  • Ingress Gateway - e.g., http://192.168.0.113
  • VirtualService - nginx-vs.gw-test
  • Service - nginx-svc.gw-test
  • Pod - nginx-56856bc749-86r6n.gw-test

But what if something goes out of sync? How do we debug each step individually?

Is the Pod working?

First, check that the Pod has started as expected:

kubectl get pod -n gw-test

If the Pod failed to start, you can inspect the cause with:

kubectl describe pod nginx-56856bc749-86r6n -n gw-test

Next, verify that the Deployment was created successfully:

kubectl get deployments -n gw-test

You should also confirm that the Pod has a listener configured that the Service can connect to:

istioctl proxy-status -n gw-test
istioctl proxy-config listeners nginx-56856bc749-86r6n.gw-test
Is the Service working?

First, check that the Service was created:

kubectl get svc -n gw-test
kubectl describe svc nginx -n gw-test

To test whether the Service is reachable, you can use a temporary BusyBox pod:

kubectl run -i --tty busybox --rm --image=busybox --restart=Never -n gw-test -- sh
wget -qO- nginx-svc.gw-test.svc.cluster.local
Is the VirtualService working?

You can use istioctl to confirm that the VirtualService is correctly associated with the Service:

istioctl x describe svc nginx-vs.gw-test

You can confirm that the VirtualService and Gateway have been configured correctly by analysing the cluster.

istioctl analyze -n gw-test

This command checks for common misconfigurations and will report any detected issues.

Is the Gateway working?

You can check that the routes from the Ingress Gateway are correctly configured by running:

istioctl proxy-config routes istio-ingressgateway-9c954544f-dn4x7.istio-system

Replace istio-ingressgateway-9c954544f-dn4x7 with the name of your ingress pod, and istio-system with the namespace you deployed your ingress into.

You should see output similar to this:

NAME          VHOST NAME               DOMAINS               MATCH                  VIRTUAL SERVICE
http.8080     istio.nginx.local:80     istio.nginx.local     /nginx                 nginx-vs.gw-test

This means that visiting istio.nginx.local/nginx will route your request to the nginx-vs.gw-test VirtualService we created.

You can test this externally by sending a request to your Ingress Gateway’s IP with a mocked Host header. For example, if your Ingress Gateway is exposed at 192.168.0.113, run:

wget --header="Host: istio.nginx.local" -qO- 192.168.0.113/nginx

You can also test this internally by using a BusyBox Pod, for example:

kubectl run -i --tty busybox --rm --image=busybox --restart=Never -n gw-test -- sh
wget --header="Host: istio.nginx.local" -qO- istio-ingressgateway.istio-system.svc.cluster.local/nginx

This is particularly useful when we are not ready to expose the cluster to the outside world yet.

Remember to replace istio-ingressgateway.istio-system.svc.cluster.local with the service path of your existing istio ingress.

You can find this with:

kubectl get svc -A

Summary

Hopefully, this guide helps you understand where to look when connections between the various Istio and Kubernetes resources fail.

Rhuaridh

Please get in touch through my socials if you would like to ask any questions - I am always happy to speak tech!