Canary Deployment with Istio in Kubernetes

13 min readFeb 14, 2022

We are retrofitting an existing K3D Cluster installation with Istio to use its Canary deployment capability. What could go wrong?

This article expands on a series of articles where we introduced quite a few technologies to a local K3D cluster to have an environment for experimentation. We started simple and added a lot of components:

K3D cluster with Ingress-Nginx (the beginning),
A NodeJS application exporting metrics for Prometheus and Grafana running locally, with Health Endpoints implemented,
Alertmanager running locally, with Slack alerts,
FluentBit and InfluxDB running locally to process application logs,
Grafana Cloud with Loki, Prometheus and Grafana hosted by Grafana as an alternative to the local Promentheus/InfluxDB/Grafana instances,
Goldilocks for resource estimates,
Keda for autoscaling,
Kubernetes Dashboard, and
Nginx to serve some web pages, including a version table.

The next target is to get Canary deployment working, and we use Istio as the solution, installed through its Helm charts. Introducing Istio enables other additions such as Kiali for traffic visualization or Jaeger for tracing, but these are topics for another article.

This article covers how to manage Canary deployment, i.e. how to split traffic between two services. That is handy when a new version of a service is introduced, and only a certain fraction of users of the service should initially be exposed to the newer version to make sure that things work well. Both service versions exist in parallel and access is transparent to the user. Usually, if things go well, the new service version is getting initially low load, more over time, and eventually, all traffic is directed to the newer version, and the original service can be retired. If things do not go well, and issues are seen with the newer version, it is possible to switch back right away before the majority of users are exposed to the issue.

Another use case is to direct some traffic to a stage or QA version of an application, and once that version is validated, promote it to the production environment. But this feels not as safe for service continuation as the parallel production-grade deployments approach.

We are using two different namespaces for the two deployments, and use a manual script to change the load distribution between the namespaces. Both workloads in the namespaces can be scaled independently, either using Keda or a Horizontal Pod Autoscaler (HPA).

Installation

This article and the other articles in this series are about getting you your own running system for your experiments. So you should go ahead and clone the repository and get things going. You will need a Windows PC with Windows Subsytem for Linux or a Linux PC, preferably with more than 8 GB RAM for the full install.

Before starting the cluster, you may want to configure a few options, such as Slack, Keda/HPA, Kubernetes Dashboard, etc, by editing the configuration file ./config.sh. The “out of the box” experience without any changes in the configuration file includes Nginx, Istio, and HPA. Istio will be installed in two namespaces, istio-system and istio-gateway, by way of their Helm charts. The values files are in the ./istio folder.

A noteworthy thing to mention is that we are using K3D release 5.3 with K3S 1.23, plus an enabled feature gate HPAContainerMetrics for managing replicas based on a container CPU load rather than an aggregated Pod CPU load.

Further installation instructions are now moved to the repository itself so that the article here on Medium.com is not filled with content that is essentially the same for all articles in the series… Check it out.

Istio Gateway vs Ingress Nginx

After an initial attempt to replace the existing Ingress-Nginx load balancer with the Istio Gateway, we ended up with a hybrid, where Ingress-Nginx co-exists with the Istio Gateway. Below is a high-level illustration. The Istio Gateway supports canary deployment with a pretty straightforward setup: we create two namespaces where we deploy our NodeJS application. We can then use a simple script to change the distribution of API calls between the two namespaces.

In the illustration, we show the two entry ports 8080 and 8081. The Ingress-Nginx routes use port 8080 and handle most traffic directly, except those calls that go to the NodeJS application. These requests are going to the Istio Gateway, all controlled by Ingresses. The Istio Gateway and the NodeJS application namespaces can also be reached directl ythrough 8081. The routing from the Istio Gateway to the NodeJS application in either the green or blue namespace is defined by a VirtualService.

Ideally, only one entry point to the cluster would be needed, but there was a hiccup: The Istio Gateway in the current version appears not to support rewrite with regex as Ingress-Nginx does. You can use a redirect configuration with the Istio Gateway, but the type of redirect supported by Ingress-Nginx shown below is not there, yet. See the ingress-nginx values file ./ingress-nginx/ingress-nginx-values.yaml.template:

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  className: nginx
  paths:
    - /dashboard(/|$)(.*)
  hosts:
    - localhost

Please correct if this is wrong… but there is no way to tell the Istio Gateway to use this type of rewrite-target notion. Because of this, and we need that for example for Kubernetes Dashboard in a subpath like http://localhost:8080/dashboard/, Ingress-Nginx remains in place, and we use a second port for direct Istio Gateway routing. This results in two paths to get to the NodeJS workloads, with the direct path through Istio Gateway being more efficient.

We also created a Grafana Dashboard to see the Canary deployment in action:

In the chart above, we deployed the application in both the Green and the Blue namespaces (explained later in the article), and generate traffic with a command ./app-traffic 0. This produces infinite API calls with half a second pause in between calls. Initially, all traffic goes to the Green namespace, and after some time, we created a 20:80 distribution of the API workload. You can see the traffic shifting from 100% Green to the 20/80 distribution.

$ ./app-canary.sh 20 80

The Green and Blue ratios on the right side are calculated from the actual HTTP traffic metrics. There is some fluctuation around the target, but the reality is generally pretty close to the target.

The script to change the traffic split takes two values that should total up to 100, indicating the percentage of share of traffic. It is possible to use 0 on one, and 100 on the other to direct all traffic to one namespace. The lowest percentage that can be used is 1%, resulting in 99% going to the other namespace.

Scaling Canary Deployments

In the lower section of the Dashboard, we can see the number of pods over time for the separate deployments — if there would be more than 80% load on CPU, it should scale up, if Keda or HPA is enabled — see details about Keda in the Keda article.

Below is a screenshot showing how the Green namespace scales up and the Blue stays constant under more load. Green gets the extra pods as it carries 80% of the traffic, hence there are more pods needed to keep the load within limits — this is expected.

You can also observe the lag before going back to a minimum of two replicas. In the picture above we used terminals running ./app-traffic.sh 0 0.1 to generate continuous API calls with 0.1 seconds delay between calls. All processes except one have been terminated once the scaling happened, and with some delay, scaling back happens automatically.

However, when using Keda under load and comparing the scaling trigger with the CPU load display in the Scale Demonstration Dashboard at the same time, we can see that the CPU load is much higher than expected, usually well exceeding the 80% trigger point. Here is why: the Scale dashboard shows the CPU load of the NodeJS app container within the Pod. However, Keda creates a Horizonal Pod Autoscaler object that triggers on the CPU load of the Pod, which includes the Istio Proxy. While the HPA in very recent Kubernetes releases includes a feature to trigger on the specific containers metric, there is currently no way to specify this in a Keda ScaledObject. So while Keda Autoscaling works in general, it seems not accurate to use Keda together with Istio due to the Istio Proxy, and it may be more effective to utilize HPA directly. Because our NodeJS application does not really do a lot, the influence of the Istio Proxy is comparatively large. In fact, we request just 10m CPU for the NodeJS container, and 40m for the proxy, which virtually guarantees misjudgment.

Because of the Keda situation, there is a new configuration in config.sh called HPA_ENABLE. Setting this to “yes” will disable KEDA_ENABLE, i.e. Keda does not get installed with ./start.sh. With the HPA setting “yes”, a HPA will be created during application deployment in app-deploy.sh. Note that at the time of writing, Lens version 5.3.4 does not support the proper display of HPA with API version 2 — it shows in Lens as v2beta1. But this kubectl call reflects the reality:

$ kubectl get hpa -n myapp-green -o yaml
[...]
  spec:
    maxReplicas: 20
    metrics:
    - containerResource:
        container: myapp-container
        name: cpu
        target:
          averageUtilization: 80
          type: Utilization
      type: ContainerResource
[...]

The difference between Keda and this more specific HPA manifest is that the scaling with Keda responds to the Pods aggregated CPU load, while the individual HPA manifest responds to the application container only. Keda is probably catching up in the future.

It should be noted that in order to be able to use the newer version of HPA with the current K3D Kubernetes version, you will need to enable a feature gate when launching the cluster. That is achieved through the K3D configuration file k3d-config.yaml.template; look for the k3s extraArgs: where it is saying HPAContainerMetrics=true.

If you disable both Keda and HPA in config.sh, you can try manual scaling, e.g. using ./app-scale.sh 5 to replicate five pods. Note that the script ./app-scale.sh now takes an optional second argument, indicating which deployment to scale; myapp-green is the default, myapp-blue is the other namespace. Note also that ./app-scale.sh should only be used when Kedia and HPA are disabled, otherwise, the scaling would adjust automatically.

Istio Dashboards

Other than the new custom dashboard, there are also a handful of Istio-specific Dashboards installed, which can be accessed through the Grafana Dashboard Browser. Most of them only work if there is some traffic, so it is a good idea to have a ./app-traffic.sh 0 process running all the time in a separate terminal. Using the call with a single 0 as an argument will generate an infinite loop of API calls 0.5 seconds apart.

Setting up Canary Deployment

The section above covers how to use the Grafana Dashboard to see the Canary Deployment effects. This is, belated, how to set up the configuration that we discussed in the beginning of the article.

The first step is to review the config.sh file and launch the cluster with ./start.sh. You will need to have ISTIO_ENABLE="yes" in config.sh, other feature settings are optional. For the first run, you may not want to change anything as the config.sh version in the repository should be good to go. Further installation options are covered in README.md. The default includes a direct HPA deployment, i.e. Keda is disabled.

It is needed to generate some traffic to the service for metrics to be available. You can use this:

$ export KUBECONFIG=~/.k3d/kubeconfig-mycluster.yaml
$ cd app
$ ./app-traffic 0

This generates an infinite sequence of calls to the API at localhost:80801/service. The default installation settings deploy the application in the myapp-green namespace. You can visit the Dashboard and see “no pods” for the Blue metrics, but some data for the Green metrics. The Dashboard lags reality by a minute or so, because of the scraping intervals, so give it a little while to catch up.

We need to manually create a second deployment and then split traffic between Green and Blue.

$ export KUBECONFIG=~/.k3d/kubeconfig-mycluster.yaml
$ cd app
$ ./app-deploy.sh myapp-blue
[...]
$ ./app-canary.sh 90 10

This should create an additional deployment, using the myapp-blue namespace, and distribute 10% of the traffic to the Blue. That can be observed in the Dashboard after a short while. You can use ./app-canary-status.sh to see the traffic configuration in a terminal.

Some more detail: as seen above, ./app-deploy.sh now takes an optional argument, either myapp-green or myapp-blue. myapp-green is the default. This script does not build the image, that can be done by ./app-build.sh. The deployment script removes a possibly previously existing deployment, creates a new one, and sets up the Virtual Gateway and secondary route from Ingress-Nginx to the Istio Gateway. There is some special processing in case of an already existing Virtual Gateway so that a previous Canary setup remains in place.

However, as we remove the previous installation, there may be a service interruption. An alternative way to update an existing deployment is the use of ./app-update.sh which would require that there was a previous deployment. ./app-update.sh does not remove an existing deployment, and only updates the container image of an existing one. The update script changes the deployment date for that updated deployment, which would be reflected after a little while in the Dashboard.

The Deployment date in the Dashboard was introduced to support the experimentation. Usually, you may want to use version numbers to differentiate between deployments, but we in fact deployed the same application version 1.7.0 twice (see package.json). So we now capture the time and date of the initial deployment or update, and you can see which of the namespaces is the newer one.

You can use ./app-canary-status.sh to reveal the target ratio that is set in the Virtual Gateway. An example output is here:

==== ./app-canary-status.sh: VirtualService myapp in namespace istio-gateway
Green Route exists with weight 30.
Blue Route exists with weight 70.==== ./app-canary-status.sh: Route Detail:
{
  "destination": {
    "host": "myapp-service.myapp-green.svc.cluster.local",
    "port": {
      "number": 3000
    }
  },
  "weight": 30
}
{
  "destination": {
    "host": "myapp-service.myapp-blue.svc.cluster.local",
    "port": {
      "number": 3000
    }
  },
  "weight": 70
}

Side Effects

This section covers some items that are specific to the changes to the installation compared to earlier versions within this series of articles — you may not care if you did not follow the evolution of installation.

Fluentbit Data Source

We are using Fluentbit to process log data from the app. In the Fluentbit values file, we configured a filter with a wildcard that matched logs from the NodeJS application namespace. This was covering both Green and Blue, which is good. But it also matched log output from the Istio proxy, which was not intended to be included in the scraping and which broke some custom Grafana Dashboards. So that matching filter needed to be more specific to the NodeJS application logs. the change is in ./fluentbit/fluentbit-values.yaml.template.

Source Code Directory Structure

Nearly all scripts and config files have been in a single flat directory, which became unmanageable due to the number of components. There is now a reasonable set of sub-directories, which in turn forced one more environment variable PROJECTHOME to allow each script to find the global configuration file. PROJECTHOME should be the directory where the config.sh file is located. It is automatically guessed in most scripts but can be overwritten by the PROJECTHOME environment variable.

$ export PROJECTHOME="<path-to-config.sh>"

Dependency between Istio and App Deployments

There is an overall pattern of independent deployment scripts and associated undeployment scripts (such as ./app-deploy.sh and ./app-undeploy.sh), basically one pair of those for each namespace. The idea is that you can remove components, change parameters, and redeploy components, in any order. However, with Istio, there are now more dependencies between Istio, Ingress-Nginx, and the NodeJS app. This caused a not-too-elegant approach to guess or detect an existing configuration and possibly change a component configuration in another namespace.

For example, if Istio is re-deployed with ./istio-deploy.sh while the whole system is running already, we now need to remember the load distribution between Green and Blue before Istio’s removal, reinstall it after deployment, and restart the app after Istio completed the reinstallation, as the existing proxies in the NodeJS app namespaces lost their connection to the original IstioD. These proxies need restart as well, but they are in a different namespace. This is a complex process, and probably error-prone.

Grafana Dashboards

The custom Grafana dashboards for Application Logs and Keda Scaling needed some rework, as we have now different namespaces for the apps. Therefore, another matching filter was used. Note that the two mentioned dashboards do not differentiate between the Green and Blue deployments, and aggregate the values for both namespaces. That’s the intention. For example, per default, the Scale demo shows four pods, as there are per default two in Green and Blue each.

As mentioned earlier, there is now a new custom dashboard showing how the Canary Deployments work.

K3D and K3S updates

This version introduces K3D 5.3 (and K3S 1.23, see k3d-config.yaml.template), plus we enable a feature gate to support HPA V2. Version details of running components can be seen at http://localhost:8080 if Nginx is enabled.

Grafana Cloud

This is not directly related to Istio, but as we added more and more components, we generate more metrics, and there is a limit in the free tier with Grafana Cloud. We removed some metrics from the export, e.g. filesystem-related data. The active metrics are in ./grafana-cloud/grafana-cloud-metrics.sh. You can enable the full set of metrics in that script.

Where to go from here?

The introduction of Istio opens a lot more opportunities, for example using tools that depend on a service mesh, like Kiali or Jaeger. More work could be done to eliminate the need for ingress-nginx in favor of the Istio Gateway, although the co-existence of two load balancer environments may be a good thing for education.

A deeper look into the scaling of a Canary deployment is certainly important, as discussed in the article. Until Keda supports container-specific measurements, it is probably good to disable Keda when using Istio and have an HPA utilizing the relatively new ContainerResource based scaling.