Use Pulse Virtual Traffic Manager to route traffic to Kubernetes pods

Pulse Virtual Traffic Manager (vTM) v18.1 introduced new Service Discovery mechanism to help with situations where vTM sits in front of an application made up from dynamic components.

Many organisations are now either use or experimenting with Kubernetes (K8s). Let’s see if we can get this Service Discovery mechanism to help us expose an application running on K8s to the Internet:

• In a way that works both in public clouds and on-prem;
• Supports session persistence and per-pod health checks;
• Can scale really well; and
• Is highly available*.

*The above assumes that you operate a production-grade HA cluster of vTMs somewhere where they can reach your K8s nodes running your application pods.

Our test bed set-up

First, we’ll need a K8s cluster to deploy our sample app to, that we can then discover. For the purpose of this blog, I’ll be using Quick Start for Kubernetes by Heptio on AWS. I simply deploy a new copy of the Quick Start into a new VPC, wait until it’s all up, and then retrieve my kubeconfig by using command shown in the CloudFormation Output:

Get kubeconfig command

Once we have it, quickly test that we can talk to the cluster, and let’s move on (we’ll assume that you’ve saved the kubeconfig file into your current working directory):

export KUBECONFIG=$(pwd)/kubeconfig

kubectl get nodes

# [ .. output skipped ..]

 

Next, we’ll need to procure a Developer Edition of vTM 18.1 to play with. If you have a host with Docker installed on it, the easiest way to get around it is to use the unofficial copy I’ve put up on Docker Hub – dk114/pulse-vtm:18.1. On your Docker host, you could run something on the lines of:

docker run --name vTM --rm -e ZEUS_DEVMODE=yes -e ZEUS_EULA=accept -e ZEUS_PASS=abc123 -p 9090:9090 -p 9070:9070 -p 80:80 -p 443:443 --privileged -t -d dk114/pulse-vtm:18.1

 

This should pull and deploy a vTM 18.1 in Developer Mode for you, and make it accessible through its UI interface (on port 9090) and REST API (port 9070). It will also expose ports 80 and 443, which at the beginning won’t be listening.

Alternatively, you can deploy an HA cluster of vTMs into AWS using my CloudFormation Template, making sure to set vTM version parameter to 181 when deploying. You will need an existing VPC with at least one public subnet that you will need to pass to the template when deploying. You can use the same VPC that Heptio’s Quick Start creates. To allow vTMs to talk to pods running on K8s nodes, add the id of the Security Group called k8s-cluster-security-group created by Heptio’s Quick Start to the vTM template’s parameter EnvSGs, e.g., sg-b47d5fcd.

For the rest of this blog, we’ll assume you’re using vTM in a Docker container.

Intro to vTM’s Service Discovery

On your Docker host, you should be able to access your vTM container through your host’s IP and port 9090; in my case I’m running it on my laptop, and can access it through https://127.0.0.1:9090. The page will throw a warning that it’s using a self-signed cert, which is fine in this case. Proceed to get to the login page, and log in with admin / abc123 (password we’ve set through ZEUS_PASS variable in our docker run command above).

Once logged in, hit Catalogs -> Service Discovery:

Service Discovery Menu

This will bring you to the page where we can add our own Service Discovery plugin.

The documentation for Service Discovery is included in the vTM User Guide, page number 79 (pdf page 99).

In a nutshell, this feature allows you to upload any arbitrary program to your vTM as a Custom User Plugin. You can then create a pool, and choose your plugin as the means to populate that pool with IPs and ports of this pool’s nodes.

vTM will then run your plugin program every X seconds (the number you can specify; 10 by default), and read / parse the output that plugin returns to find IPs, port numbers, and state for the nodes in your pool. It will then compare what the plugin just returned with the current state of the pool – IPs, ports, node state, etc., and figure out if it needs to make any changes. If not – nothing further is done; if yes – vTM will update pool’s config accordingly.

This may all sound harder than it is, so let’s get our hands dirty and create a very simple (if useless) plugin to see how it works.

Create a file called test-plugin.sh with the following contents:

#!/bin/bash

cat <<EOF
{
    "version":1,
    "nodes":[
        { "ip":"192.0.2.0", "port":80 },
        { "ip":"192.0.2.1", "port":81, "draining":true }
    ],
    "code":200
}
EOF

 

If this script is run, it will output a JSON file that’s been copied from the vTM User Guide, page number 83 (pdf page 103).

Let’s see what happens if we add it to our vTM. On the Service Discovery page click Custom User Plugins, then Upload Plugin -> Browse -> select your test-plugin.shfile, and click Upload.

Plugin Upload

Once uploaded, you’ll be able to Test plugin, by running it once with optional Arguments. Since our plugin doesn’t need any, simply click Test to see what vTM thinks this plugin is returning:

Plugin Test

In our case, as expected, it is seeing two nodes, one of which is draining.

Ok, let’s create a pool that uses this plugin:

New Pool

Hit Create Pool, and on the next page you’ll have a chance to adjust settings of the Service Discovery, including arguments, scheduling interval (how often it is run), and run timeout.

Ok, let’s see what we now have in our pool! Click Services -> Pools -> Test to see what’s going on:

Populated Pool

As you can see, you now have nodes in your pool, that came from our (admittedly useless) plugin.

Let’s reflect and take this in a bit.

What you essentially have is an ability to run any code directly on vTM to do any sort of thing to discover what the nodes for your pool should be. It could be as simple as running a curl to fetch a file that your application’s provisioning system populates with IPs / Ports of your app components, to something that talks to AWS API and K8s API to dynamically find components of your application that is distributed across on-premises and public cloud.

Why not event-driven?

Next question that pops into your mind is probably “why couldn’t vTM listen for external events, instead of polling? This is a valid question, but when you consider the complexity involved in making this work, universally, vs. the benefit you get, in most cases simple polling would suffice just fine.

Additionally, the current implementation allows your plugins to tell vTM to adjust its polling interval (up or down) by using the optional interval_override parameter.

Ok, so when can we get to K8s?

How about “now”? 🙂

First, clone or download the following repo: https://github.com/dkalintsev/vTM-K8s-ServiceDiscovery, and change your working directory to the one you’ve downloaded the repo into.

Disclaimer Note: I wrote this plugin as a “proof of concept”, and it should not be used in production “as-is”. Please treat this code as “unsupported” and “experimental”. The idea is to provide a working prototype that can be used as an inspiration for your own implementation.

You should have something like this; we’ll get to what’s what in here in a moment:

.
|____deploy-template
| |____files
| | |____K8s-get-nodeport-ips.sh
| | |____my-kubeconf
| |____main.tf
| |____terraform.tfvars
| |____variables.tf
|____Manifests
| |____K8s-get-nodeport-ips.yaml
| |____my-nginx.yaml
|____README.md
|____test-plugin.sh

 

The game plan for this exercise is as follows:

• Our Service Discovery plugin will be a script that calls kubectl to find the IPs and Port number for the Pods we’d like to send traffic to
• The script then formats this information in the shape vTM expects it and prints it out
• vTM reads the output, and updates a pool, as/if necessary.

Let’s break it down.

The plugin itself

In the repo you’ve cloned above, there’s a file deploy-template/files/K8s-get-nodeport-ips.sh, which will do the work for us. It may look a bit complex, but once you strip down all data massaging, prerequisites, and error handling logic – it’s very simple:

• Take a name of a K8s Service that you pass it as a parameter;
• Confirm that the type of this Service is NodePort;
• Find the ephemeral port K8s allocated to that NodePort – this is the port we’re looking for;
• Find the pods backing our Service by reading Service’s Endpoint list;
• Find the IPs of the K8s hosts where each of the pods runs – these are the IPs we’re after;
• Format the IPs + port into a JSON string vTM expects, and print it out.

What does this plugin need?

Since our plugin is a simple Shell script, the work of talking to K8s API and processing JSON falls on to “helpers”. In our case we use kubectl and jq, both of which do not ship with vTM.

vTM has a concept of Extra Files in its Catalogs. All files uploaded into that section of the Catalogs end up in the directory $ZEUSHOME/zxtm/conf/extra/, which is automatically replicated between vTMs in a cluster.

Our plugin uses this to its advantage. When it is run, it will check if kubectl and jq are already available, and, if not, it will, if told to through a -g parameter, attempt downloading them from their respective origins and place under the extras directory mentioned above.

What about authentication when talking to K8s?

The easiest but the least secure way to let the plugin talk to your K8s cluster is to supply it with a copy of kubeconfig file we’ve downloaded at the start of this blog.

Let’s however pretend that we care about security, and do it a bit better. Instead of using the K8s admin user, let’s create a new one with a restricted set of permissions that only allow R/O access to what it needs.

In the repo you’ve downloaded, have a look at the file Manifests/K8s-get-nodeport-ips.yaml. When created, it makes a ServiceAccount called vtm-user and similarly named ClusterRole with restricted permissions, and then binds them together. As the result, user vtm-user has the restricted permissions that we’re after.

Next up, we need to create a kubeconfig file that uses our vtm-user instead of the default admin.

In the repo, there’s a file deploy-template/files/my-kubeconf that you can use as a guide:

1) Open your real kubeconfig file and copy the top part of it:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [Cert contents skipped]
    server: https://k8s-k8sst-apiloadb-<blah>-<blah>.ap-southeast-2.elb.amazonaws.com
  name: kubernetes

 

2) Edit the deploy-template/files/my-kubeconf and replace the respective part with the one you’ve copied above, leaving everything starting with the line contexts: and below intact.

3) Create the ServiceAccount, ClusterRole, and their binding for our vtm-user:

kubectl create -f Manifests/K8s-get-nodeport-ips.yaml

 

4) Retrieve the token for the vtm-user:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep vtm-user | awk '{print $1}')

 

You should see something similar to:

Name:         vtm-user-token-6rzkh
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=vtm-user
              kubernetes.io/service-account.uid=d11872bd-4f32-11e8-84a3-060a49790776

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciO[ ...blah-blah... ]4So2k8g <<== WE'RE AFTER THIS
ca.crt:     1025 bytes
namespace:  11 bytes

 

5) Copy the long string after the token: and replace the simlar-looking long string inside your deploy-template/files/my-kubeconf.

6) Test if you got it all right:

kubectl --kubeconfig=deploy-template/files/my-kubeconf get nodes

# You should see a list of nodes in you cluster

kubectl --kubeconfig=deploy-template/files/my-kubeconf get secrets

# You should get an error message, since vtm-user isn't authorised to do that

 

Ok, we’re all good; so let’s get ourselves something to discover:

kubectl create -f Manifests/my-nginx.yaml

 

This should spin up one instance of an Nginx pod that serves a document containing its own internal hostname on port 80, that is then exposed via a Service called my-nginx-service on a NodePort.

Run kubectl get pods to make sure our Nginx pod is running, and let’s get our vTM to see it!

Adding plugin to vTM

Following the same process as we did for test-plugin.sh above, add deploy-template/files/K8s-get-nodeport-ips.sh as a Custom User Plugin.

Next, we’ll need to add our my-kubeconf file to Catalogs -> Extra Files -> Miscellaneous:

my-kubeconf upload

At this point, we should have all moving bits in place, so we can test it out! Fill the following into the Arguments for our Custom User Plugin K8s-get-nodeport-ips.sh, and hit “Test”:

-s my-nginx-service -p http -c my-kubeconf -g

 

Refer to the comments inside the plugin code for details, but what we’re telling it is:

• Look for the K8s Service called my-nginx-service
• Get a NodePort value that corresponds to that Service’s port called “http”
• Use the file from Extras directory called my-kubeconf to communicate with the K8s cluster
• get the prerequisite helpers (kubectl and jq), if they’re not already in Extras

After clicking “Test” for the very first time, it will take a good few moments, depending on your Internet speed, as it will be downloading a copy of kubectl and jq (and curl, if you’re running a Docker version of vTM). Then, if all went well, we’ll see something like this:

Successful Test

It worked!

It’s now time to create a pool and see what happens then.

New Pool – K8s

Hit “Create Pool”, and then go back to the list of pools to see if the nodes got populated:

K8s Pool – one node

Looks good! Now, what happens if we scale up our Nginx deployment to two pods? Let’s find out:

kubectl scale deployment my-nginx --replicas=2

 

Wait 10 seconds (default poll timer), refresh – and here we go, two nodes:

Pool scaled up

Summary

What we’ve just done is added a new, completely bespoke Service Discovery plugin to our vTM 18.1, and got it to discover our “application” running on K8s, exposed through a NodePort Service.

If our vTM was able to reach the IP addresses of K8s nodes, we could have started directing traffic to them; however in our sandbox where vTM was running on my laptop while K8s cluster was sitting in AWS, this would not work.

In a more realistic scenario, you would have a vTM cluster configured for High Availability with Traffic IPs sitting next to your K8s cluster(s). This set-up would solve the following problems:

1) Scalability: you would be able to direct as much traffic to your application as your vTM cluster can handle, which is typically a lot. vTMs in a cluster can operate as “all-active”, so capacity a cluster can handle scales directly with the number of nodes in the cluster.

2) High availability: vTM cluster running outside of K8s can use its built-in high availability features built around Traffic IPs, without any limitations.

3) The need to dance around with K8s Cluster IPs is eliminated; especially in AWS where VPC doesn’t support more specific routes. Traffic to your K8s Service is sent directly to the IPs of the actual nodes where your pods are running. Yes, since we’re using NodePort, this traffic will then be handled by kube-proxy, but since we have our Service configured with externalTrafficPolicy: Local*, it will only forward traffic to the pods residing on the node. Traffic from vTMs will never be sent to a node that’s just gone down, because we’ll discover the fact that the pods on the failed node went down, and the IP of that node will be gone from the vTM’s pool.

*Note: at the time of writing (8 May 2018) Heptio K8s appears to have a bug that prevents use of Services with externalTrafficPolicy: Local – traffic gets silently dropped. See this issue for any updates. This means that it won’t be possible to guarantee session persistence and per-pod health checks from vTM. One possible work-around would be to use hostPort on your application pods; however this will require changes to the Service Discovery plugin code.

You probably noticed that my-nginx.yaml includes node anti-affinity configuration in the Deployment’s spec. This is needed to ensure that each K8s node where vTM sends traffic has only one replica of your application’s pod, so that session persistence and per-pod health checks can work.

You should take this into consideration and decide whether these benefits (persistence + health checks) outweigh the “cost” associated with this: you won’t be able to run more than one replica of your app pod on the same K8s node. One possible work-around is to have more nodes than the maximum number of replicas your app requires. If your K8s nodes are VMs, this should be relatively easy to do.

Um, what’s with those Terraform-looking files in the repo?

The repo also contains a Terraform template that can deploy a pool with Service Discovery using freshly-released Terraform Provider for vTM. Please feel free to play around with it. See the my other blog post on intro to Terraform with vTM to get yourself started.

To give it a quick go, you can do the following:

• Update the deploy-template/terraform.tfvars file and make sure that vtm_rest_ip is set to the IP of your vTM, and vtm_password is set accordingly.
• Change your working directory to deploy-template, and run the following command:

docker run --rm -v $PWD:/root/k8s-sd -w /root/k8s-sd dk114/try-vtmtf:1.1 /bin/ash -c "terraform init && terraform apply -auto-approve"

 

This will run the dk114/try-vtmtf:1.1 Docker container that is an Alpine linux with a copy of terraform + Terraform Provider for vTM, that will apply the Terraform template from your current directory. After it successfully applies, your vTM should have a new uniquely-named copy of our plugin, similarly uniquely-named copy of the kubeconfig file, and a new pool configured to use this plugin.

See you next time!