Attacking Kubernetes

I recently moved into a new role in IBM Security. As I am finding my way around what to me are new technologies and, actually, a new set of vocabulary, I wanted to link my new role back to my old role, and ask "How would an attacker go after Kubernetes?" And I found that on top of the 'usual' approaches - i.e. trying to exploit known vulnerabilities in the software, especially in open source (remember Log4J?); installing malware and ransomware via phishing and other means; DoS attacks, etc etc - there is a entirely new set of challenges due to the fact that we run software on OpenShift/Kubernetes and in containers.

Many of the sources I found when looking into the topic of breaking into Kubernetes implied that it is fairly easy to do so. Almost all of them suggest that the best way to do it is not to break into the front door (i.e. the ingress / proxy / gateway that sits in front of the cluster), but to try getting into a pod that is running in the cluster, and from there to penetrate further by starting additional pods and 'breaking out' of the pod, i.e. taking over the worker and master nodes.

What's more, virtually all of the mechanisms that I found documented - at least the simpler ones - suggest that you start a shell inside a running pod. If you don't have credentials into the cluster that would allow to run a simple kubectl exec command, one way of achieving that is to exploit weaknesses in the software running in the pod to establish a reverse shell. I think it is fair to assume that those weaknesses have existed and will continue to exist, despite all efforts to prevent them. So, for the rest of this post, I will assume that a hacker found a way into a pod and managed to get a shell in it that can be used to execute commands. For my own experiments, I used the "Damn Vulnerable Web Application" - never use that thing in a public environment, though! Moreover, I ran it all in an OpenShift cluster. but I believe the behavior described below is the same in any Kubernetes cluster.

Once you are running a shell inside a pod, you can try to use this pod to create other pods for further exploitation. Each pod will have a token that can be used to access the cluster's API server, in the /var/run/secrets/kubernetes.io/serviceaccount folder. The API server's address is bound to the "kubernetes.default.svc" DNS name. Using those two, I can run kubectl commands (assuming, of course, that I can install the kubectl binary if it is not already there).

And here is where the good news start: in Kubernetes, each pod is associated with a service account, and this service account lets you limit the rights of what the pod can ask the API server to do. And guess what, in Kubernetes, a default service account cannot be used to create pods out of the box. 

What's more, it also stops you from getting into a pod that runs with higher privileges than the one you are in. For example, let's assume that you run in a pod with a service account that is allowed to list pods and exec into them. Using kubectl get pods, I find a pod that looks like it may have privileged access and try to get inside of it. The API Server will not allow that, telling me that I cannot elevate my privileges this way: 

Error from server (Forbidden): pods "abc" is forbidden: exec operation is not allowed because the pod's security context exceeds your permissions. By the way, I always use the kubectl auth can-i command to find out what the pod that I am in is allowed to do. Specifically, what I am looking for is the ability to list/get/create pods, as well as list/get secrets.

However, let's assume that I found a privileged pod and managed to create a shell inside of it. A commonly suggested way to break into a cluster is to have a pod mount the root directory of the node it is running on. That allows me to get access to additional configuration files, keys and certificates. (By the way, mounting the root folder without having the privileged attribute turned on in the pod doesn't get me very far, even when running as root.) For example, I can now read the /etc/shadow file to find out what other users may be defined. Or, I can steal the pull secrets that are used to access images in the cluster, for example, in the case of OpenShift, the Red Hat images(!), and any other pull secrets that may be configured to retrieve images (in /var/lib/kubelet/config.json).

Remember that each worker node runs an agent called a "kubelet". This kubelet is also associated with a service account, and even though it has slightly higher privileges than, say, any regular pod (e.g. it can list pods), it is also prevented from creating pods on its own. One way out of this is the use of so-called static pods, i.e. pods that are not under control of the API Server. These pods are based on YAML definitions stored directly on the worker node, so I can insert my own pod definition, which would then run my malicious software and let me further penetrate the cluster, of course.

If I can find (or create) a pod that not only has the privileged attribute set to true but which also has an additional set of attributes set, for example, hostPID or hostNetwork, I can even start a shell in the actual node itself. This allows me to gain direct access to the kubelet. I can try to ssh into other nodes, specifically, the master nodes which host the etcd database. I can run my own software on that node, possibly leveraging the fact that container runtimes and other things are available to me now. Or, I could simply try to delete data and/or destroy the node altogether, if my attack is meant to be destructive.

The key to the castle lies in the privileges a pod has, and the service account it is associated with. Keep in mind that a lot of this (accessing the worker node root directory and files, starting pods, retrieving and decoding secrets that contain passwords, etc) requires that the pod I am using is running in "privileged" mode, or at least that it is associated with a service account that has elevated privileges, and that I am running as the "root" user. If we follow the logic that there will always be a way to get inside a pod, then running this pod with no privileges, not allowing them to run as root, and tying it to a service account that has no rights to the cluster whatsoever is your best bet. OpenShift (as well as probably most commercial Kubernetes setups) has solid default settings, too, which make it hard to break out of a cluster. And having in-house standards that simply prevent the use of elevated privileges goes a long way, at least as the first line of defense.

 

Speaking of standards, the Center for Internet Security (CIS) has published a benchmark for Kubernetes, describing a long list of settings that are recommended to properly secure Kubernetes clusters. I found it very useful as a reference.

 

Needless to say that I am barely scratching the surface here. I hope you'll find that the take away from this is that (a) running software in Kubernetes requires a whole new set of defensive strategies against attacks, and that (b) if you have a solid set of security related rules in place, it is actually not easy to penetrate a cluster.

Photos by FLY:D and Mohammad Rahmani on Unsplash