Kubernetes Security

This blog post covers the security features in Kubernetes and best practices from a security point of view.

1. Secure The Cluster

1.1 Kubernetes Components

Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services. A Kubernetes cluster typically consists of a set of worker nodes and one master node providing the fault-tolerance and high availability. The control plane hosts important components like API server, etcd , kube-scheduler and kubelet, etc., it is responsible for making decisions about the cluster, and pushing it toward the desired state.

In the early days of Kubernetes, the default configuration left the Kubernetes control-plane insecure. Although the default settings have been improving from the security perspective, it is still well worth checking the configuration in your Kubernetes environment.

1.2 Secure API Server

API Server validates and configures data for the API objects. It exposes RESTful APIs over HTTP, performs all the operations and stores API objects into a persistent storage. When you interact with the Kubernetes cluster with command line, you are actually communicating with API Server.

Controlling API server is equivalent of root access on every machine in the cluster.

To protect API sever:

1. Use Transport Layer Security (TLS) for all API traffic

• Secure port

All API communications in the cluster are expected to be encrypted by default with TLS, and the secure port is used to serve https with authentication and authorization. If you disable it, all traffic is served unencrypted. So ensure — secure-port argument (the default port is 6443) is either not set or is set to an integer value between 1 and 65535 and for older version ensure — insecure-port 0 to disable localhost:8080 which requires no authentication and authorization.

• HTTPS traffic encryption

Using — tls-cert-file, — tls-private-key-file to configure x509 certificates and a private key for HTTPS connections.

2. Enable authentication to API Server

Kubernetes uses client certificates, bearer tokens, or an authenticating proxy to authenticate API requests through authentication plugins.

Client certificate authentication is enabled by passing the — client-ca-file argument to API server.

To disable anonymous access, set — anonymous-auth to false.

3. Authorize a request

Authorization checks if a user is allowed to perform specific tasks. There are several authorization modes: Node, ABAC, RBAC, Webhook.

Restrict Kubelet nodes to reading only objects associated with them by using Node as one of — authorization-mode.

Use RBAC to allow different entities to perform different objects in the cluster.

Ensure — authorization-mode argument is not set to AlwaysAllow.

4. Admission Control to validate requests

Admission control guarantees that an authenticated and authorized user performs tasks correctly and securely. The Kubernetes API server uses — enable-admission-plugins to define a comma-delimited list of admission control plugins to invoke prior to modifying objects in the cluster.

Admission control plugin AlwaysAdmit allows all requests and do not filter any requests. So ensure — enable-admission-plugins argument is set, its value does not include AlwaysAdmit. Other plugins can be used to secure cluster in different scenarios.

1.3 Secure Kubelet

Kubelet is the agent on each node to interact with container runtime to launch pods and ensure the Kubernetes created containers are running and healthy.

Some configuration options of kubelet to minimize the attack surface:

1. Enable authentication using certificates

Check — client-ca-file argument, and set it to the location of the client certificate authority file. If the — client-ca-file argument is absent, check Kubelet config file sets authentication: x509: clientCAFile to the location of the client certificate authority file.

Set — anonymous-auth false to disable anonymous access.

Close read-only port by setting — read-only-port to 0 to prevent anonymous users from accessing information of running workloads.

2. Enable authorization to kubelet API

By default, all authenticated requests are allowed. To restrict the behavior ensure — authorization-mode is present and set to Webhook rather than AlwaysAllow.

3. Restrict kubelet permissions

Include NodeRestriction in the API server — admission-control setting, so kubelet will only be allowed to modify their own Node API object, and only modify Pod API objects that are bound to their node.

Include Node in the API server — authorization-mode argument to only allows kubelets to read Secret, ConfigMap, PersistentVolume, and PersistentVolumeClaim objects associated with their nodes.

4. Certificate Rotation

Kubelet uses a client certificate to communicate with the API server, from 1.8 onward, kubelet enables automatic certificate rotation with — rotate-certificates flag, so a new certificate will be requested and issued automatically as the expiry deadline approaches.

1.4 Secure Etcd

Etcd is a highly-available key value persistent storage used to store highly sensitive configuration data. Unfortunately, etcd misconfiguration remains rampant. The same security principles should be applied to it as any other data storage.

1. Encrypt information in Transit

• client-to-server

Set — cert-file and — key-file arguments to ensure SSL/TLS connection with etcd.

• server-to-server

Set — peer-cert-file and — peer-key-file arguments to ensure SSL/TLS connection with etcd.

2. Enable authentication

• client-to-server

Check all in coming HTTPS requests for a client certificate signed by the trusted CA. Enable client authentication on etcd service by setting — client-cert-auth argument to true and specify trusted CA that has signed the client certificate using — trusted-ca-file argument.

Set — auto-tls=false to disallow the generation and use of the self-signed certificates.

• server-to-server

Similar to client-to-server authentication, for server-to-server authentication, set — peer-client-cert-auth to true, specify trusted CA for peer with — peer-trusted-ca-file argument.

3. Encrypt sensitive data at rest

Encrypting etcd’s data stored on disk is especially important if you are storing Kubernetes secrets in etcd rather the external secrets store.

— encryption-provider-config controls how API data is encrypted in etcd. The configuration is provided as an API named EncryptionConfiguration for API server.

/usr/local/bin/kube-apiserver  --encryption-provider-config=/var/lib/kubernetes/EncryptionConfiguration-example.yaml ...

EncryptionConfiguration-example.yaml is structured as below:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - identity: {}
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ==
           ...

Encrypting secrets with locally managed key fails to protect against a host compromise, so use KMS if available for additional security.

2. Authentication and Authorization

How to configure authentication and authorization has been stated in Secure The Cluster -> Secure API server. Only other topics related to Authentication and Authorization are outlined here.

For the API server to authenticate a request, the request issuer needs to possess an identity.

2.1 Service Account

In Kubernetes, service accounts are used to provide an identity for any processes or applications in the container which resides within the pod. service accounts are namespaced resources.

To automate service account management when you create a pod without specifying a service account, the ServiceAccount should be included in — enable-admission-plugins of API server, and not being turned off with — disable-admission-plugins argument.

A service account token is created to access API, and it is mounted as a volume to the pod. Mounting service account tokens inside pods can serve as a avenue to escalate privilege if the pod is compromised. If the workload running in the pod doesn’t need to communicate with API server, set automountServiceAccountToken to false on ServiceAccount or Pod definition to opt out of automounting token.

Alternatively, you can create your own service account and grant particular role to it according to least privilege principle instead of using default service account.

2.2 Role and Cluster Role

Kubernetes uses Role and ClusterRole to manage the identities(users or workload) access to API. A role or clusterRole contains a set of rules to represent different additive permissions (no “deny” rules). When create Role or ClusterRole, avoid the use of wildcards to matches all items. The use of wildcard may result in inadvertently permissions to access new resources.

API server creates a set of default ClusterRole, some of these belong to system:masters group which is hardcoded into API server source code and has unrestricted access to perform any action on any resource.

Membership of system:masters group is substantially dangerous when combined with client certificate authentication mode, as such kind of client certificates can’t be revoked. It should be avoided to make any user a member of system:masters group, grant someone cluster-admin rights instead which will have the same effect as using system:masters group.

3. Running Containers Securely

In order to run containers securely in Kubernetes, you need to do the following:

• Use least privilege to the carry out the tasks
• Do the minimal host mounts as necessary
• Limit the communication between applications, and to and from the outside.

When it comes to security from pod perspective, the chief concern is preventing a process running inside a container from escaping the isolation boundaries and gaining access to the underlying host. There is a variety of attributes in the pod specification can restrict the pod’s capacities.

• Privileged containers have full access to do nearly everything that the host can do. Pods should be prevented from running in privileged mode, and generally container should not be run with allowPrivilegeEscalation flag set to true.
• Likewise, other host resources should be restricted to be shared with container like hostNetwork, hostPID, hostIPC, hostPort, hostPath.
• runAsNonRoot /MustRunAsNonRoot as true or or MustRunAs with the range of UIDs not including 0 prevents Containers from running as non-root users.
• Container allowedCapabilities should be minimized from the perspective of the principal of least privilege, the inappropriate capabilities can be exploited to perform privilege escalation or container breakout.
• The sysctl interface allows an administrator to modify kernel parameters at runtime, if use improperly, it can disable security mechanisms , only safe subset of sysctls should be allowed.
• Containers are not a sandbox, when run untrusted programs in container, the extra isolation provided by Sandboxing can prevent malicious code from harming the host. gVisor, a sandboxed runtime tools, integrates with Kubernetes through runsc handler (short for “run Sandboxed Container”) as RuntimeClass to create a strong security boundary between an application and its host.
• Use CPU / RAM limits to prevent DOS. The values should be the minimum for the containerized application to work. LimitRange and ResourceQuota can be used to manage resource usage of tenant workloads.

For the network communications, by default, there is no limitations for all pods in Kubernetes cluster to send and receive traffic. Network segmentation is important to ensure all pods only communicate with a selection of pods they are supposed to. You can also use network policies to block traffic to the Metadata API in Cloud platform.

All namespaces should have NetworkPolicy to ensure only legitimated traffic is allowed. But the prerequisite of using NetworkPolicy is that the CNI in use must support such a feature. Calico is one of CNIs that support the NetworkPolicy.

4. Container Runtime Security

Container Runtime is the underlying software in each node. It is container runtime’s responsibility to run containers. Runtime security is to detect and prevent malicious activity from occurring inside the container. These are Linux kernel security features that can also be set up via the SecurityContext to provide active protection for containers:

• Secure computing mode (seccomp) is a mechanism which restricts access to system calls by processes. It works by intercepting syscalls and only allowing those that have been allowlisted to pass through
• Security Enhanced Linux (SELinux) is an implementation of fine-grained Mandatory Access Control (MAC).
• AppArmor allows the system administrator to restrict programs’ capabilities with per-program profiles which determine what files and permissions the application requires. Since the building AppArmor profiles is challenging, it is recommended you use a tool like bane instead.

5. Secure Image

In a multi-tenant cluster, the private images should only be used by those who have credentials. To disallow unauthorized users to use the image, use AlwaysPullImages admission plugin of API server. Besides, the AlwaysPullImages can ensure the image with the most recent version that matches the specific tag is obtained.

When you pull the off-the-shelf image from the public registries or internal private registries, you can rely on signing images to ensure the images are genuine and intended ones. For a better solution, try to have kubernetes only pull images from a trusted registry and block all others.

When you build your image, it is important to start with a secure base image from a fully vetted registry in the first “From” statement.

A container image scanner is to inspect the packages included in an image. Some scanner also report on known malware or the presence of sensitive data. Dockerfile and images should be scanned on a regular basis by automated scanners.

6. Secrets Management

If you application code needs access to credentials and other secret information, you have two options: storing secret in etcd or third party stores. The major cloud providers have key management systems, use them for secret information in conjunction with etcd for less sensitive information. Restrict reading and writing the secret with RBAC.

When passing secrets to application code, prefer using secrets as mounted secret files rather than from environment variables as it is common to log out the environment to expose the secrets, and never store sensitive information (passwords, tokens, certificates) in the image.

7. Log and monitor API access events

There are different log sources in Kubernetes. Applications, workloads and Kubernetes components generated their own logs. Ingress traffic is very important source of log. Kubernetes events detail the errors and changes to resource state. Audit logs includes API calls to Kubernetes.

Logs should be rotated regularly to ensure they are stored before deleted. Some logs, like event logs and audit logs, should be stored in a persistent locations outside the Kubernetes cluster. Relying on third party security monitoring tools (Falco, Aqua Enterprise ) on all cluster nodes is an option.

From Infosec Writeups: A lot is coming up in the Infosec every day that it’s hard to keep up with. Join our weekly newsletter to get all the latest Infosec trends in the form of 5 articles, 4 Threads, 3 videos, 2 Github Repos and tools, and 1 job alert for FREE!