Kubernetes Security Policy Enforcement Using OPA

Published in

InfoSec Write-ups

10 min readJun 20, 2022

1. Policy as Code via OPA

Kubernetes is a well-known orchestration engine which can automate the deployment, management and scaling of containerized workloads. Cloud-native containers are being used on a large scale, making it imperative to secure Kubernetes environment.

With the ever-changing technology and security landscape, various institutions offer standardized frameworks and guidelines to help administer dynamic Kubernetes ecosystem security.

There are a variety of security frameworks available for Kubernetes, such as CIS, NIST, and MITRE ATT&CK. But how to make Kubernetes ecosystem adhere to the handbook-based security guideline and regulation might be a challenge for Kubernetes professionals. Not everyone has the bandwidth to work through the details of these security guidelines and determine which ones are most important to implement with the limited resources. Police-as-code comes into play to help with enforcement of compliance standards. This rational behind this approach is to represent the policies in a form of programming languages such as Yaml or Rego, and utilize policy-as-code enforcement engine to ensure policies are met across the environment.

In Kubernetes system, when interacting with API server by sending requests, there are three steps before the API requests access the system: authentication, authorization and admission control.

Admission controllers in the third step are plugins for the Kubernetes orchestration system to govern and enforce how the cluster is used. It acts as gatekeeper, interpreting the API requests after it is authenticated and authorized, altering it or checking it against predefined policies to reject or allow it before it enters the cluster.

Open Policy Agent (OPA) is one of admission controller plugins. With the aid of it, Rego language can be used to implement policy-as-code. It administers policy enforcement across the different systems, pre-check whether requests violate the policies or restrictions. Just as other admission controller plugins, OPA ensure the polices are met when workloads are created, updated and deleted.

This document gives introduction about OPA along with some simple use cases, and how to integrate OPA as an Admission Controller into Kubernetes system.

2. How does OPA work

OPA is a policy engine which allow users or other systems to query policies to make decisions on data fed as an input. Users define policies using Rego, with these policies, Rego inspects and transform data input into structured documents, evaluates whether it complies with, or violates custom policies.

A policy can be thought of as a set of rules. Thanks to OPA, policies can be shared between different applications, distributed from a central location, isolated from application business logic.

OPA can be embedded as a library, deployed as a daemon, or simply run on the command-line, the easiest way to wrap your head around OPA is to experiment with different policy creation using the Rego Playground.

Before dive into simple OPA use cases, it will be helpful to introduce some core concepts for OPA.

2.1 Policies

Policies are written with Rego, each Rego file defines a policy module using a collection of rules which can be used to evaluate certain state or input.

One simple policy file example:

package example

allow[result] = true {
    input.user.role == "HR"
    result = "allow HR"
}

Don’t worry if you don’t understand the syntax here, more on it will be introduced later.

2.2 Rules

Rules are if=then logic statement. Rules expect their inputs to come from the input object. User can query the value or decision of any rule loaded in OPA using an identifier which is an absolute path hierarchically starting from the root data node : data.<package-path>.<rule-name>. Please pay attention to the root data node, it is an important component in OPA.

Rules have a head and a body.

The “result” is the name of the rule, the “action” and “username” inside the head define a object structure as a value of rule formatted as {“<value of action>”:”<value of username>”}, part of the rule query response will be like:

{
    "result": 
    {
         "<value of action>":"<value of username>"
    }
}

Rules can either be “complete” or “partial”. For partial rules, there are Set Generation rule which returns a Set as a value for the rule in the query response, and Object Generation rule just like the example above.

Note: Rules are supersets of functions, don’t confuse Rule with Function, the later is a Rule with arguments. One function example:

result(str) = true {
    str == "GET"
}

2.3 Documents

Document is the primary unit of data in OPA. OPA performs the reasoning around information represented in structured documents like JSON. There are two types of documents in OPA: Base Documents and Virtual Documents.

Base Documents

Base Documents contain static, structured data stored in-memory. The Base Documents can be used to describe the current state.

Virtual Documents

Virtual documents are computed by evaluating the rules included in policy modules. Rules can generate structured data, commonly based on other rules, data or built-in functions available to the rule that was queried.

Data Documents

Data is a global variable, data document branches out in various layers like a tree, both base documents and virtual documents can be accessed using data as a root in the hierarchical structure, you can query them via the /v1/data HTTP API like /v1/data/foo or identifier like data.foo .

Input Documents

In some cases, policy expects input value from the input document when users query rule decisions. To supply input value, use a POST HTTP request for OPA query with input in the request body.

Document API

The Data API exposes endpoints for reading and writing both base and virtual documents in OPA.

One of common use cases is to query the rule via Data API, you can use /v1/data/<package-path>/<rule-name>, if the rule expects input object, then create a POST request with input as request body, a computed virtual document will be returned.

The chart in the below explain how a rule is invoked with Data API.

3. OPA Use Cases

OPA can be running in different deployment environments. You can interact with OPA directly on your own machine using command lines or run it as a server, interact with it using HTTP requests.

3.1 Interact with OPA using command line

It is convenient to download an OPA binary from Github releases, and interact with it via the command line. In some cases, if downloaded OPA can’t work properly on your local machine, using docker container to run OPA inside might be an easier option.

Prepare a OPA directory under the current directory on the host and go inside the directory and download an OPA binary and change mode for the binary.

> cd OPA

# choose v0.37.2 in order to run it inside ubuntu:18.04 container
> curl -L -o opa https://github.com/open-policy-agent/opa/releases/download/v0.37.2/opa_linux_amd64

> chmod 755 ./opa

The unbuntu 18.04 container is chose here, run the image and mount $PWD/opadir from host as /opadir inside the container:

> docker run -it -v $PWD/opadir:/opadir -p 8181:8181 ubuntu:18.04

> cd /opadir

After navigating to /opadir directory in the container, you can see OPA binary reside in the current directory.

The complete OPA CLI list can be found here. We only use opa eval to demonstrate how to use opa CLI based on the core concepts outlined above.

Rego query can be evaluated with opa eval sub command, you can use --data flag to set policy or data file(s), recursively load all *.rego, *.json or *.yaml files under the specified directory or load each of them individually.

for example:

policy1.rego

package opa.examples

import input.example.flag

allow_request { flag == true }

input1.json

{
    "example": {
      "flag": true
    }
  }

run OPA as below:

> ./opa eval -i input1.json -d policy1.rego 'data.opa.examples.allow_request'
{
  "result": [
    {
      "expressions": [
        {
          "value": true,
          "text": "data.opa.examples.allow_request",
          "location": {
            "row": 1,
            "col": 1
          }
        }
      ]
    }
  ]
}

The "value": true in expressions block in response refers to the value of the OPA query for the rule of data.opa.examples.allow_request.

3.2 Interact with OPA as a server

opa run command starts an instance of the OPA runtime, it can be interactive shell or a server.

To run a server, use opa run --server sub command line locally or inside a docker container.

docker run -it -v $PWD/opadir:/opadir  -p 8181:8181 openpolicyagent/opa  run --server --log-level debug

By default, OPA server listens for HTTP connections on 0.0.0.0:8181. Browse http://localhost:8181, if everything works fine, you can see the GUI like below:

If you intercept the traffic when click Submit button, you will find that a POST /v1/query request is sent out to backend server to allow you to execute an ad-hoc query and return value of the query. There is an simple example to use the GUI to execute query based on Input Data.

Besides the /v1/query request sent from the GUI, you can interact with OPA server with data API mentioned in the aforementioned Document API section. When the server starts, there is no data and policies, so the first thing to do is to push policy files to the server using /v1/policies endpoint with PUT request. For example

> curl -X PUT http://localhost:8181/v1/policies/policy-1 --data-binary @policy1.rego

The policy1.rego is the same as example above. policy-1 is the policy name stored in OPA server.

Check whether the policy has been pushed successfully by executing the below, and you will get the raw data of policy-1:

> curl http://localhost:8181/v1/policies/policy-1
{"result":{"id":"policy-1","raw":"package opa.examples\n\nimport input.example.flag\n\nallow_request { flag == true }\n","ast":{"package":{"path":[{"type":"var","value":"data"},{"type":"string","value":"opa"},{"type":"string","value":"examples"}]},"rules":[{"head":{"name":"allow_request","value":{"type":"boolean","value":true}},"body":[{"terms":[{"type":"ref","value":[{"type":"var","value":"eq"}]},{"type":"ref","value":[{"type":"var","value":"input"},{"type":"string","value":"example"},{"type":"string","value":"flag"}]},{"type":"boolean","value":true}],"index":0}]}]}}}[

Create a new input file: input2.json. The difference between previous input1.json and input2.json is that input2.json wraps the content of input1.json into “input” block.

{
    "input": 
    {
        "example": 
        {
            "flag": true
        }
    }
}

Query the allow_request rule decision against input2.json:

> curl -X POST http://localhost:8181/v1/data/opa/examples/allow_request  -d @input2.json   -H 'Content-Type: application/json'

{"result":true}

4. Run OPA as Kubernetes Admission Controller

Now it is time to get back to the track, and use OPA for Kubernetes security police enforcement. OPA is a general purpose policy engine which automates and unifies the implementation of policies across various IT environments. Kubernetes Admission Control is one of OPA’s use cases.

OPA can be integrated with Kubernetes as an admission controller directly or using OPA Gatekeeper.

OPA Gatekeeper was created to enhance and facilitate the integration of OPA into Kubernetes. You can find the installation guide from the Gatekeeper website, but in some cases, to reuse Rego rule files, integrating OPA directly into Kubernetes system can be taken as an easier solution. The integration process of OPA into Kubernetes directly is lengthy, so this part is skipped here. It is elaborated on OPA website.

There is two things need to point out about the integration guide on OPA website.

The first one is when serve the OPA bundle, it might cause some trouble to use the command provided in the website:

docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest

The OPA bundle served this way couldn’t be accessed successfully via http://host.minikube.internal:8888/bundle.tar.gz specified with flags when start OPA server in admission-controller.yaml.

- "--set=services.default.url=http://host.minikube.internal:8888"
- "--set=bundles.default.resource=bundle.tar.gz"

The failure of the bundle file service will cause the OPA container not to run successfully inside the pod.

Instead you can use Python SimpleHTTPServer to serve the bundle file.

The second one is when combine multiple policies, the guide defines main.rego, but the main.rego might not work in some cases. For example, to execute the following block-wildcard-ingress.rego against disallowed_wildcard_host.yaml, the OPA doesn’t block the creation of the ingress as expected unless two ingress rules are swapped. These two files are provided as below, and the original files can be found in gatekeeper-library repository in github

block-wildcard-ingress.rego

package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Ingress"
  hostname := object.get(input.request.object.spec.rules[_], "host", "")
  contains_wildcard(hostname)
  msg := sprintf("Hostname '%v' is not allowed since it is a wildcard.", [hostname])
}

contains_wildcard(hostname) = true {
  hostname == ""
}

contains_wildcard(hostname) = true {
  contains(hostname, "*")
}

disallowed_wildcard_host.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wildcard-ingress
spec:
  rules:
  - host: '*.example.com'
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: example
            port:
              number: 80
  # Extra test to ensure the rule still detects invalid hosts in files containing valid hosts
  - host: 'valid.example.com'
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: example
            port:
              number: 80

There are good use cases on the OPA website about how to use OPA to block incompliant operations inside Kubernetes system based on the bundle of Rego rule files.

The input object in Kubernetes system is a reserved global variable whose value is equal to the Admission Review object. The API server takes this object and provides it to any admission control webhook.

For example, in ingress-allowlist.rego file on OPA website, inside “deny” rule, you can find input.request.kind.kind == “Ingress”, which is an use case of AdmissionReview object.

The complete input object for OPA in Kubernetes looks as below:

{  
 "input": {
    "apiVersion": "admission.k8s.io/v1",
    "kind": "AdmissionReview",
    "request": {
      "dryRun": false,
      "kind": {
        "group": "",
        "kind": "ConfigMap",
        "version": "v1"
      },
      "name": "ingress-controller-leader",
      "namespace": "ingress-nginx",
      "object": {
        "apiVersion": "v1",
        "kind": "ConfigMap",
        "metadata": {
          "annotations": {
            "control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"ingress-nginx-controller-cc8496874-6f992\",\"leaseDurationSeconds\":30,\"acquireTime\":\"2022-06-18T04:43:42Z\",\"renewTime\":\"2022-06-18T14:56:30Z\",\"leaderTransitions\":0}"
          },
          "creationTimestamp": "2022-06-18T04:43:42Z",
          "namespace": "ingress-nginx",
            ...
        }
      },
      "oldObject": {
        ...
      },
      "operation": "UPDATE",
       ...
    }
  }
}

You can follow the OPA logs to get the complete webhook requests:

kubectl logs -l app=opa -c opa -f

Thanks for reading, wish this blog can help you working with OPA.