GitHub Actions Security for Actions Runner Controller (ARC) and GitHub-Hosted Runners
Protect Against CI/CD Attacks that Poison GitHub Actions Workflows to Exfiltrate Credentials or Tamper Release Builds
Introduction
GitHub Actions is a CI/CD platform designed to automate your build, test, and deployment pipeline. While GitHub Actions jobs run in the cloud by default, you may want to execute your jobs in your own environment. For these use cases, a self-hosted runner can be employed, although this does require provisioning and configuration of a virtual machine instance. If you already have a Kubernetes cluster, it’s logical to run the self-hosted runner on top of it.
This is where the Actions Runner Controller (ARC) comes into play. ARC, a Kubernetes operator, orchestrates and scales self-hosted runners for GitHub Actions. It presents a simplified way to set up self-hosted runners for GitHub Actions. However, securing the cluster and ARC CI/CD Environment remains the customer’s responsibility.
GitHub Actions have the potential to execute third-party code, which may not always be trustworthy, in a privileged environment. Consequently, a compromised or poisoned workflow can lead to exfiltration of CI/CD credentials or tampering of release builds.
If you are looking to secure your Actions Runner Controller (ARC) environment, get in touch with StepSecurity.
To illustrate this, recall the Codecov incident where CI/CD secrets were stolen from thousands of build servers over a period of two months without detection. Or consider the SolarWinds incident where a source code file was tampered with during the CI/CD pipeline process to inject a backdoor.
In this article, we’ll guide you on how to protect your ARC environment against such CI/CD attacks by employing StepSecurity’s GitHub Actions Security Platform.
Implementing Runtime Security for Actions Runner Controller (ARC) Environment
StepSecurity is a Kubernetes-aware security observability and runtime enforcement platform, purpose-built for ARC. It employs eBPF, thereby allowing for reduced observation overhead and real-time enforcement of policies to prevent CI/CD attacks.
Security Observability with StepSecurity
Upon deploying StepSecurity on your ARC Kubernetes cluster, you gain comprehensive security visibility across all your GitHub Actions workflow runs. Remarkably, it achieves this without necessitating any code or repository alterations. This enables insights into the network and file events associated with each step of the GitHub Actions workflow runs. For example, the screenshot below depicts outbound network calls made by each step of a workflow run. Clicking on the step in the dashboard takes you to the build log for that step.
The StepSecurity dashboard enables you to click on any workflow run and view the security insights for that run.
Preventing Code and CI/CD Credentials Exfiltration
A compromised GitHub Actions workflow, whether due to a breach of a developer credential or a compromised dependency or build tool, can be used to exfiltrate code or CI/CD credentials. These credentials are typically high privileged and may grant an attacker access to your cloud environment.
The insights page for each workflow run offers a recommended policy to block outbound traffic to permitted destinations. This policy is based on the outbound calls previously made for a given workflow. Workflows generally download code, dependencies, and publish artifacts to the cloud, hence the outbound traffic from each workflow is predictable and seldom changes.
You can enforce this recommended policy in your workflow by adding the StepSecurity Harden-Runner GitHub Action. Once this is done, any outbound calls not included in the allowed list will be blocked. Plus, you will also receive a notification for such block events. The policy operates at the DNS (Layer 7) and Network (Layers 3 and 4) layers, meaning you’ll see a list of approved endpoints based on domain names and ports where the traffic should be allowed.
Here’s an example where we simulate the Codecov breach in a workflow with a block policy:
As evidenced by the workflow run and insights page, the outbound call was effectively blocked. This critical feature aids in preventing CI/CD attacks in your ARC environment.
Check out a practical demonstration of this in the StepSecurity GitHub Actions Goat project, which is a purposely vulnerable GitHub Actions Environment designed to facilitate learning about GitHub Actions Security.
Here is a workflow run on a self-hosted ARC environment with a block policy. Here you can view the security insights for the run. As you can observe, the outbound call that was not in the allowed list was successfully blocked.
Detecting Tampering of Release Builds
If source code files or artifacts are tampered with during any of the GitHub Actions workflow jobs, you will spot an event for it in the insights view. Overwrites of files are not expected in a standard release build. Choose to be notified of these detections via email or Slack. Then, investigate whether the overwrite is a false positive or warrants further examination.
To illustrate this, let’s simulate the SolarWinds incident where a source code file is tampered with during the build. The screenshot below demonstrates how this is detected:
You can see a real example of this in the StepSecurity GitHub Actions Goat project. Here is a workflow run on a self-hosted ARC environment being monitored by StepSecurity, and here you can view the security insights for the run. As you can see the suspicious file event has been detected without making any changes to the workflow file.
Runtime Security for GitHub-Hosted Runner
While this article focusses on runtime security for ARC Environment, the Harden-Runner GitHub Action provides the same set of features for GitHub-hosted runners as well. This means that you can easily change from GitHub-hosted to ARC runners, and the runtime security will work seamlessly across the two environments.
Harden-Runner GitHub Action is being used by over 1,200 open-source projects, including projects from industry giants like Microsoft, Google, and DataDog. You can explore how these projects are using the Action in the Harden Runner GitHub Action repository.
Conclusion
We’ve examined the key threats in an Actions Runner Controller (ARC) Environment — the exfiltration of code or CI/CD credentials and tampering of release builds. We then explored how Runtime Security can be implemented for the ARC cluster using StepSecurity. Lastly, we saw that the same solution can also be used for GitHub-hosted runners.
If you are interested in securing your ARC environment or wish to use it for your GitHub-hosted runners, learn more about StepSecurity’s GitHub Actions Platform at https://stepsecurity.io.
