The 7 Most Important CI/CD Security Best Practices in 2022

Is CI/CD Security Really That Important?

CI/CD pipelines usually need a lot of permissions to do their job. They also need to deal with secrets for applications and infrastructure. This means that whoever can get unauthorized access to your CI/CD pipeline gets almost unlimited power to breach all your infrastructure or deploy malicious code.

Therefore, you should take securing CI/CD pipelines as a high-priority task. Statistics show that there has been a significant surge in software supply chain attacks in recent years. We're talking about an increase of over 400%. Therefore, leaving CI/CD security as a last extra step on your security to-do list is definitely not the best idea. Here are some best practices for increasing your CI/CD security posture.

CI/CD Access

First things first: the access to the CI/CD tool itself. It's pretty straightforward⁠—you want access to your CI/CD to be well controlled and organized. Not everyone in the company should have access to your CI/CD, and even if someone gets access, they shouldn't automatically get access to all pipelines and have all possibilities. SSO and RBAC capabilities are your friends here. Make sure to follow the least-privileged approach. Developers should only have access to the pipelines they need. There is no point in having access to other teams' pipelines. Managers or team leads should probably have access to CI/CD for reporting purposes, but they shouldn't necessarily be able to create pipelines.

Secure Your Secrets

The next tip on our list may sound obvious, but secure handling of your secrets, tokens, and other credentials is crucial in CI/CD. There are secrets that your CI/CD tool itself may need to deploy applications and also secrets that your application needs. There are two main rules here. Firstly, you don't want to pass any secrets in plain text anywhere in the pipeline. Most modern CI/CD tools come with a secret management solution, which means you can securely store your secrets in your CI/CD tool and pass them as environment variables to your pipelines.

Security Scanning As Part of Your CI/CD

Another best practice on our list shouldn't come as a surprise either. You should include security scanning early in the CI/CD process. There are plenty of open-source tools that let you do that, so there's no good reason not to do it. There are a few ways to do security scanning in your pipelines.

The first and the most obvious is static code security scanning. This process reads the code of the application you're trying to deploy and tries to find common security vulnerabilities or signs of malicious behaviors. But that's not the only security scanning you can do.

There is also registry scanning, especially common in the case of deploying Docker containers. Registry scanning scans every image you try to pull into your pipeline.

Last but not least is runtime scanning. In this case, you deploy an instance of your newly built application to a testing environment and run the tests "on the living organism." Combine all three strategies, and you'll definitely improve your company's security posture.

Don't Leave Test Environments Wide Open

Usually, you can deploy to various test environments to test your product. But these test environments are usually also freely available to developers to do some more manual testing. Such test environments might lack the security of staging or production environments. But they are fully working environments, which means if an attacker gets access to it, they may use it as a stepping stone to other places in your infrastructure. Therefore, it's important to secure your test environment so it's just as secure as your other environments.

Clean Up Any Temporary Resources

In addition to testing environments, your CI/CD pipeline may also create temporary resources, like virtual machines or Kubernetes clusters, to run tests. And while test environments are usually always alive, these temporary resources are meant to be created for a single test purpose and destroyed after the pipeline run. But sometimes, we forget about that "destroy" part. And over time, you could accumulate dozens of unused resources, which not only waste money but also pose a security threat.

Imagine a virtual machine that was created months ago and hasn't been patched since. It could have some unnecessary ports open or even some old test applications running. For an attacker, these forgotten resources are a gold mine. Sometimes, these old resources aren't even covered by your firewalls. The solution here is simple: clean up resources you don't need anymore. If you create them from the pipeline itself, don't forget the destroy stage. If you create them manually, create some processes or reminders to help you keep them under control.

Keep Your CI/CD Tool Up to Date

Sometimes overlooked, sometimes even feared, updating your CI/CD tool is not something you want to postpone. Your CI/CD tool will also have bugs and vulnerabilities. If you don't update your CI/CD, you'll be vulnerable⁠, and the aforementioned best practices will go to waste. There is little value in implementing good access management if you leave your CI/CD tool in a version that has a vulnerability that allows an attacker to simply bypass authentication.

Auditing

Last but not least: audit logs. Even with the best security measures, someone still might manage to run a malicious pipeline. And while your security scanning stages should help inform you when your team deploys something fishy, security measures aren't perfect either. And what does an attacker do after successfully deploying their malicious code? They cover their tracks by deleting the pipeline, so you'll never find out something unwanted happened.

Audit logs will help you out in this case. Pipelines can be deleted for various reasons, and it's not something that you want to prevent entirely. What you want is to create an audit log and save it somewhere completely different from your CI/CD system. Such an audit log should give you clear information on who deployed what, when, and from where. If all previous measures fail, an audit log will at least help you find the back doors afterward so you can quickly delete them.

Summary

Securing CI/CD pipelines is a very crucial yet often overlooked task. CI/CD often holds keys to your kingdom; therefore, as we mentioned at the beginning, protecting your pipelines shouldn't be merely an extra task on your security to-do list. The CI/CD security best practices we discussed will definitely help improve your security posture. But don't forget that you're never done with security. It's a constant process as vulnerabilities and threats evolve continuously. If you want to learn more about DevOps Security in general, take a look at our post here.