Attackers are targeting your continuous integration and continuous delivery (CI/CD) infrastructure. And they don’t just want to disrupt your workflow; they want to stop you from shipping secure code. Think about it: Your CI/CD pipelines are at the heart of your company’s digital transformation efforts. Disrupting these systems can have significant business implications, as they are responsible for releasing new features and updates to customers efficiently.
The consequences of a compromised CI/CD pipeline can be devastating. In this article, we’ll teach you multiple ways to boost your CI/CD pipeline security as well as deployment strategies to help you roll back your new application in case of any major security issues.
What is risk-based security and why does it matter?
Risk-based security identifies and mitigates the most severe threats to an organization. While this approach can be applied to various contexts, it is particularly well suited to the fast-paced world of software development.
Implementing risk-based security measures reduces the likelihood of security vulnerabilities and prevents data breaches. Risk-based security should be an integral part of your CI/CD pipeline to ensure the safety and security of your applications. It can also help you meet compliance requirements and improve your overall security posture.
By adopting a risk-based approach, organizations can maximize return on investment (ROI) and improve performance while maintaining quality standards. It enables businesses to develop strategies best suited to their operating and threat environments as well as to their business objectives.
What is a CI/CD pipeline?
The goal of a CI/CD pipeline is fast and timely delivery of applications and minimizing the risk of human error. CI involves the continuous integration of code changes into codebases that are automatically tested prior to being added to a codebase. Developers use repositories and version control systems to collaborate when working on a specific project.
The continuous delivery (CD) stage, on the other hand, employs automation to deliver and deploy applications to your production environment. In this stage, components are tested before being deployed to production. If the component fails, it will be taken back to the continuous integration phase.
Figure 1: Illustrating security in a typical CI/CD pipeline
Understanding the CI/CD security threats
Your CI/CD pipeline is the backbone of your software development process, so it’s critical to ensure security. Incorporating security into your pipeline is a must for a number of reasons:
- Protect code from attack: Your code is your most valuable asset, so it’s essential to protect it from malicious actors.
- Prevent data leaks: A breach in your pipeline could leak sensitive data, with devastating consequences for your business.
- Comply with policies: Helps to ensure your CI/CD pipeline meets security policies.
- Ensure quality assurance: A secure pipeline will ensure high-quality code that is free of security vulnerabilities.
How to ensure CI/CD pipeline security
As you set up your CI/CD pipeline, it’s important to include security checks at various stages to ensure that your code is secure and compliant with security standards. There are a number of measures you can take to secure your CI/CD pipeline.
1. Planning phase
This stage involves gathering requirements and consumer input to develop a product roadmap. It also encompasses the best practices and policies for a successful DevOps strategy.
You should also take advantage of threat modeling to help identify potential areas of attack and take steps to secure your pipeline. In threat modeling, security vulnerabilities are identified, and countermeasures are determined to mitigate them. By applying threat modeling to CI/CD pipelines, you can identify potential attack areas and take measures to secure them.
Supply-chain Levels for Software Artifacts (SLSA) is also useful during the planning phase. This security framework comprises a checklist of standards and controls to prevent supply-chain attacks, safeguard against integrity challenges, and safeguard software packages and infrastructure in your organization.
2. Coding phase
In the coding phase, the developers write the necessary code to build the software. The code must be written in accordance with predefined standards and design guidelines.
You should use source code scanners such as CAST Application Intelligence Platform (AIP) or CodeSecure to detect pieces of code that might be vulnerable to security threats.
3. Build phase
During the build phase, the developers are responsible for committing their source code to a shared repository. Once the code changes are checked into the repository, builds are triggered, and automated tests are executed to verify if the builds comply with the requirements.
Here are some tips for setting up security checks in your CI/CD pipeline:
- Include a static code analysis tool in your build stage to check the code for common security vulnerabilities and compliance issues.
- Use static application security testing (SAST) tools like SonarQube, Veracode, AppScan, or Codacy.
- Use software composition analysis (SCA) tools such as Veracode, Sonatype Nexus Platform, etc.
- Set up security-related test cases based on organization policies. These tests can check for things like cross-site scripting (XSS) and SQL injection flaws.
- Use a code-signing service to sign your code ahead of deploying the code to the production environment. This will help ensure that the code has not been tampered with and that it comes from a trusted source.
4. Testing phase
Once a build is successful, the software is tested to detect any potential bugs. If new features are added, a new build is generated and regression testing is performed on the new build to verify if the functional tests succeed.
At this stage, you should run container scanning tools (e.g., Datadog, Clair, Anchore, and Qualys) or dynamic analysis security testing (DAST) tools (e.g., Netsparker and Acunetix).
5. Deployment phase
In this phase, the build is deployed to the production environment.
6. Monitoring Phase
This is the last stage in a typical DevOps CI/CD pipeline. During this phase, the build is monitored to ensure that it works as expected. The application deployed in the production environment is observed to evaluate performance and other aspects.
CI/CD security checklist
Black-hat hackers are constantly trying to crack new encryption code and detect security vulnerabilities. These are some of the security threats cyber attackers use to jeopardize your CI/CD pipeline:
- Insecure codebase
- Security misconfigurations
- Using flawed third-party libraries
- Sprawling secrets
Next, let’s look at best practices and methods to boost your CI/CD pipeline security.
1. Code repository access restriction and using audited code
Continuous integration relies heavily on code repositories and version control systems to host the codebase and to facilitate collaboration and sharing. These platforms are useful, but attackers can wreak havoc to your codebase if it has been open-sourced and has poor security.
They simply search for any sprawling secrets and security vulnerabilities that exist in the code. If a security vulnerability exists in the latest codebase version, then it will obviously be present in the live application. Therefore, it is important to encrypt your keys; and most importantly, do not embed your secrets in application code. Instead, use secret managers such as Doppler to secure and safely share your keys.
With so many free tutorials and application source codes available today to suit virtually every application use case, many developers simply copy and paste third-party source codes without scanning and auditing them first. Using source code that is not scanned can lead to flaws in your codebase. The consequences of this “if the code compiles, then leave it’” attitude can be severe. CI/CD security issues may indeed be caused by third-party tools, insecure code, and components that are not audited.
2. Reviewing code efficiently
Code has many aspects and components that have to be analyzed during the code review process. In some cases, the program you need to analyze can have more than 200,000 lines. While this can be overwhelming, it’s important to maintain focus when reviewing code, as this will determine the quality of your code review results. The ability to review code efficiently without losing focus requires analyzing and assessing no more than 400 lines of code per session. This will help you spot bugs and detect mistakes that may be overlooked when your focus and attention have diminished.
Before starting the code review procedure, make sure that the code author has annotated the code to make it readable. Well-annotated code clearly explains why certain code modifications or additions were made and how to use the program. After reviewing the code, make sure that the feedback you give to developers and testers is articulate and constructive. This will enable a deeper understanding of the bugs found in the code and how to fix them.
3. Maximizing testing accuracy and test coverage
Testing tools do a great job of helping you find bugs before rolling out a product. However, developers sometimes unintentionally forget to test certain aspects and components of the code. In some cases, a product is rolled into production with bugs because hidden files were not tested. These bugs are hard to find, as programmers are unaware that some files were not fully tested. Code coverage tools are the main solution if you want to be able to test your application’s functions fully.
Code coverage tools check whether your program has been fully tested by analyzing test reports made by the testing tools to find any untested code or files. After analyzing the code, it will alert you on the lines that have not been tested. The report will provide details on how many functions, branches, and statements have been executed. It will also show you how many lines have been tested.
The code coverage percentage is calculated as: (number of lines tested/number of lines found)*100. Examples of code coverage tools include SonarQube and Codecov. These tools help you focus your attention on the right aspect of the code.
4. Image scanning and repository auditing
Consistency is key when it comes to ensuring a safe and secure environment. Regularly auditing and scanning your images and repositories is crucial, as most of the images used are third-party components. Security can be compromised when images are downloaded from insecure repositories.
Images have to be scanned in production and in development. The docker scan command should be used regularly to scan images to find any existing security issues. These tools are also recommended for scanning Docker vulnerabilities:
- Docker Bench for Security
- Dive
- Inspec
- AWS ECR
- Trivy
5. Implementing safe deployments by using deployment strategies
Your deployments require a strategy that will ensure your application experiences zero downtime and that there is a safe option to perform a rollback in case things do not go according to plan. Effective deployment strategies include:
- Blue-green development strategy: This strategy uses two production environments: a blue production environment and a green production environment. The new version of the application runs in the green production environment, while the old application runs in the blue production environment. If the new application fails, traffic will be rerouted to the old application in the blue production environment.
- Canary deployment strategy: This strategy deploys updates in phases, selecting only a small number of users who will use the new application features. It has a variation called the dark deployment strategy, which rolls updates to users based on the information retrieved from the users’ cookies. If the application does not perform well when it is being used by a small number of users, then it will not be rolled out to the whole user base.
- Recreate deployment strategy: Of all the deployment strategies listed, the recreate deployment strategy is the only strategy that involves downtime. This happens because all pods are replaced at once rather than gradually, in phases. This strategy is only helpful when the application requires serious and major updates.
- Rolling deployment strategy: This strategy deploys pods in a gradual manner. Also, the readiness probe is used to make sure that the pod being deployed is ready to receive traffic, thus eliminating downtime.
Levelling up your CI/CD pipeline security
The concepts and best practices covered in this article are essential for protecting your applications from bugs and security vulnerabilities. In addition, education and keeping up to date on how to implement security policies, security measures, and responsiveness when a security vulnerability is discovered are all key to eliminating security breaches.
The effects of cybercrime, expected to cost companies $10.5 trillion by 2025, can be devastating. Experiencing a security breach can lead customers to lose trust in your software products and website and harm your brand. Keeping your CI/CD pipeline secure will help to maintain trust and thus strengthen your brand.
But securing your CI/CD pipelines alone is not enough to reduce the chances of cyber attacks. You also need to be able to prioritize, analyze, and mitigate risks. The Vulcan Cyber risk management platform helps you with every stage of the cyber-risk management process and integrates with your existing tools for true end-to-end risk management. Get your free demo and start owning your risk today.
FAQs
Who is responsible for CI/CD?
While developers are responsible for constructing and maintaining secure software, security teams also hold the responsibility of guiding developers in safeguarding their software. Hence, developers have the de facto accountability for ensuring software security, but security teams play a crucial role in providing the necessary guidance.
How does security fit into DevOps?
DevSecOps is a combination of cultural values, practices, and tools that integrate software development (Dev), IT operations (Ops), and security (Sec) to improve an organization’s capacity to deliver applications and services swiftly and securely.
Is CI/CD part of DevSecOps?
DevSecOps is a term that describes a collection of practices that bring together development and operations teams with security teams. Its objective is to safeguard the application development process right from the start. A fundamental aspect of DevSecOps is continuous integration/continuous delivery (CI/CD).
Dive deeper
We went into more detail on CI/CD best practices on the Security Weekly podcast. You can catch that here:
