Continuous integration and continuous delivery and/or deployment (CI/CD) has become a staple within the modern software development landscape, and it is now extending into patch management. The importance of your environment’s security cannot be overstated, nor can the difficulty of maintaining that security. The question then becomes: What exactly is CI/CD and how can you relate this to patching rather than traditional code commits?
Continuous Integration represents the constant patching of test environments and validation of those patches. Continuous Delivery represents the automated deployment of validated patches to production environments, As a whole, CI/CD has shown itself to be incredibly useful even outside of software development, and its methodologies can in fact be applied just as well to include system security and patch management.
In this article, we’ll take a look at practical examples of patching using CI/CD and how it can be applied to your systems.
How CI/CD Keeps Infrastructure Up to Date
A typical environment has a mix of operating systems, primarily Windows and Linux, which both tend to follow the same general series of steps. Translating this process into CI/CD steps can potentially look like the following:
Continuous Integration
- Retrieve a list of patches to apply to your servers.
- Apply those available patches to testing and staging environments.
- Perform post-patch testing (i.e., network availability, services running, etc.).
Continuous Delivery/Deployment
- Patch production systems:
- Deployment strategies could use blue-green or rolling deployments for further control.
- Perform post-patch testing, which could include watching for an uptick in support tickets.
Now, let’s walk through simple examples for both Windows and Linux to see how you can perform some basic scripting to make this work. For these basic examples, we are going to use PowerShell Core since its runtime works on both Windows and Linux. Additionally, these types of scripts can be integrated into many CI/CD systems, where the results then determine the next steps.
Retrieve List of Patches to Apply
Before you apply any patches, you need to determine what patches are available and what software needs to be updated. The below code examples show how you can retrieve patches from both Windows and Linux systems.
|
# Windows PS C:> $UpdateObjectSession = New-Object -ComObject ‘Microsoft.Update.Session’ $UpdateSearcher = $UpdateObjectSession.CreateUpdateSearcher() $Updates = $UpdateSearcher.Search($null)
PS C:> $Updates.Updates | Select-Object Title, RebootRequired, IsDownloaded
Title RebootRequired IsDownloaded —– ————– ———— KB4529935 True True KB4529938 True True KB4529934 True True
# Linux – Ubuntu PS /home/user> $Updates = & apt list –upgradeable | Select -Skip 1 | Foreach-Object { $Result = $_ -Split ” “
[PSCustomObject]@{ “Name” = ($Result[0] -Split “/”)[0] “Update” = $Result[1] “Previous” = ($Result[-1] -Split “]”)[0] } } PS /home/user> $Updates Name Update Previous ——- ——— ———- libssl1.1 1.1.1-1ubuntu2.1~18.04.5 1.1.1-1ubuntu2.1~18.04.4 python3-distupgrade 1:18.04.36 1:18.04.34 sosreport 3.6-1ubuntu0.18.04.4 3.6-1ubuntu0.18.04.3 ubuntu-release-upgrader-core 1:18.04.36 1:18.04.34 |
After the list of patches has been retrieved, you can go ahead and apply those patches, selectively, to the test and staging environments.
Apply Patches to Test/Staging Environment
|
# Windows PS C:> $updateSession = New-Object -ComObject ‘Microsoft.Update.Session’ $updateSearcher = $updateSession.CreateUpdateSearcher() If ($updates = ($updateSearcher.Search($null))) { $downloader = $updateSession.CreateUpdateDownloader() $downloader.Updates = $updates.updates $downloadResult = $downloader.Download() If ($downloadResult.ResultCode -ne 2) { Exit $downloadResult.ResultCode; } $installer = New-Object -ComObject ‘Microsoft.Update.Installer’ $installer.Updates = $updates $installResult = $installer.Install() If ($installResult.RebootRequired) { Exit 7 } Else { $Updates.Updates | Select-Object Title, RebootRequired, IsDownloaded Write-Host “Install Result: $($installResult.ResultCode)” } }
Title RebootRequired IsDownloaded —– ————- ———— KB4529935 True True KB4529938 True True KB4529934 True True
Install Result: 0
# Linux – Ubuntu PS /home/user> $Updates | Select-Object -First 1 | Foreach-Object { & apt install –only-upgrade $_.Name -y }
The following packages will be upgraded: libssl1.1 1 upgraded, 0 newly installed, 0 to remove and 3 not upgraded. Need to get 1300 kB of archives. After this operation, 0 B of additional disk space will be used. Get:1 <http://archive.ubuntu.com/ubuntu> bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.5 [1300 kB] Fetched 1300 kB in 1s (1286 kB/s) Preconfiguring packages … (Reading database … 46489 files and directories currently installed.) Preparing to unpack …/libssl1.1_1.1.1-1ubuntu2.1~18.04.5_amd64.deb … Unpacking libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) over (1.1.1-1ubuntu2.1~18.04.4) … Setting up libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) … |
Finally, before moving on to production deployment, you need to make sure that your services are still up and running and that the patches have not impacted any functionality critical to operations.
Thus, post-test testing is required.
Perform Post-Test Testing
|
# Windows PS C:> $services = @( “Schedule” “wuauserv” “EventLog” )
$results = $services | Foreach-Object { $service = Get-Service -Name $_
[PSCustomObject]@{ “Name” = $service.name “Status” = $service.status } } PS C:> $results Name Status —- —— Schedule Running wuauserv Running EventLog Running # Linux – Ubuntu # Verify that the cron, rsyslog, and httpd services are running PS /home/user> $services = @( “cron” “rsyslog” “httpd” )
$results = $services | Foreach-Object { $result = & service $_ status Switch ($result) { {$_ -Match “is running”} { $status = “Running” Break } {$_ -Match “is not running”} { $status = “Stopped” Break } Default { $status = $False Break } } [PSCustomObject]@{ “Service” = $_ “Status” = $status } } PS /home/user> $results Service Status ——- —— cron Running rsyslog Stopped httpd Stopped |
These Linux examples assume that you are using an Ubuntu-based environment with the APT command available to you. Any distribution should work though and would require inputting the appropriate commands to return the information needed.
Tracking Changes the Hard Way & How CI/CD Can Help
It is important to know what patches have been deployed to your test and production environments. If you are tracking these changes manually, it can quickly become difficult and prone to error. If you have multiple administrators managing the environment, these issues can be amplified, demonstrating how important tracking your changes can be.
CI/CD, with its reliance on automation, can handle this work much more efficiently–not only in terms of tracking and recording the patches applied to your systems, but also when and how they have affected the environment overall. Additionally, in the deployment phase, it’s common to build in approvals in the event that you want to exercise more control and review over the patching process.
With that in mind, how can we build automated patch reports for our Windows and Linux systems? Again, we will be using PowerShell Core for the scripting, as it integrates well into various systems.
Windows Patch Report
|
PS C:> $updateSession = New-Object -ComObject ‘Microsoft.Update.Session’ $updateSearcher = $updateSession.CreateUpdateSearcher() If ($updates = ($updateSearcher.Search(‘IsInstalled=1’))) { $updates.Updates | Select-Object Title, LastDeploymentChangeTime } Title LastDeploymentChangeTime —– ———————— KB4529935 11/10/2019 12:00:00 AM KB4529938 11/12/2019 12:00:00 AM KB4529934 11/14/2019 12:00:00 AM |
Linux Patch Report
|
PS /home/user> & grep ” upgrade ” /var/log/dpkg.log | Foreach-Object { $line = $_ -Split ” “ [PSCustomObject]@{ “DateTime” = “$($line[0]) $($line[1])” “Name” = $line[3] “Update” = $line[5] “Previous” = $line[4] } } | Select-Object -Last 4
DateTime Name Update Previous ———– ———– ———— —————- 2019-11-23 21:03:57 libssl1.1:amd64 1.1.1-1ubuntu2.1~18.04.5 1.1.1-1ubuntu2.1~18.04.4 2019-11-23 21:04:02 ubuntu-release-upgrader-core:all 1:18.04.36 1:18.04.34 2019-11-23 21:04:02 python3-distupgrade:all 1:18.04.36 1:18.04.34 2019-11-23 21:04:06 sosreport:amd64 3.6-1ubuntu0.18.04.4 3.6-1ubuntu0.18.04.3 |
Tying On-Premises and Cloud Together
More and more common today is the combination of both on-premises and cloud environments. Often termed hybrid clouds, these attempt to offer the best of both worlds. Hybrid clouds allow an organization to retain control of on-premises data and servers while allowing the gradual expansion into cloud offerings.
Of course, this type of arrangement can potentially make patching more difficult. For example, your on-premises environment could be based on VMware, whereas your cloud environment could be based within Azure. Both offer different interfaces and abilities.
Let’s take the example of a set of traditional Windows VMware servers combined with Azure Windows virtual machine (VM) instances. How would you report on this combination of environments? Using PowerShell Core as before, you can pull patch reports from both, as shown in the following sections.
Report on Patch Status
|
# Azure VM Windows Patch Status $Params = @{ “Name” = ‘AZWindowsVM’ “ResourceGroupName” = ‘azure-vms’ “ScriptBlock” = { $updateSession = New-Object -ComObject ‘Microsoft.Update.Session’ $updateSearcher = $updateSession.CreateUpdateSearcher()
If ($updates = ($updateSearcher.Search($null))) { $updates.Updates | Select-Object Title, RebootRequired, IsDownloaded } } “Credential” = $AZCredential } $AZResults = Invoke-AzVMCommand @Params # On-Premise VM Windows Patch Status $Params = @{ “Name” = ‘WindowsVM’ “ScriptBlock” = { $updateSession = New-Object -ComObject ‘Microsoft.Update.Session’ $updateSearcher = $updateSession.CreateUpdateSearcher() If ($updates = ($updateSearcher.Search($null))) { $updates.Updates | Select-Object Title, RebootRequired, IsDownloaded } } “Credential” = $Credential } $OPResults = Invoke-Command @Params |
To use PowerShell remoting with Azure VMs you need to use the Azure Cloud Shell, which grants you access to four cmdlets: Enable-AzVMPSRemoting, Disable-AzVMPSRemoting, Invoke-AzVMCommand, and Enter-AzVM. Examples are below.
|
Enable-AzVMPSRemoting -Name ‘AZWindowsVM’ -ResourceGroupName ‘azure-vms’ -Protocol https -OsType Windows Enable-AzVMPSRemoting -Name ‘AZLinuxVM’ -ResourceGroupName ‘azure-vms’ -Protocol ssh -OsType Linux |
Keeping Containers Up to Date
It’s nearly impossible to talk about cloud environments these days without mentioning containers and orchestration platforms such as Kubernetes. How does patching work in an immutable infrastructure environment?
The principle idea behind containers is that they are immutable, such that they are not intended to be changed. Additionally, they are meant to be ephemeral–you can create and destroy them continuously as your demands change.
So what does this mean for patching these containers? Well, you don’t really patch the running containers; you patch the image that the container is based on. Here below, we’ll show what that looks like in a Kubernetes environment.
Update & Validate Container Images
|
# Update all deployment/nginx images to NGINX version 1.17.6 and initiate a rolling update kubectl set image deployment/nginx nginx=nginx:1.17.6 # Verify that the rollout status kubectl rollout status deployment/nginx |
How then does this fit into CI/CD? Just as before, you would build the patching and validation of these containers into your CI pipeline. By applying the patch, spinning up a test environment, and running validations against your containers, this process fits perfectly into your patching CI/CD process.
Docker containers utilize Dockerfiles to define a container’s configuration. By scanning this configuration file for known software versions or operating systems that are vulnerable, you are able to avoid scanning the entire container itself. This can potentially avoid an expensive and lengthy scanning of your containers as well as costly container re-creations.
As Kubernetes itself updates, it’s also important to keep the management platform up to date–not only for the security that updates bring, but also updated functionality to make orchestration of your containers that much easier.
Let’s take a look at a common example of upgrading Kubernetes.
Upgrading Kubernetes
|
# Verify that v1.16.0 is available and the cluster is upgradeable kubeadm upgrade plan v1.16.0 # Perform a dry-run of the upgrade kubeadm upgrade apply v1.16.0 –dry-run # If everything looks good, apply the upgrade kubeadm upgrade apply v1.16.0 |
It may be a bit more difficult to see how the updating of Kubernetes itself fits into this process. But just like any CI/CD pipeline, you can create one that works just the same. By spinning up a new Kubernetes version and testing against a test environment, you accomplish the same level of control and set of approvals that can then be automatically deployed into production.
Conclusion
While there are many different types of systems, tools, and platforms that can be used to manage your infrastructure, CI/CD methodologies and practices can be applied to all of them. Taking control of your processes and procedures can help speed up your organization’s reaction time when patching goes wrong. It also simply helps to be proactive and avoid the next big misstep.
