With the advent of the cloud computing movement, organizations have been shifting to managed infrastructures to offset IT costs. Yet in a complex and ever changing multi-OS environment, patching is not always so easy. Different operating systems and cloud environments as well as the potential of spanned on-premise and cloud environments further complicate the patching process.
Traditionally, DevOps is thought of as a practice specific to coding and development. The same principles, however, can be applied to patching in a modern infrastructure. These proven DevOps concepts can be used to reduce IT burden and failure rate as well as resolve difficulties in patching complex environments.
DevOps Patching Methodologies
So how can the DevOps methodologies be applied to patching? Instead of thinking in terms of code, it is about employing the core principles of DevOps in the context of patching.
Automation
Automation is a crucial aspect of DevOps that facilitates the management of large environments. Not only does the ability to script deployments across multi-OS environments save system administrators a significant amount of time, but utilizing proven and repeatable steps also decreases error rate.
So what does this look like in practice? There are a number of tools that can manage and update your patches across systems, as seen in these basic Ansible playbook examples below. These playbooks can also be plugged into a system that schedules the scripts to run as needed using tools like Ansible Tower or RunDeck.
Ansible
Ansible is a tool that allows patches to be applied to both Windows and Linux systems. For Windows, the win_update module provides functionality for easy patching, as in the following example:
| – name: Install all security updates without a scheduled task win_updates: category_names: – SecurityUpdates |
To do the same in Linux, you would script against the appropriate patch manager there. In this case, here is the yum tool used within Red Hat Linux:
| – name: Update All Packages yum: name: ‘*’ state: latest |
PowerShell
There are a number of different methods to do this in PowerShell. These include the built-in COM objects (Microsoft.Update.AutoUpdate) to interact with the local system, or you can use the Windows Server Update Services (WSUS) cmdlets themselves, UpdateServices.
Monitoring and Reporting
Knowing whether a system needs automation applied to it in the first place requires monitoring. This might come in the form of agents that report on the patch status, Windows systems reporting to WSUS, or cloud systems such as Azure Update Management and AWS Systems Manager Patch Manager. Each system has its advantages and disadvantages, but ultimately they offer solutions that verify that each system checks in and that the patches were applied successfully.
You could also script a solution to do this. In general, there are two approaches here: Have each system report to a central location or have one system reach out and check this on each system itself. For example, either Ansible or PowerShell could be used for either of these tasks.
Report In
With this type of system, you’d most likely want to create an Ansible Playbook or PowerShell script that utilizes the native reporting ability of the OS to determine what patches are needed, have been applied, and their status. Using a scheduled task, a status file or API call (depending on the endpoint) can then be sent to the centralized server to report on the system.
Fan-Out
The fan-out approach follows the same basic principle, but instead of each system having a script running locally and a scheduled task to report the results, the centralized system is used to reach out to each system remotely. Though there is only one point of failure here, this requires far less infrastructure. In this case, the code would be run remotely and then saved to the centralized server to report on.
Ultimately, it may be easiest to use one of the managed solutions or a solution such as WSUS, Red Hat Satellite, or Ubuntu Landscape, which all offer individual OS patch management solutions.
Agile
While practices such as automated building and testing, continuous integration, and continuous delivery came from Agile, they are integral to DevOps as well, and so the two complement one another.
Within the context of patching, for example, this may mean an emergency patch has been released or a zero-day exploit is being actively used, and you are attempting to protect your machines against this. There are many ways to achieve this type of response, such as pushing a patch via WSUS, a one-off install via Ansible (if it’s a cross-platform), distributing the MSP/MSU file itself via PowerShell, or even a DEB file via Bash. Still, it ultimately comes down to being able to respond quickly to an evolving threat.
Ansible, for example, allows you to create an ad-hoc playbook that pushes out only the required patch to every system. This could be further automated with Ansible Tower or Rundeck. The same could be achieved with PowerShell by using WinRM to remotely connect to each system and apply the patch as necessary.
Continuous Integration and Deployment
What is the role of continuous integration and deployment in the patch management world? One of the best practices with patching is staging and controlling the roll-out of patches. With this of course, comes the need for constant research, reporting, and monitoring of those roll-outs. In addition, instead of a monthly patching roll-out, scheduled group patches must continuously be sent to the systems.
Within WSUS for example, this is usually used with Group Policy and Computer Groups to determine patch times and schedules. Within Linux, this would most likely be different host files within Ansible, for example, that run the appropriate updates at the necessary time. By staging out the updates, if something does go wrong, the impact is limited and it’s easier to handle any problematic updates.
Practical Application of the DevOps Mindset
What then does an actual environment setup implementing these procedures look like? Here’s a day in the life of an organization employing these methods:
At the beginning of the day, the system administrator responsible for patch management checks for the latest available updates for both their Windows and Linux systems. In the past, this would have, for the most part, been a manual process. But being resourceful, the administrator has scripted a report to pull the available updates from both Linux systems and WSUS for Windows.
This combined report shows a number of updates that need to go out. Using Ansible to run update scripts to both Linux and Windows, the system admin decides first to push the updates to the evaluation group for testing. Once the updates have been applied, using an Ansible playbook, the admin reaches out to each evaluation system and reports on the status of the installed patches.
In one system, the patch appears to have failed due to a code conflict. The script then sends an automated notification with the failure details to the service owner. This software developer looks into the issue, and when the code is fixed, sends approval to try the patch again. The system administrator then attempts once more to deploy. If all systems report a hundred percent success, the patches are sent to the next deployment group.
Patching like the DevOps do
As in the above scenario, all of these methods and technologies are able to work together to bring automation, feedback loops, and continuous integration in order to create a successful and automated patch management process.
DevOps, as it turns out, isn’t just for development. It works hand-in-hand with the processes across your entire infrastructure. Without a proper patching system in place, you won’t be able to secure your environment properly. Responding to threats quickly, applying best practices, and properly reporting on and monitoring your systems is crucial for a secure environment in any cloud, hybrid-cloud, or on-premise/cloud environment.
To learn more about how to drive forward your vulnerability remediation process, speak with one of our experts today.
