On the surface, patch management sounds like a straightforward task. But patching in a production environment means making a change to potentially every device in the enterprise. Let’s take a look at some of the complex challenges of patching production environments and some ways to improve the process.
Why Patching in Production is Challenging
While patching a single workstation with a single patch is relatively easy, things can get pretty challenging in a complex environment. Enterprises often use more than one operating system, with different versions being used in different departments. Compounding the problem of different devices and versions is the increasing adoption of open source applications, many of which are not supported by common patching tools. Put it all together and it means that patching in production requires a lot of hands on management and tracking.
Despite the challenges, it’s crucial that enterprise vulnerability management teams maintain a level of flexibility. Enterprise environments change quickly- whether it’s adding new assets or just updating software versions, vulnerability managers need to stay on top of every new asset and program. Otherwise, by the time they’ve scanned the environment and planned updates, the environment will have changed. That means having a process of continuous monitoring in order to have an effective patching program.
There are many tools available that are designed specifically for patch management. However, these tools don’t cover every application and operating system. For example, in Linux, each distribution can have a different method of handling patches, which can be cumbersome and offer limited control. Without the ability to integrate these various patching solutions, many organizations have turned to configuration management tools such as Ansible, Chef, or Puppet. The lack of a standard solution again leads to a lot of manual work around patches, hardly an ideal situation when patching in a production environment.
If this wasn’t difficult enough, sometimes it’s the patching itself that can cause problems. Patches don’t always work, and the impact of patching in production is quite unpredictable. While testing patches in a testing environment is a wise and recommended aspect of patch management, test environments rarely mimic production precisely enough to leave no doubt that issues from patching in production won’t arise.
How Can We Improve the Process?
Despite all these challenges, there are steps that can be taken to ease the pain of patching in production environments:
Method 1: Only Patch What Matters
One of the steps that can have the biggest impact is prioritizing remediation. By understanding the risks associated with the enterprise’s assets, data, and applications, patching can be prioritized based on what’s most critical for the business.
Katie Moussouris, founder and CEO of Luta Security, also advises, “unless exploit code is released, it is less likely that attackers will use [a] vulnerability versus others that are more easily exploitable.” Patching only what’s really necessary, while continuously monitoring and checking apps, is crucial to ensuring an efficient patching program.
Predictability can be improved by leveraging data that can be found in the public domain. Information such as dependency maps for Linux, version change logs, product advisories, and release notes, can be used to assess the need for a patch and its potential impact.
Method 2: Have a Rollback Plan
As with any change being made in a production environment, no plan is complete unless it includes a rollback plan. As Vishal Kara, the former head of Product at CCleaner warns, “patching without proper rollback plans can be so catastrophic, recovering from the repercussions can be even more challenging and overwhelming than every pre-deployment stage!”
Depending on the patch and the platform type of the asset being patched, a rollback plan should include virtual snapshots, recovery point backups, or some other method of restoring the system to its pre-patch state. Some patches have a built-in rollback feature.
Method 3 : Perimeter Security Tools and Workarounds
Patching isn’t the only option when mitigating risk in production environments. Consider implementing configuration changes and workarounds instead of patching, or adding additional security controls. Firewalls and WAFs can be used to reduce risk of a breach without taking the chance of making a potentially damaging change to your environment.
Method 4: Invest in Automation
Ultimately, manual processes are inefficient from both a cost perspective and a remediation perspective. Automatically prioritizing the vulnerabilities and orchestrating the tools needed to remediate can save time when patching in a production environment. Keep in mind that it’s important for any automation solution to be able to identify which assets are the most critical from a business perspective.
The traditional methods for patching are no longer feasible as they create too many challenges to overcome. There are new methods that address the needs in today’s modern enterprise. A continuous remediation process is necessary to keep up with the constant changes in the environment as well as the never-ending release of new vulnerabilities. Data is available that just needs to be collected and correlated in the environment. Read more in our ebook about automatically prioritizing vulnerabilities according to the real threat level in the environment.