So, it’s time to patch again. Kind of like getting your flu shots – you know it’s good for you, but nobody likes doing it. Let’s do a quick analysis of the challenges that patching poses to your environment, and some of our recommended coping mechanisms
Downtime
The most frequent complaint we hear about patching is that it requires downtime. Either the system needs to be rebooted, a service needs to be restarted, or advanced troubleshooting is required in case things don’t go smoothly. In most organizations, downtime can cause anything from mere frustration to major financial damage – all of which serves to cover the patching process with a big blanket of stress.
Vulcan’s Recommendation – schedule a recurring window for patch installations on the server. It’s also a good idea to determine ahead of time whether the specific patch would require a full reboot, or perhaps just a service reload would suffice.
Asset Management
Since our time is always limited, one must choose the most critical assets to patch first. In order to prioritize correctly, the servers and assets must be carefully inventoried. This is a tough mission for any organization, regardless of its size, and could very well seem impossible for larger enterprises.
Vulcan’s Recommendation – manage your asset inventory using a dedicated software, make sure that criticality tags are in place, and work to fuse the data together.
The Never-ending Quilt
As long as there is software, there will be bugs and security issues. New patches will introduce fixes to the current problems that we face, but they may actually cause new problems too. This is more likely to be the case in new releases and versions, and they will probably require future patches as well. This causes a never-ending patching and upgrading process – and who likes taking on never ending tasks?
Vulcan’s Recommendation – deploying smaller patches that solve very specific issues might be a better course of action. This could be more efficient than installing a full blown patch, and has a smaller chance of introducing new bugs.
Dependency Hell
In many cases, when you upgrade a piece of software, you’ll have some prerequisites – in the form of a requirement to upgrade another package (if not many more). Each package may affect different parts of the operating system – or have its own dependencies. This expands a single upgrade process into unplanned, multiple upgrade processes.
Vulcan’s Recommendation – before upgrading, use the package manager to check what extra packages will be upgraded. This can be done before the upgrade window, and prevent “surprises” later on.
Patching is hard
Linux patching and patching tools require certain professional knowledge such as scripting, which often relies on open source tools that lack professional support. This means that teams must be trained properly in order to ensure the patching goes smoothly.
Vulcan’s Recommendation – Use Linux automation utilities or patching systems to take the technical load off your hands.
Testing
Before bringing a patched system back online, some tests are required to make sure that the system is functioning properly, and that the new version did not introduce any regression or new issues. The challenge that these tests pose are twofold:
- First, they are hard to define, since the implications of the patch are usually unknown until the problem is encountered.
- Second, it’s hard to actually perform the testing. Whether it’s the mundane task of running tests manually or maintaining an automation platform – both cost time and money.
Vulcan’s Recommendation – testing automation is a good practice when repetitive tests are needed – but they do have their limits and require a lot of maintenance. Use cases should be documented and implemented thoroughly in order to achieve a good test coverage.
Package management software
Different Linux distros have different package managers (Yum, dnf, apt-get, pacman, Zypper, Protage), and different package types (rpm, pkg, debs, tar.gz). Each one of these have different repositories, different architectures, and different package versions which causes confusion when trying to manage a multi-vendor environment.
Vulcan’s Recommendation – the best medicine here is avoidance. Try to keep a small variance of operating systems for your servers. This will make the process easier for patching and all other aspects of server management.
Compile Flags
When installing applications, compilation flags are occasionally used in order to add support for non “out-of-the-box” features. This needs to be accounted for when upgrading these packages. When compilation flags are used they need to be handled manually to make sure that the configuration is retained.
Vulcan’s Recommendation – There’s no real solution here other than keeping track of those pesky packages, and look through the notes when performing the upgrade.
Bleeding Edge vs. Stability
Finally, when choosing a Linux distro, recognize that you’re choosing where you want to be on the technology scale. Choosing a more stable approach will probably lead to more stable patches, but will also increase the wait cycle, which might introduce a security risk.
Vulcan’s Recommendation – when it comes to production environments, try to stick to the major distros which will probably get the most community QA, and produce a more reliable package stream.
We hope this sheds some light on the patching process, both in the decision making aspect, as well as in execution.
