With new vulnerabilities and threats constantly making headlines, frequent patch releases is a must for operating systems. Keeping software and operating systems up to date is critical for ensuring system security.
Microsoft’s mantra is to be diligent in patching. But introducing changes into a system, as in patching, does not come without risks, and problems may indeed arise. How then do you keep your infrastructure up to date while minimizing the risk?
Windows Patching Architecture
Within the Windows ecosystem, patching is generally performed by Windows Update and Windows Server Update Services (WSUS). While the names are similar, there are some key differences in how they function, as explained below. Another available patching architecture is Microsoft Update, a superset of Windows Update that offers packages for Microsoft Office, among others.
Windows Update
Microsoft’s core update service Windows Update releases a number of patches every month on a Tuesday, hence the origins of the rather infamous “Patch Tuesday.” These patches are relevant for all Windows operating system versions that are currently supported. While Windows Update is a free service, the downside is that users have limited control over the settings, such as when and what type of patches to install.
Windows Server Update Services (WSUS)
Frequently, businesses require greater control of the patching process, including what patches are being installed to which systems. To that end, Microsoft offers WSUS. This server-based component allows the user to choose which patches are made available. By directing your managed systems to WSUS instead of Windows Update, you can define exactly which patches each system will receive and create a system to manage their rollout and installation.
Many enterprises use Systems Center Configuration Manager (SCCM) for the management of their endpoints, applications, and also patching. Interestingly enough, SCCM actually uses WSUS in the background. There are additional scheduling and reporting options that SCCM offers above what WSUS does. Depending on your needs, SCCM may be a good fit as well.
Microsoft offers other products such as Intune and Endpoint Configuration Manager. Intune is a cloud service provided by Microsoft, while Endpoint Configuration Manager offers a broader set of functionality than just patching. Though both products offer advantages and may be appropriate in certain cases, there are plenty of benefits to WSUS as well.
In the case of Intune, not everyone prefers a cloud service. Moreover, it lacks some of the features WSUS has. As for Endpoint Configuration Manager, an organization may not need the broader set of functionalities offered, and in any case it would still use WSUS behind the scenes. Ultimately, knowing which tool is right for you depends on your specific needs.
Third-Party Patching
There are a number of third-party companies that offer patching products as well. Similar in functionality to Microsoft products, such as WSUS, these third-party platforms offer functionality that goes above and beyond what Microsoft can deliver. For example, Automox’s platform, has patch management that goes across OS platforms and also allows for program updates and patching as well. Ivanti offers a few products that plug in to existing systems and add abilities such as pre-tested patches and vulnerability scanning as well.
Group Policy
While the Windows registry can certainly be modified to set individual system settings, Group Policy allows for centralized management of your update policies. In a managed environment, having to set configurations on each individual machine would quickly become untenable. Using a system such as Group Policy to push out the settings to all devices at once is usually is the easiest way to achieve this.
Examples of Patching Issues
So why have Patch Tuesdays become so dreaded? As noted previously, the minute changes are introduced to a system, the likelihood that the service will break increases. Windows patching is no different in this respect. Below are some examples of issues caused by patching.
Problems with Boot or Login
A particular Patch Tuesday update led to issues for a number of Windows 7, Server 2008 R2, Windows 8.1, and Server 2012 systems. Certain systems were stuck on the welcome login screen for up to an hour and then continuously froze up. It was later discovered this was due to an unexpected interaction with Sophos and Avast antivirus products.
Failed Networking
In yet another example of how networking can be affected, KB 4480970 and 4480960 patches led to problems with SMBv2 network shares. Users who were also local administrators on the system were unable to access remote shares. The remedy for this was coincidentally applying another patch.
Smaller Risks
In addition to such major risks as failed networking or problems with booting up, other, smaller issues may arise while patching. For example, many updates require a reboot, which can cause downtime and result in lost productivity. Often, this means that these updates must be planned, but this doesn’t always work with business schedules. In addition, updates can often come bundled with others, which can make rolling back a single specific update difficult. So it’s not always easy to manage a single update within a batch of others.
How to Mitigate the Risks
While not a comprehensive list, the below practices are vital to maintaining a controlled patching environment in order to minimize risk.
Research
Understanding patches and the potential issues they can cause is critical, albeit challenging at times. When Patch Tuesday comes around, it’s important to review Microsoft’s list of available patches. Being aware of any potential issues and knowing which patches can help avoid them is key. This can be achieved by monitoring system administrator forums, news reports, and blogs that track these particular patch issues.
Stage Your Rollouts & Incorporate Canary Groups
One of the best ways to avoid problems that can affect all of your systems at once is to stage patch rollouts. One possible way to do this in WSUS is to create multiple computer groups with different installation times, such as days of the week. Using Active Directory computer security groups and Group Policy targeting makes it easy to control which systems get which updates.
To further automate the balance of systems in your Active Directory targeting groups, PowerShell can be used to distribute your systems among the computer groups. Creating an early-adopter or canary group will also help to identify potential issues early on. Employing these methods allows you to balance the rollouts as needed and exercise positive control.
Windows Settings
There are a number of settings available for configuring Windows around updates. These include Automatic Update Detection Frequency, Configure Automatic Updates, Enable Client-Side Targeting, and No Auto-Restart with logged-on users for scheduled automatic update installations.
Setting Automatic Updates to “auto download and schedule the install” is common, as this allows you to define the schedule, which in turn enables control of the behavior. Controlling the restart behavior, such as with “no auto-restart” for logged-in users, makes for a better user experience. Of course, this still requires having a plan in place to handle the updates in the event that your users never restart.
These are just a few ways to allow for a higher degree of control over your Windows patching.
Windows 10 Servicing Channels & Update Rings
To evolve Windows from a traditionally on-prem product, Microsoft introduced Windows-as-a-Service (WaaS). The way that WaaS keeps up to date is through a concept called servicing channels and update rings. Naturally, this added to the complexity of patch deployments. Servicing channels include Semi-Annual, Long-Term, and Windows Insider channels. These channels control how often systems receive patches from Microsoft.
Further control can be exercised via update rings, with designations such as Preview, General, or Critical. Combining both service channels and update rings with a patch management solution such as WSUS or Endpoint Configuration Manager allows for maximum control over your Windows 10 systems.
The multitude of ways that servicing channels and update rings can interact often leads to confusion and challenges in implementing an effective patching strategy. With so many different types of patches and methods of delivery, it can be difficult to know if you’re getting all of the patches you need.
Using a management system such as WSUS, a third-party tool, or Intune will ensure you’re seeing the information you need to in order to make an informed decision. This includes seeing all of the patches necessary, ensuring any conflicts are automatically handled by the patching system, and that only the systems in need of the updates are targeted.
Remediation
Finally, what do you do when something goes wrong? Assuming best practices were implemented and the process was carefully managed, the risk should be minimized. If only a few machines need to be fixed, it is far quicker to do them all at once. Following are a few ways to remediate systems with problematic patches.
If only a handful of systems are exhibiting the behavior, a hands-on approach for identifying the information needed to prevent future issues may make the most sense. Provided your systems are easily accessible—whether remotely or physically—you would uninstall the patch via standard Windows processes to immediately fix those systems. To prevent the issue from recurring, disable the patch from deployment.
This can, however, be tricky, especially if many systems need to be accessed. In such cases, an automation tool may be required to assist with the remediation. For Windows system administrators, the go-to tool is usually PowerShell. Using PowerShell Remoting, you can remotely access the system and send an uninstall command to remove the patch.
If a system is truly inaccessible from the boot, tools such as Intel vPro can be used to remotely mount and boot into the recovery operating system in order to remove the offending or incorrectly patched system files and thus restore functionality. Each scenario therefore requires a different toolset.
Mitigating the Perils of Patching
Patching is a crucial part of maintaining a secure and reliable infrastructure, but it does not come without risks. Creating a robust patching infrastructure and employing best practices will help minimize the risks while keeping your critical systems patched and up to date. In a fast-paced, dynamic computing environment, creating a reliable infrastructure capable of quickly adapting to emerging threats is of vital to the security of your environment.
