Testing patches before rolling them out is essential.
Depending on the criticality of the system you’re patching, the amount of testing you undertake can vary.
If you have access to a non-prod (non-production) environment, you should ensure patches are tested there. This is where you can undertake extensive testing of systems, integrations, performance, staging and more, in ways that closely resemble your live, or production, environment.
If possible, you can also choose to test patches in SIT or UAT environments.
A SIT (System Integration Testing) environment is where you can verify that all related systems within the same environment are able to maintain data integrity and operate in coordination with each other. A UAT (User Acceptance Testing) environment is when an end-user performs tests to ensure the system works correctly before going live.
Not all organisations have access to non-prod, SIT or UAT environments. In such circumstances, you can still undertake ‘pilot’ or ‘canary’ testing.
‘Pilot’ testing involves rolling-out the patch to a select group of people in your organisation. This allows you to test how the patch affects other systems before extending it to the rest of the organisation. End-users are aware that you have updated their systems with a patch, so they can report any functionality difficulties they experience in the hours and days after the patch is deployed.
Alternatively, you can also conduct ‘canary’ testing. It is similar to ‘pilot’ testing, except the end-users are unaware that the patch has been deployed. In both testing modes, end-users should be selected that have lower levels of privileges so any negative consequences are less likely to affect critical systems.
Whatever type of testing you conduct, you should test rigorously. Make sure you test all the functionalities of all the applications thoroughly before rolling the patch out to the live environment.
This level of testing is essential to ensure the patches do not inadvertently result in unforeseen consequences that could damage systems your organisation relies on.
Whilst undertaking testing, you will be able to determine whether computers will require rebooting, and whether this will occur automatically or not. This is important to know so you can plan when to roll out patches. Allowing for a maintenance window during which you apply patches in the production environment will help you avoid unexpected reboots that could lead to downtime, or possibly damage databases.
Bear in mind that the majority of patches will require computers to reboot.
Another consideration as you’re preparing to start patching is whether you need to follow rules-based IT workflows, also known as Change Management procedures. These can include obtaining authorisation from management prior to patching. They should be followed whenever the addition, modification, or removal of anything could affect IT systems.
While an organisation probably wouldn’t be expected to follow strict Change Management procedures every time a minor patch is rolled out, they should when the update is substantial or risks business as usual operations.
The objective of Change Management procedures, such as those established by ITIL (Information Technology Infrastructure Library), is to ensure that rules are established and followed so changes to IT infrastructure are efficient and reduce the likelihood of any potential problems.
In addition, make sure you have your redundancy plans in place. This includes ensuring all data is backed-up. In the event an update does result in any negative side-effects, you will have the ability to roll-back, or uninstall, the patch. You will also be able to retrieve lost data. It is recommended that you document any anticipated changes to your IT environment that will occur as a result of the patching. This will help you in the event of any unforeseen problems resulting from the deployment.
Importantly, make sure you notify people within your organisation that you are planning to roll-out patches and how this could affect the systems they use. They should be advised to save and backup any work. Other people in your organisation can also help you notice any performance issues or problems once a patch is rolled-out.