Applications and systems are increasingly more complex in their function, interaction, and form. There is an increasing dependency between resources and applications that can negatively impact operations if not managed and orchestrated in an organized fashion. Effective management and communication of updates, maintenance, and regular releases help to minimize customer impacts. From time-to-time systems require outages for planned upgrades, maintenance, or fine-tuning. Managing these changes is a critical part of providing a stable infrastructure.
The purpose of this policy is to manage changes in a well-communicated, planned and predictable manner that minimizes unplanned outages and unforeseen system issues. Effective change management requires planning, communication, monitoring, rollback, and follow-up procedures to reduce negative impact to the user community.
This policy applies to all Door staff involved in application or systems changes, updates, or patches.
For the purposes of this document and for Door, a change will be defined as anything that transforms, alters, or modifies the operating environment or standard operating procedures of any system or service that has the potential to affect the stability and reliability of the infrastructure or disrupt the business of the Door platform.
Changes may be required for many reasons, including, but not limited to:
- User requests
- Vendor recommended/required changes
- Changes in regulations
- Hardware and/or software upgrades
- Hardware or software failures
- Changes or modifications to the infrastructure
- Unforeseen events
- Routine Maintenance
All system and application changes (e.g. operating system, computing hardware, networks, applications, data centers) are subject to this policy and shall follow unit change management procedures.
The following general requirements shall be met in the change management process:
- Scheduled change calendars and departmental communications operational procedures shall be developed to inform stakeholders of upcoming application and system changes that impact system availability or operations
- Regular planned changes shall minimally be communicated to all stakeholders on a monthly basis through a communication mechanism of the Change Manager’s choosing
- Unplanned outages shall be communicated immediately to stakeholders with regular updates on progress towards resolution and resumption of service
- Regular system and application patching schedules shall be communicated to users and performed in such a way as to minimize system downtime and user productivity
- Changes affecting computing environmental facilities (e.g., air-conditioning, water, heat, plumbing, electricity, and alarms) shall be reported to or coordinated with Facilities and stakeholders shall be notified through change management communications
- Processes shall ensure that production data is not unnecessarily replicated or used in non-production environments
- Device configurations shall be backed up and rollback procedures must exist prior to implementing a change
A Door Change Management Committee shall convene to discuss system changes, interactions, and any perceived issues. This committee shall be made up of network and systems staff, application development owners, developers, Information Security and any other relevant party. The Committee will be chaired by the Chief Technology Officer (CTO) or their designee. The following procedures shall be implemented by the committee:
The committee shall meet on a schedule determined by the CTO but shall minimally meet monthly to discuss plans for future updates and patching.
The following procedure shall be implemented surrounding the change management process:
- Change requests shall be submitted for all changes, both scheduled and unscheduled
- All scheduled change requests shall be submitted in accordance with departmental change management procedures so that the Change Management Committee has time to review the request, determine and review potential failures, and make the decision to allow or delay the system update
- Change requests shall receive Change Management Committee approval before proceeding with the change
- A change review must be completed for each change, whether scheduled or unscheduled, and whether successful or unsuccessful
The CTO or their designee may deny a scheduled or unscheduled change for reasons including, but not limited to:
- Inadequate change planning or unit testing
- Lack of stakeholder acceptance (where applicable)
- System integration or interoperability concerns
- Missing or deficient roll-back plans
- Security implications and risks
- Timing of the change negatively impacting key business processes
- Timeframes do not align with resource scheduling (e.g. late-night, weekends, holidays, or during special events)
All information and activity related to a change shall be logged and maintained within Door’s ticketing system. This log must contain, but is not limited to:
- Date of submission and date of change
- Owner and custodian contact information
- Nature of the change
- Indications of success or failure
- Notes and follow-ons
¶ 5. Audit Controls and Management
On-demand documented procedures and evidence of practice should be in place for this operational policy as part of the Door audit procedures. Satisfactory examples of evidence and compliance include:
- Historical logs of change events
- Archival Change Management Committee meeting minutes
- Anecdotal documentation and communications showing regular compliance with the policy
There are two categories of changes. They can be platform code or infrastructure and of these categories, there are four types: Minor/Routine, Major/Significant, Emergency/Unscheduled and New Development.
All platform code changes will pass through the automated code pipelines. The pipelines has code verification toolsets in-built, automated testing cycles and peer-review practices enforced.
All infrastructure changes are currently performed using IaC (Infrastructure as Code). Code pipelines are peer reviewed and follow the same practices implemented against the platform code changes process.
This section defines the different type of changes:
- Minor/Routine Change: These are changes that may be done at any time as these have been categorised as low risk to the Door platform and the procedures are known and well documented.
- Examples of this type of are:
- Application-based security or business needs patches
- Regularly scheduled maintenance
- Operating system patches (critical, hot-fixes, and service packs) *
- Major/Significant Change: These are classified as needing approval changes and these must be planned in advance, discussed with the team during daily reviews.
- The change request should also suggest a time for this change to take place via the change form before being carried out. The Product Owner will have ultimate say if the change goes ahead at the suggested time or not. Detailed in the change request should be the documentation about what work is going to happen and the perceived benefit and impact to the users. These types of changes should always have a back out plan or mitigating action plan attached.
- Examples of this type of are:
- Change that results in an interruption to a service, or has a significant risk of an interruption to service
- Change that results in a business or operational practice change
- Changes in any system that affect disaster recovery or business continuity
- Introduction or discontinuance of a service
- Operating system patches (critical, hot-fixes, and service packs) *
- Emergency/Unscheduled Change: Unscheduled outages (server crashes, etc.) may require immediate attention whenever they happen. The Change should still be logged, but this could be done retrospectively. Please see the section on Emergency/Unscheduled Changes for more information.
- Examples of this type of are:
- Department or Building is without service
- A severe degradation of service requiring immediate action
- A system/application/component failure causing a negative impact on business operations
- A response to a natural disaster
- A response to an emergency business need
- New Development: This type of change is specifically for the deployment of new features/functionality, services or applications and is not a fix to a problem.
*This appears in both categories as the impact can vary depending on the content of the patch. The Product Owner will be able to provide guidance on which category a particular patch fits into and whether it needs approval before applying.
For at least two days prior to annual leave or leaving the Door platform, a Door member of staff should only be permitted to make Minor/Routine Changes. If there is an urgent requirement to make an Emergency/Unscheduled change during this time, please ensure that there is sufficient documentation provided and colleagues know where to find it and what the change involves.
At certain critical times of the year, it will be necessary to impose a non-essential change freeze period. During this time you should only make changes that are deemed essential to either the running of or fixing of a problem with a particular service. If you have the need to make a change during this time, then please follow the instructions sent out with the change freeze dates. If in doubt, contact the change manager. The dates of any change freeze will be communicated well in advance so that you are enable to plan your work around them.
This policy is to be distributed to all Door staff and contractors using Door information resources.