• September 10, 2019

The Toppling Tower of Technology Deferred Maintenance

The Toppling Tower of Technology Deferred Maintenance

The Toppling Tower of Technology Deferred Maintenance 1000 667 Vantage Technology Consulting Group

Many institutions are challenged with reduced funding, changing enrollments, and rising operating costs that make competition for departmental funding fierce. Often, out of necessity, important capital expenditures are prioritized lower in order to fund strategic or more visible needs. Like their campus facilities counterparts, information technology can feel a great impact when the capital funding is reduced. The reality is that many institutions find themselves putting off technology capital expenditures, using equipment that is past maintenance support, and even past end of life when equipment is no longer manufactured or supported.

What is technology deferred maintenance?

Facilities departments often track deferred maintenance on campus buildings. This term usually refers to a backlog of building and upkeep projects that are delayed from the normal operating cycle, usually due to budget cycle considerations or lack of funds. In this sense, deferred maintenance includes physical structure updates and maintenance on building electrical, safety, and environmental systems.

However, the deferred maintenance list is rarely comprehensive across the institution’s critical infrastructure. Some higher education institutions also include classroom updates as part of their deferred maintenance projects. Other institutions track a corresponding deferred maintenance list for IT infrastructure. Contributing to the confusion is that deferred maintenance may be funded from specific and multiple sources. This makes it more difficult to understand where the most crucial deferred maintenance risks lie and means that important campus decisions about how to allocate for capital spending can be even more challenging.

Technical deferred maintenance occurs when limited funding precludes reasonable lifecycle planning or investments to ensure that technology architectures and topologies can keep pace with campus needs. Areas of the infrastructure that might be included in deferred maintenance include cable plant, telecom rooms, data centers, network switches and routers, firewalls, Wi-Fi, servers, storage and backup, identity management platforms, and other middleware solutions. Additional technical debt occurs when more of the budget and staff resources are dedicated to maintaining aging equipment and platforms instead of leveraging automations in newer technologies.

Don’t be complacent about technology deferred maintenance

Most institutions have some deferred maintenance—it is a common reality. As technologists, we often see it as a badge of honor to show that we are frugal stewards of institutional funds and innovative and creative in squeezing the last little bit of useful life out of our IT equipment. But sometimes this mindset forces us to develop processes dedicated merely to keeping end-of-life equipment operational, and over time, a once carefully planned architecture resembles a toppling tower. When aging equipment is involved, the failure of a critical component can be felt across the IT department and institution. As we develop more complex ways to keep our equipment running, that complexity creates an opaque view or dependencies, making the probability and consequence (risk) of what should have been a mundane failure far more significant.

Have you become too resigned to the state of your institution’s technology deferred maintenance?

You may be on the path to complacency if:

  • You’ve stopped asking for money because you never get any, or you only ask for capital funding in dire emergencies to deal with crisis situations.
  • You have given up tracking (or never even start tracking) infrastructure components in a lifecycle management plan to note areas that are approaching the end of manufacturer support or the end-of-life.
  • You spend all your time in firefighting mode.
  • You’ve set eBay alerts for spares of key pieces of your infrastructure
  • You have been placing band-aids on and devising workarounds for outdated infrastructure areas for so long that you’re not sure how to unwind that gordian knot in order to move institutional technology operations forward.

Warning: Deferred maintenance can magnify failures

The danger of excessive maintenance is that it could mean that your architecture and/or infrastructure components are heading for a failure that you do not, or due to unnecessary complexity, cannot see coming. The warning signs include:

  1. Some equipment components are at end of supported life and the vendor is no longer addressing vulnerabilities (if they are even in business) and feature issues with security and/or firmware updates. Staying at a last available state for security or firmware updates is a ticking time bomb for service disruptions and security exploitations.
  2. Physical replacements for specific equipment models are no longer available. Sometimes end-of-life equipment is only available used, grey market, or third-party refurbished. Re-purposed equipment can introduce performance issues that are hard to pinpoint in delicately balanced architectures.
  3. The campus is implementing a new software platform that cannot perform adequately with aging infrastructure. Applications are usually developed, optimized, and tested for use on current infrastructure equipment. Older servers, middleware, backup systems and network equipment may not support modern enterprise applications. Even the integration of enterprise cloud software platforms with aging campus equipment and architectures can introduce conflicts, bottlenecks, scalability issues, and unacceptable performance drops.

Take action to address technology deferred maintenance risk

If you suspect you’re in a situation of imminent failure, there are a series of risk conversations that must take place at the institutional leadership level. Infrastructure risks affect the business of the institution and leadership should at least not be surprised if a failure occurs. While it is a challenging conversation, the risk of technology deferred maintenance should be understood, addressed, and/or accepted at the institutional leadership level. Leadership teams must come together to find the right path forward.

You need to take action to help move your institution away from risk due to technology deferred maintenance:

Be ready

Document all your major infrastructure components and key pieces of information about those components. Consider this your deferred maintenance tracking system. Track where each is in relations to the equipment lifecycle; note risks and dependencies for non-supported and end-of-life components. Design a lifecycle plan for current and future IT architecture.

Manage your toppling architecture

Use the deferred maintenance tracking system as a jumping off platform to note dependencies and workarounds made necessary by deferred maintenance and aging equipment.

  1. Documenting workarounds and mitigations will create agility when you finally find yourself catching up on deferred maintenance or replacing aged equipment. It will also assist the team with being thoughtful of the consequences in working with old architectures. Finally, you can use this as an opportunity to facilitate brainstorming on ways to undo some of workarounds while waiting for new infrastructure. Challenge your teams to find ways to get systems in better alignment, address incremental improvements, and follow best practices.
  2. Understand, document, and monitor single-points-of-failure in your aging architecture. Service and performance monitoring are critical when running old equipment, as is capacity planning. Older architectures will be increasingly vulnerable to performance issues, making monitoring more critical for managing disruptions. Baseline and historical monitoring can also be used to watch systems carefully as you integrate new components and transition to new systems.
  3. If you have aging infrastructure due to deferred maintenance, your disaster recovery plan may be pressed into service. Be vigilant with updating and testing your plan to best manage potential service disruptions.

Bring the problem and potential solutions to institutional leaders.

  1. Get stakeholders up-to-speed on the implications of aging infrastructure and deferred maintenance. These conversations are enhanced when you understand how sister and peer institutions are handling infrastructure investments, and how you stack up against them.
  2. Work inter-institutionally and with vendors on creative solutions. Use your situation as an opportunity to move beyond business as usual upgrades, with a strategic mix of shared, cloud and on-premise solutions that meets the future needs of the institution.

Deferred maintenance and the level of aging infrastructure in higher education will lead to more service disruptions across campuses. Digital transformations, complex integrations with cloud/on-premise platforms, IoT devices, and increasing security threats will overburden aging infrastructure and the workarounds that keep components operational. In a myriad of ways, deferred maintenance is holding back technical capabilities and staff resources that could be supporting digital transformations.

Dealing with deferred maintenance is a challenging, but necessary reality for IT leadership. It requires management, communication, and acceptance of risk among institutional leadership while ensuring a thoughtful approach and considerations at the institutional level for services and funding.

This post was authored by Associate Vice President Cathy Bates, who advises clients on technology strategic planning, information security, and initiatives that transform institutional academic, administrative, and research capabilities.