Organizations that rely on third-party technology seldom consider what happens when it fails. Blind faith in software vendors can be costly, so having a solid “plan B” is critical for business resilience. Here’s how to find the right balance.
With technology at the core of most today, companies are reliant on third-party cloud providers and software vendors like never before. The benefits are clear: cloud and software-as-a-service solutions allow companies to scale in line with their growth while minimizing issues with employee turnover and knowledge gaps — common weak points in an organization’s technology environment.
Despite the advantages of outsourcing key technologies, there are business risk decisions that must be made. Events surrounding the debilitating CrowdStrike incident underscore the severe impact to organizations created by unanticipated third-party vendor failures. The takeaway? At the end of the day, technology is ultimately reliant on people, and the human factor continues to be the greatest risk to reliability. When — not if — a vendor’s processes fail, it can perpetuate its way right into your business.
Despite the advantages of outsourcing key technologies, there are business risk decisions that must be made.
Business resilience planning is critical
Are you putting blind trust in your third-party vendors? If so, what’s the cost of doing so? Answering these questions requires a conversation around impact analysis — looking at the potential business vulnerabilities created by your vendors and deciding how much risk you’re willing to push out to them.
Three things that every CEO and CFO should do right now
As the CrowdStrike incident illustrates, industries and organizations that thought they were invulnerable to technology interruptions learned they didn’t have business resilience where they should. Don’t let this happen to you.
1. Assume responsibility for evaluating business impact before trouble strikes
The critical first step is for executives to acknowledge that business resilience is a major issue and give it the attention it deserves. Many CEOs and CFOs assume resilience is an IT issue and don’t give it another thought. But the reality is only the business can determine the true impact of a technology disruption, what drives each form of recovery, and who the right people are to fill key roles.
The reality is only the business can determine the true impact of a technology disruption.
2. Conduct a business impact analysis
Understand where your true weaknesses are in terms of business continuity and recovery from third-party technical outages. This requires an organizationwide business impact analysis to understand how long you can be down, and how quickly you need to recover in order to continue business without truly affecting your customers or your financials.
Key steps in the analysis should include:
Identifying critical systems. A system is critical if it has the potential to take down the entire business.
Determining the impact and scope of reach if each of these systems goes down. What cascading effect could it create across your systems and processes?
Establishing the maximum allowable downtime before there’s a severe business impact.
The answer to many of these questions must come from those in the organization who understand the business, how the processes work, and can understand whether, how, and for how long the business can be driven without a given piece of technology.
The final part of the analysis is documenting your gaps and deciding who should own each one.
When setting up an impact analysis, many organizations use a “responsible, accountable, consulted, and informed” (RACI) chart to assign responsibility and accountability for the outcomes. This format forces an evaluation of who’s responsible for each step, who’s a contributor, who’s informed along the way — and most importantly — who’s accountable for the outcome.
It can’t be emphasized strongly enough: business resilience is the responsibility of the CEO and CFO, full stop.
3. Plan for resiliency
With the impact analysis in place, build out a plan for business continuity. The plan should be built on a business level, based on how you’ll continue your business during each type of disruption. Consider the following:
What’s the ease of implementing a workaround? Not all systems and business processes are created equal. In some cases, you may be able to work offline following a simple process. In other situations where you rely on technology to keep a dynamic process moving, there may be no straightforward offline alternative.
Who should own the gaps that come out of your business analysis? That’s the “R” in your RACI chart. If they’re technology gaps, IT can be responsible for them, but they should report to the business owners who are ultimately accountable for filling the gaps.
What resources will you need to achieve the necessary resilience? Financial decisions around the gaps will require risk/investment decisions.
Categorize potential disruptions based on risk. There’s a spectrum, and you’ll need to find a balance. If a disruption puts millions of dollars at stake, then doing nothing is completely unacceptable. Alternatively, if a lower-level risk requires an investment of millions of dollars to alleviate, you may decide to take a chance and see how it plays out. The important thing is you understand the scenario, communicate around it, and then accept it. In many cases, not having those conversations is what gets organizations into trouble.
Address the most impactful areas first, then work your way through the list as time and money permits. Set realistic expectations in light of feasibility and the available budget. The key to success is prioritizing and focusing your investments based on risk and getting the “biggest bang for the buck.”
How to build business resilience
When building out your organization’s business resilience, consider the following:
Responsibility lies at the top. It can’t be emphasized strongly enough: business resilience is the responsibility of the CEO and CFO, full stop. For too long, executives have thrown business continuity over the fence to IT. But in reality, IT is there to support resiliency through disaster recovery efforts, but business processes are the responsibility of operating departments. If the business leaders aren’t driving business impact analysis and subsequent resiliency efforts, and working closely with IT to support business operations and understand risks, you’ll likely fail.
Know your vendors. Learn about and understand your vendors’ processes. Is their QA process standardized? Do their teams follow the policies and procedures in place?
Understand what you can regulate in your own environment. With some cloud solutions, you can set up your own environment to test and control updates before pushing them out to the organization. In some situations, vendor updates are automatic and beyond your control. When you can’t prevent updates from being pushed out or installed in your client environment, your risk increases exponentially.
Don’t put all your eggs in one basket. Many times, an organization will go all in on a single provider because it’s the cheapest option. But sometimes you can spread risk by investing in different systems and different providers for different business processes. By doing so, you may be able to keep running in times of trouble.
Read the fine print in your vendor contracts. People often think there’s unlimited liability for vendor mistakes and are surprised to learn they don’t have the legal recourse they think they do. Vendor contracts build in a balance of how much each party is responsible for; make sure you understand this when considering the financial implications of downtime.
Make your plan a living document. Develop, periodically test, and update as needed a business continuity plan to maintain operations during a disruption; a disaster recovery plan to resume operations after a disruption based on the business’ maximum tolerable downtime and data loss; and an incident response plan to coordinate actions in response to a cybersecurity incident.
The answer to many of these questions must come from those in the organization who understand the business, how the processes work, and can understand whether, how, and for how long the business can be driven without a given piece of technology.
Responsibility can’t be outsourced
Should you put blind trust in your third-party vendors? If you do, what are the potential costs of doing so? Have the conversation, then decide whether it makes sense to invest in the needed resources to prevent or circumvent downtime or just take the loss when third-party tech goes out. The bottom line is you can outsource your people and processes, but you can’t outsource responsibility for keeping your business afloat — that’s on you, the business leader.
You can outsource your people and processes, but you can’t outsource responsibility for keeping your business afloat.
