NYSE Outage - Lessons Learned
By Michael Thompson on Jul 9, 2015

NYSE Outage - Lessons Learned

CNBC Article

July 8th was not a great day for United Airlines, Wall Street Journal, or the New York Stock Exchange. Although the root cause of the NYSE outage has not been identified, NYSE is stating that it was not an outside threat, but an internal configuration error stemming from a software upgrade the night before. While it is comforting to know that the outage was not caused by malware or an outside attack, it does highlight the fact that a majority of “disasters” that impact a business are still self-inflicted.

So, what are the effects of this outage? We may never know the full extent, but here is what we do know:

1. NYSE Competitors had a huge day. NYSE immediately lost trades to the competing exchanges. In fact, even when the NYSE was back and operational, the traffic was still low until brokers were confident in the system being operational and stable.

2. Integrity of the NYSE network and confidence in the NYSE systems is now in question. While the outage does not have an easily quantifiable impact on the financial markets confidence in the NYSE, it does beg the question…how much time, effort, and resources at NYSE need to be spent the gain that confidence back?

3. Loss of productivity. Tens of thousands of trades that were pending from the morning session had to be cancelled. Although those orders were filled through other exchanges, the amount of “rework” required and the lost productivity from the outage caused staggering down time and redundant work process for the markets.

4. Regulatory is now involved. As with a highly regulated industry, a situation like this now requires regulatory agencies to get involved. There will be an extensive and costly audit that will need to be conducted and the findings, may or may not lead to additional business requirements, new processes and policies for upgrades, and possibly fines for the outage.

Many of our customers that we work with on a day to day basis may not global brands like NYSE and don’t have the immediate impact on the world financial markets as they do, but all of our customers need to understand the real life impacts that happen when a network is down.

Many of the clients we work with daily are not global brands like the NYSE. They don’t have immediate impact on the world financial markets, but all of our clients need to understand the real life impacts that happen when a network is down.

NYSE was not hit by a fire, flood, or malware—the disaster in this case was caused by software and “configuration issues”. All organizations and businesses that we touch upgrade hardware, change software, move to different applications; this is a normal business operation. Yet, this normal business operation cost the NYSE close to 4 hours of downtime and that impact reaches much farther than what can gleaned from media outlets today. So, although our clients may not all be NYSE, these takeaways are something that all businesses should be taking a hard look at and asking themselves the tough question. "How are we protecting ourselves today and what is our ability to recover from a disaster, man-made or otherwise?”