Disaster Recovery and Business Continuity Strategies Evolve Forward

When it comes to the key, interrelated subjects of disaster recovery and business continuity, there’s no learning like learning in the moment. That reality was emphasized during a cybersecurity-focused panel held on February 2, the second day of the Health IT Summit in San Diego, sponsored by Healthcare Informatics, when Sri Bharadwaj, director, information services, and CISO, at UC Irvine Health (Irvine, Calif.), led a panel entitled “Ransomware Risks: What We Learned From NotPETYA and WannaCry.”

As has been widely noted, the May 2017 cybersecurity attack dubbed “WannaCry” grabbed storylines internationally and across the healthcare landscape as tens of thousands of hospitals, organizations, and agencies across 153 countries had their data held hostage, while the June PETYA/NotPETYA attack unleashed further damage worldwide.

“I was actually at a Healthcare Informatics conference” when the global WannaCry attack hit last May, Bharadwaj noted as he opened the discussion on Feb. 2, and referred to the Health IT Summit in Chicago held in May 2017. “I was speaking on a panel that morning, in Chicago, and this thing hit us. I got a frantic call, and I was on the phone call. For the first ten minutes, I said, OK, I’ll try to figure that out. That became six hours. I almost missed my flight home that day. It was one call after the other, providing updates, communication, etc. But we did not shut down the Internet, our Outlook, or any feedback back to the end users. We got the most hit from our medical devices. It was fairly easy to patch stuff and get stuff done, but we realized that our realm of exposure encompassed all sorts of things—who the heck knew that the parking system was running on a Windows 98G? Who knew that the cafeteria system was running an old version of Windows so old that we had to figure out what it was?”

“The key questions,” Banash said on that panel in response to Bharadwaj’s opening statement, “are, are you managing your risk? Do you understand your attack surfaces? What vectors are you vulnerable to? When this started out, no one knew what was going on; it was crazy. If you had one of those maps in your security center, it was all lit up, and it looked like ‘War Games.’ Initially, we thought it was via email, and we were chasing emails, but when we found out it was SMB [server message block] vulnerability, we were able to chase that down. We were hit, but there was no successful attack on us. But understanding what was in your environment—it never became more important than on that day. And those MRI machines running on Windows XP—those machines are million-dollar pieces of equipment; it’s hard to justify new purchases to the board. I would say we were lucky; I’d like to say we manage things well, but we did get lucky.”

Asked about connections with law enforcement, Christian Abou Jaoude, director of enterprise architecture at Scripps Health, San Diego, said on the Feb. 2 panel, “We do have a direct contact with law enforcement; we also have a protocol that we follow that’s been well-established. We followed those procedures, but the same thing happened to us: there wasn’t much information available during the first couple of days” following the WannaCry attack. “So I went out and read as much as I could about it, read articles to see whether there was something different about this. So we enacted that process, sent out notifications, and then a few days later, everyone learned what had happened.”

“I think we got lucky,” said Chris Convey, vice president, IT risk management and CISO, Sharp Healthcare, also on that Feb. 2 panel, “because this started in other parts of the world. Here in the U.S., we got lucky. I was at Millennium Healthcare then. SMB was blocked, that was the first thing. And then, how are our backups protected? And then patching. And it turns out, the basic security hygiene was needed. Look at what happened at NHS (National Health Service). And to be honest, we hadn’t patched as well as we could have. It’s hard to do, especially in the healthcare space, because you’ve got to test, and you don’t want to bring down patient care.”

Looking at the Bigger Picture

The kinds of experiences that the members of that Feb. 2 cybersecurity panel cited are exactly the kinds of issues that industry experts say need to be carefully worked out and strategized in advance. And some of what’s involved really is fundamentals, says Shefali Mookencherry, a principal advisor in the Naperville, Ill.-based Impact Advisors consulting firm. “First of all,” Mookencherry says, “You have to plan. Most folks will have [a broad disaster recovery strategy] in their heads. But if it’s not written down, you may forget. “You need to look at all of your business processes and put together an actual disaster recovery team; and that requires an interdepartmental, indeed, enterprise-wide, effort. And it’s related to project management. You have to look at the data, the documentation of a plan, the processes, you’re going to look at, and of course, policies and procedures. Enforcing and communicating those to everyone involved is key,” she emphasizes.

In other words, prioritization. “You need to classify people, data and technology, in terms of what’s essential, what’s important, etc.,” Mookencherry says. “And if you have a plan, who’s going to execute it? What’s your first line of communication? What would an employee say if a disaster had occurred? How would the plan be carried out?”

In fact, Mookencherry says, “Sometimes, organizations get so concerned with, I’ve got to protect my storage—storage, storage, the back-end infrastructure. But what about the front-end infrastructure? And in most hospitals, what happens if a system goes down? Most go back to paper. Disaster recovery is not sold well to the board in most organizations. If it’s a major hospital or health system, it kind of resides inside a healthcare IT silo. But what DR really means needs to be explained, including to the board. Brand reputation loss, lawsuits, and loss of data, are all important. Selling a DR plan to a board of directors is probably one of the harder things to do. It costs money and is difficult to do.”

That point is one that many CISOs are quite aware of, among them, Jason Johnson, the information security officer at Marin General Hospital in Greenbrae, Calif. “On the business continuity piece, one thing I’m working on now is to get senior leadership to understand that it’s not just an IT problem, and we need a business plan,” Johnson said during the Feb. 2 San Diego HIT Summit panel. “If I can only bring up 20 of our systems, [the members of the C-suite will] need to tell me which 20 systems to bring up. And one of our affiliate partners got hit with ransomware last summer; and they manage our billings for our clinics, etc. And their backups and everything got encrypted. So it’s 11:30 at night, and my CIO calls me and says, 'This company just got hit by ransomware, you’ve got to get back to the hospital right now.' The security was actually the easiest thing to do. But then it took three weeks of 12-hour days to rebuild Allscripts from the ground up. And between that and the WannaCry, it’s helping our organization to understand that this is not an IT problem.”

What’s more, says John Robinson, a senior advisor with Impact Advisors and a colleague of Mookencherry’s, says a fair amount of human, technological, and financial resources will need to be applied in advance to foundational activities, in order even to prepare a meaningful disaster recovery program. “This is where people fall down—it’s really understanding what’s in that data center,” Robinson says. “My bet is, you come to any hospital and say, 'Show me a list of the applications you run in your data center; they would actually struggle.' They do not have the foundational components of having an application catalogue, or a configuration management database, that says who does what, when, and what they’re allowed to do. Until you do that, all these fancy security technologies are going to be difficult to implement, and you’ll spend a lot of money delivering a security solution, because you don’t really have a full picture of your environment, so you don’t really know when you’re done.”

So how do healthcare IT leaders engage the C-suite and the board in their organization in meaningful dialogue about this? Impact Advisors’ Mookencherry reports that “I had a client, they were looking at disaster recovery from a technical standpoint, and in terms of solutions. And I said, you need a short-term solution and a long-term solution. And I had to put together a presentation for the board. And it came down to, do you want to safeguard revenue, and your brand? Really, where I went with it was, you have too many federal regulations now that are demanding that security protections be in place, and part of that is disaster recovery.”

Indeed, Mookencherry emphasizes that new sanctions could hit hospitals unprepared for the compromising of protected health information (PHI) in the future—particularly any U.S. hospital that treats patients who are citizens or legal residents of any of the 28 European Union nations. “HIPAA [the federal Health Insurance Portability and Accountability Act of 1996] is the biggest one, but with the advent of GDPR—the General Data Protection Regulation approved by the European Parliament, Council of the European Union, and European Commission and voted into law by the European Union in April 2016, and effective in May 2018—GDPR has higher fines than HIPAA,” she notes. In other words, the stakes are about to rise even higher when it comes to potentially adverse outcomes emerging out of any disasters that might take out hospital and health system information systems.

Will the European Union actually sanction and fine U.S. patient care organizations for violations of the GDPR regulation? “Yes, there’s the privacy shield, monitored by the U.S. Department of Commerce,” Mookencherry warns. “Any organization that does business with EU citizens or even residents, any data, including demographic, is protected under this regulation. And the UK Information Commissioner’s Office actually works with the Commerce Department here”—meaning that U.S.-based hospitals—a larger number of which treat European Union citizens and residents than one might think—face new perils in this area.

CISOs Lay the Foundation for Practical Disaster Recovery

CISOs and other healthcare IT security leaders at U.S. patient care organizations are laying the foundations now for successful disaster recovery. “We’ve hired a third party to work for us,” reports Thien Lam, vice president and CISO at the 14-hospital BayCare Health System, based in Clearwater, Fla. “They’re 1,000 miles away. We have a data center locally and a data center in the Northeast. And we have a disaster recovery plan, so that if we’re ever in a disaster situation, we can switch to our second system 1,000 miles away. We do two tests a year,” Liem notes. “We simulate the primary data center being down, and we switch to the second data center, and the team makes sure we have connectivity and flow, and the business signs off on the DR test. And every time you do a DR exercise, you identify an issue or issues, and we correct those.”

The fundamental set of challenges, says Fernando Blanco, CISO at the 60-plus-hospital CHRISTUS Health, which is based in Irving, Texas, and operates across the United States, as well as in Mexico, Colombia, and Chile, is “the integration between applications. Often,” Blanco says, “we don’t pay attention to interfaces and other interconnectivity with applications. So we can restore our EHR (electronic health record) in no time—we have our Meditech up and running—but it’s all the satellite systems involved. And that involves a step before the disaster recovery plan, which is the business continuity.”

Meanwhile, Blanco says, “There are three core elements to business continuity—disaster recovery, business continuity, and crisis management. The disaster recovery plan is the most tactical: do we have the backups? Do we have a place where we can restore? Basic things. Do we have an arrangement with the vendor to get the equipment we need? That’s the purely tactical.”

In the end, Blanco says, “The more strategic considerations are the business continuity concerns. If I lose my EHR, I cannot operate more than two hours; so that determines the security and other mechanisms we need to put in place. That’s something the business has to do: they have to decide which are the key applications that need to be restored, and how long they can go without those applications. Let’s say you’re the business and you say, ‘I need all my applications back up and online within one hour’; well, you can’t do that, the cost would be outrageous.” So ultimately, as he points out, “Everything can be done, but there’s a cost to everything.”

For healthcare IT security leaders, who are facing escalating threat vectors across all operations, there’s never been a better—or more urgent—time to put solid disaster recovery and business continuity plans into place. And that urgency is only expected to ramp up going forward.