An investigation into IBM Watson by STAT News has revealed that the artificial intelligence supercomputer has not lived up to its potential.
The piece, penned by Casey Ross and Ike Swetlitz, examined Watson for Oncology’s use, marketing, and performance in hospitals across the world, from South Korea to Slovakia to South Florida. The in-depth article included interviews with doctors that have deployed Watson at prominent healthcare institutions as well as other healthcare experts—many of whom seem far from thrilled about the results Watson has delivered so far.
The idea behind Watson for Oncology is to help physicians quickly identify key information in a patient’s medical record, surface relevant articles and explore treatment options to reduce unwanted variation of care and give time back to their patients, according to the website of Watson Health, a unit of IBM that was launched at the 2015 HIMSS conference and which employs more than 7,000 people.
But according to the STAT article, Watson for Oncology’s “treatment recommendations are not based on its own insights from these data. Instead, they are based exclusively on training by human overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.” In fact, investigators said that while IBM sales executives say that Watson for Oncology possesses the ability to identify new approaches to cancer center, in reality, “the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.”
Despite being supported with major funding and incredible brand marketing, Watson is falling short in its ability to transform cancer care, sources in the article said. According to the STAT piece, “Three years after IBM began selling Watson to recommend the best cancer treatments to doctors around the world, a STAT investigation has found that the supercomputer isn’t living up to the lofty expectations IBM created for it. It is still struggling with the basic step of learning about different forms of cancer. Only a few dozen hospitals have adopted the system, which is a long way from IBM’s goal of establishing dominance in a multibillion-dollar market. And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care.”
What’s more the authors attested, “The actual capabilities of Watson for Oncology are not well-understood by the public, and even by some of the hospitals that use it. It’s taken nearly six years of painstaking work by data engineers and doctors to train Watson in just seven types of cancer, and keep the system updated with the latest knowledge.”
Watson, the Jeopardy!-playing supercomputer, is built to mirror the human learning process through the power of cognition. Its executives, some who were interviewed for the STAT piece, attest that Watson is till on an upward trajectory and is a major part of IBM’s business. They admitted that Watson is still in its infancy but that it’s growing rapidly; by the end of this year, noted the executives, “The system will offer guidance about treatment for 12 cancers that account for 80 percent of the world’s cases.”
Watson has had both successes and failures reported in the media in recent months. A Forbes report released the day before IBM’s global CEO Ginni Rometty's keynote at HIMSS17 revealed that the partnership between IBM and MD Anderson Cancer Center, part of the University of Texas, soured to the point where the $62 million project for the cancer center to deploy Watson had been scratched.
Bizarrely, the Forbes report also noted that MD Anderson is still on the hook to pay for the whole thing and that its leader in charge of the project never got approval for the implementation from the center's IT department. Questions remain regarding MD Anderson's current relationship with IBM,
Lynda Chin, M.D., who oversaw the Watson project at MD Anderson before it fell apart, told STAT reporters that it was quite challenging to make the technology functional in healthcare. “Teaching a machine to read a record is a lot harder than anyone thought,” she told STAT, noting how her team spent countless hours trying to get the machine to deal with the idiosyncrasies of medical records.
To the contrary, however, one report out of Japan last year claimed that Watson aided Tokyo researchers in detecting a type of leukemia, helping to save a patient's life. Specifically, according to a report in New Delhi Television Limited (NDTV), the disease had gone undetected using conventional methods. The 60-year-old female patient had been mystifying medical professionals from Japan after treatment—and all previous treatment—being prescribed for the condition was proving ineffective, according to a story in the International Business Times. But in just minutes, the Watson machine concluded that the patient suffered from a rare type of leukemia after her genetic information was compared with 20 million clinical oncology studies, which had been uploaded to its system by doctors from the University of Tokyo's Institute of Medical Science.
The patient was initially diagnosed with acute myeloid leukemia, which she was unsuccessfully treated for with several different anti-cancer drugs, leaving doctors confounded, a member of the medical team said in a separate report from last week. As such, the medical team decided to use the Watson system, which, on analyzing the data available, concluded the patient suffered from another form of leukemia and recommended a different treatment which was successful.
Along with MD Anderson, IBM Watson early on developed a partnership with the renowned cancer hospital Memorial Sloan Kettering. According to IBM officials, “Memorial Sloan Kettering clinicians and analysts are partnering with IBM to train Watson Oncology to interpret cancer patients’ clinical information and identify individualized, evidence-based treatment options that leverage our specialists’ decades of experience and research.”
But according to STAT, “Several doctors said Memorial Sloan Kettering’s training injects bias into the system, because the treatment recommendations it puts into Watson don’t always comport with the practices of doctors elsewhere in the world.”
Nevertheless, Watson continues to make partnerships and relationships around the world. It just announced this week in fact that it had reached a 10-year research partnership agreement with MIT worth $240 million. The two organizations will create a joint artificial intelligence lab with the aim to “advance AI hardware, software, and algorithms related to deep learning and other areas; increase AI’s impact on industries, such as healthcare and cybersecurity; and explore the economic and ethical implications of AI on society,” according to officials.
