Some AI Lessons from Watson’s Failure at MD Anderson

Last week, The Wall Street Journal(WSJ) broke a story about the failure of a $62 million implementation of IBM Watson at the University of Texas MD Anderson Cancer Center. The reasons outlined in the article don’t question the capabilities of the Watson platform but rather the inability to developer the project by a joined team of MD Anderson specialists, IBM and Pwc engineers.

According to the report, the goal of the project was to build an Oncology Expert Advisor( OEA) that will improve recommendations about cancer care and treatment options. OEA was supposed to integrate with MD Anderson electronic medical records(EMR) as well as medical literature in order to build knowledge that feeds the AI models.

The WSJ article describes that the project was initially focused on leukemia treatments but MD Anderson decided to shift the attention towards lung cancer in the middle of the project. According to an audit conducted by the University of Texas System Audit Office, MD Anderson paid about $39 million in fees to IBM and another $23 million to Pwc for implementation services. The audit declared the project “not ready” citing challenges integrating with EMR systems as well as outdated records. In an unrelated event ;) MD Anderson President Ronald Dephino resigned last week.

The objective of this post is not to criticize the failure of the Watson implementation at MD Anderson. The project was nothing short of ambitious with ambition come risks. These type of failures should be expected in a new and challenging discipline like AI on which traditional corporate IT teams lack knowledge and expertise. However, there are some interesting lessons about AI projects that can be extrapolated from the MD Anderson experience. For starters, the project highlighted how advanced AI technologies need to be accompanied by solution delivery methodologies, techniques and processes in order to ensure the delivery to real world AI solutions. In the case of MD Anderson, it seems that Watson’s technical capabilities were far ahead and disconnected from the other aspect of the solution.

Practical Lessons About AI Projects We Can Learn from the MD Anderson Experience

1 — Data Quality Matters

The MD Anderson audit repeatedly refers to outdated data as one of the main roadblocks of the project. This challenge highlights the importance of incorporating data curation and quality processes as part of AI projects. Modern data quality platforms such as Trifacta, Tamr or Paxata should be considered in these efforts.

2 — Data Integration is Key

The MD Anderson experience shows the importance of streamlining the integration of AI platforms and back office systems. Interoperability with modern ETL-ELT platforms should be considered as an important aspect of the implementation of AI solutions.

3 — Lean and Continuous Delivery

AI projects are notoriously long and complex but 4 years and $62 million before detecting a problem seems a bit overkill. Aligning the new generation of AI platforms with established lean and continuous delivery methodologies. should help in this area.

4 — Regular Model Training & Performance Monitoring

Establishing the mechanisms to regularly train and validate the performance of AI models is an essential and often ignored aspect of AI solutions. The MD Anderson project clearly shows the importance of incorporating training and monitoring tools and processes as part of the AI solution delivery lifecycle.

CEO of IntoTheBlock, Chief Scientist at Invector Labs, I write The Sequence Newsletter, Guest lecturer at Columbia University, Angel Investor, Author, Speaker.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store