Five caveats for choosing predictive analytics

Oct. 23, 2017 / By Kristine Daynes

Years ago, I briefly thought my calling in life might be as a bookie in Las Vegas. I was in business school and had developed a statistical model that could predict who would win best actor and best picture at the Academy Awards. It had over 95 percent accuracy. I was quite proud of the model, until a classmate noted that the prediction was based on who won the Golden Globe and People’s Choice awards, which are announced weeks before the Oscars. He said a really great predictive model would choose winners before any awards, nominations or critics’ reviews were released. Even better would be a model that could predict a good movie while it was still in production. Such a model would have many more factors and interdependencies than I had patience to consider. I gave up on a career in the gaming industry.

My classmate pointed out something that is true for the healthcare industry today. It is easier to predict what happens next when you already know a pattern of behavior. It is harder to predict an outcome before there is a sign that something might be happening. Or when you have incomplete information. Or when the factors contributing to an outcome are nuanced and interrelated.

Fortunately, not everyone gives up on complex statistical models as quickly as I do. Within health care, predictive models abound. There are algorithms to detect avoidable readmissions, prevent sepsis, decrease length of stay and anticipate heart failure. In fact, there are over 24,000 medical calculators, decision support tools, risk scores and other predictive models available for download or subscription.

And there are a lot of promising technologies to support and enhance predictive models—natural language processing, unstructured data mining, machine learning and artificial intelligence. As new and unproven options emerge, health leaders have to be thoughtful about how to support an organization’s decision-making.

Here are five things to keep in mind about predictive models. (They come from colleagues and opinion leaders who have more tenacity with predictive models than I do. Their experience carries some weight.)

1. Predictive models are meant to be a supplement to human expertise. They augment human judgment, not replace it. Engage the consumers of your data to find out what information would make them more effective and confident in managing health. Consider how automation or prediction could simplify their work, for example, by prioritizing worklists or surfacing the root-cause factors of adverse events.
2. Choose models developed by teams with deep understanding of medical practice and the business of healthcare—domain knowledge. They are more likely to surface assumptions in the data and interpret the results correctly within context. Likewise, make sure the way the tool risk adjusts, normalizes and benchmarks data is clinically relevant, not only statistically sound. People are more likely to use information when it is consistent with their own experience and can be translated into action.
3. Look for algorithms that have been developed using large, clean data sets. Healthcare data sets are many times smaller than in other business applications. You want to be confident that the model is not skewed by missing information or outliers.
4. Choose tools that are appropriate for the intended use. Some algorithms are designed for research or disease-specific clinical settings, but cannot be scaled to other care settings or populations. Match your algorithm to the types of data available, care delivered and patients treated.
5. Assess your information infrastructure to make sure data can be easily extracted. Strategically pursue data use agreements to get permission for additional data. Make sure all information is accessible only to be used for the right purpose.

For more information about predictive algorithms, you might also be interest in this educational content:

Podcast: How data analytics is changing value-based care

Webinar: Predictions about predictive analytics: From telescope to microscope

Video Playlist: Better care management with predictive analytics

Blog: Deep learning and clinical natural language processing (NLP)

Blog: Overcoming skepticism about value-based care

Kristine Daynes is senior marketing manager for payer and regulatory markets at 3M Health Information Systems.

Making use of unseen data. What insights can you uncover?