to us asking for a model.
We built a proof of concept. Got the green light. Delivered the model.
Weeks of work…all to hear nothing.
It’s a tale as old as time, and one that plagues data professionals everywhere, from analysts to ML engineers.
So, what happened?
Your Model is a Mystery
Our profession is one rooted in modern computer science and technological advancements. Many of the most powerful solutions at our fingertips are ones that would have been too computationally expensive decades ago. With the reliance on the newest, most capable technical breakthroughs, comes skepticism.
In data science, we have the ability to create incredibly complex models. My team alone has hundreds of standard features in our feature library that we provide to each new model build. We tune dozens of hyperparameters and use powerful algorithms that iterate over hundreds of runs to maximize predictive performance. This process can create models with incredible accuracy, but it comes at a cost: explainability.
There is a fine line between a strong model, and a black box that can’t even be explained by the ones who built it.
The explainability-accuracy tradeoff is a big factor in my industry, healthcare, in particular. Customers and stakeholders are often doctors and clinicians. These doctors are used to making clincial judgements using their years of expertise and in-depth knowledge of medicine. While a predictive model may be good at predicting a given outcome, if it cannot be explained well, clinicians will question its trustworthiness. If doctors have to choose between a trusted and proven clinical process, or a black box model with cryptic features and inexplainable algorithms, they will likely choose the clinical process every time.
So, what can you do to avoid this? I find the most success by providing customers with an easily digestible model brief. This is a set of slides that walks the customers through the model. It starts by defining the population of interest, the target, the features, and then ends with proof of concept performance and validation. Along the way, I am sure to define metrics in terms of the business question, putting myself in the customer’s shoes. I avoid pure stat-talk and keep definitions grounded in the customer’s goals. If the model is complex, I stick to high-level explanations of the algorithm and be sure to communicate why I chose such an extensive feature set (or such a simple one). Developing a comprehensive model brief is a crucial step in pulling back the curtain and allowing the customers to understand the model using terms they are familiar with.
Your Solution Took Too Long
Building working models takes time. From the back and forth correspondence with customers, to unexpected twists you didn’t see coming, designing an effective, useful model is not a quick task. And then there’s deployment. That’s a whole process in and of itself.
What doesn’t wait patiently is the real world. Customers are living their day-to-day with the tools they already have at their disposal. The tools that existed before they came to you for help. If the model build takes too long, they could abandon the idea altogether, or find creative solutions that don’t involve predictive models.
We see this all the time in healthcare. Stakeholders will request a model. After a few roadblocks (stalled communication from requestors, data access issues, deployment bugs, etc.), weeks of development stretches into months. Finally, you are ready to present findings after everything is validated and working as expected. You attempt to set the meeting and get heartbreak: “We no longer need the model, we figured it out ourselves.” The hosptial setting is a fast-paced environment. Staff doesn’t have time to sit around waiting for months on end. They can and will come up with creative solutions to improve care for their patients, even if that means sacrificing the use of a shiny predictive model.
There is a saying I live by at work: “Don’t let the perfect get in the way of the good”. Build fast. Ideate, refine, review…but always be moving forward. Perfection can prevent you from providing valuable insights. The world moves quickly, and if you get stuck in the build phase for too long, the world will move on without you. So, push that v1. If you discover a better way of doing things later on, it can be first on your list of improvements for v2. Some solution is almost always better than no solution.
If things are moving slower than planned, then you need to communicate with customers early and often. Keep them posted on your progress, and provide them with a sneak peak to keep them engaged and excited for the final product. Bide your time while you grind to get v1 working and into their hands.
Your Model Isn’t Easy to Consume
Building a good predictive model is only half the battle. In most industries, the stakeholders are busy. In heathcare, the doctors and nurses are absolutely swamped caring for patients. If the data science team comes to an on-floor care team to pitch their newest, most accurate model, but accessing the predictions adds complexity to their workflow and slows them down in the process, the model will never be used. The same can be seen in most industries. Stakeholders want solutions that can increase efficiency, performance, and productivity, not ones that only add complexity to their already busy days.
If predictions introduce friction, you are forging a path towards abandonment, not adoption.
Serving predictions that are easy to consume can be one of the biggest challenges for data scientists. We may be skilled at creating precise and accurate models, but integrating the model into customers’ daily lives comes less naturally. This part is less about numbers, probabilities, and statistical acumen, and more about operations, business knowledge, and familiarity with the day-to-day processes of the requestors.
In the hospital setting, this looks like integration into Epic, the electronic health record software used system-wide. Instead of requiring busy clinicians to log into a separate system to see predictions, they can access them right there, in the patients’ charts, alongside their other clinical tools and patient data. In other industries, the same idea applies. Don’t disupt the current process. Fit into it.
Wrapping Up
One of the biggest disappointments a data scientist can face throughout their career is their hard work going unused. It happens more than one would like to think, and it’s easy to blame the customer. After all, it’s easier on the ego.
In reality, there may be some crucial elements the data scientist neglected somewhere along the line of development. Being aware of the common pitfalls can help data scientists get their models across the finish line. The real finish line: adoption.