Machine learning (ML) and artificial intelligence (AI) are consistently touted as delivering high value returns, offering businesses everything from improved customer experiences to boosts in revenue on orders of magnitude. Respected media outlets like Forbes hypothesize an outsized impact on business, while some of the wealthiest companies in the world (Google) seem to be capturing a significant upside from ML. It’s little wonder that so many businesses are attempting to implement ML models. It’s not hard to find useful models that can make sense of the large amounts of data businesses manage. However, quite often, to implement a useful solution to a range of business challenges, one important aspect of ML model implementations is frequently overlooked. Every model, no matter how simple or sophisticated, is a tool. Just like any other tool, ML models need to be routinely maintained, monitored and refined to keep delivering the value that your business needs. Is your business successfully managing its ML model, or neglecting execution and monitoring?
What could possibly go wrong?
There are some critical areas that your team needs to take into consideration to keep your ML model humming along like a well-oiled machine. Perhaps the most important aspect of any successful ML implementation is the data. Unfortunately, it’s easy to overlook the fact that many models are developed using limited, cleaned, static datasets for their proof of concept. Since the example model seems to work well on the desktop, it’s assumed that it will work reasonably well after company wide implementation.
The problem is that your company’s data is not a static, beautiful aggregation of points just eagerly awaiting assembly. Data constantly changes. Certain types of data change rapidly (in milliseconds, like stock information) and other types only update periodically (a customer’s address change or a customer’s credit score). Multiple data streams with very different types of data can pose significant challenges to a model. Model monitoring will quickly identify inconsistencies in the production model data streams.
In addition, the data may not be completely accurate. Human error and different ways to input the same idea (such as using a “ft.”, “feet”, or single quote for the word feet) frequently contribute to challenges. As noted by Thomas Redman in the Harvard Business Review, problems with data do not just affect the output of your ML model. They also impact other parts of the business that rely on that model’s output, resulting in a cascade of erroneous conclusions from the imperfect data.
Cancer in the system
Even the brightest, most tech-savvy businesses have suffered from erroneous or unreadable data. An example comes from IBM’s Watson, its signature artificial intelligence system that utilizes natural language processing to help manage input. Watson for Oncology was an ambitious drive to address one of mankind’s most challenging health issues; it was going to change the face of how cancer patients were treated. The system was adopted in dozens of hospitals globally, and hopes were very high. Then MD Anderson Cancer Center, part of the University of Texas, decided to part ways with the tool. The center had invested three years and more than $60 million in the tool, but it simply couldn’t meet all of the hospital’s needs. Medical records contain handwritten notes, special codes and a range of abbreviations and shorthand. Watson could not successfully integrate this information into a useful outcome. Granted, not all ML solutions are this complicated; the point being if the model cannot access the data as it was developed with, the model will yield unreliable results. Consistent model monitoring will immediately highlight the need to reconcile the data from production vs development.
Keeping an eye on things
ML models require continuous monitoring and maintenance to ensure that they remain relevant and are actually delivering the value that is expected. Even the best models need to be evaluated for their stability, validity, execution and data capture. What are the recent model inputs like? Have they changed? Is the model still performing as expected? Is it consistently delivering the results that your business needs to achieve its goals? Monitoring involves answering these types of questions and marrying them with the business’ current goals and objectives. The insights garnered from data exhaust (such as timestamps, engineered data and transformed data) are useful in this evaluation.
A real-life example: a recent client engagement discovered that a client’s model developed in house was put into production by their business process implementation team and subsequently NEVER executed. Accountability for execution was not assigned, and as a result, the monthly batch process was not monitored. Meanwhile, the business strategy continued and consumed the stale model scores. Active model governance monitoring is a necessary part of any organization relying on ML to deliver solutions.
Model relevancy
Monitoring naturally leads to an important question: Is the model still relevant? The model may be generating output, but this output may not have resulted in the gains you were expecting. Relevancy is related to underlying goals and assumptions. The original model may not have actually addressed the business problem in the first place, but produced output that may have seemed close to desired outcomes. Facebook’s chatbots, Bob and Alice, were originally designed to navigate a dialogue and help improve chatbots’ conversations with real people. However, the chatbot project was shut down once it was discovered that the bots were talking to each other – in their own language.
Evaluating the model can result in other critical insights. For example, regular evaluations can uncover fraudulent or misguided behaviors that are attempting to successfully game the system. This insight is especially critical for businesses that manage personal and financial data, such as insurance companies and banks. It’s also important to other types of businesses. For example, recent history is littered with instances of attempting to game Google’s algorithms. BMW attempted cloaking (which unfairly boosts a site’s search engine rank), Overstock abused link-backs and WordPress misused doorway pages. Actively monitoring the strategy results would have revealed anomalous behavior — it’s not just the ML model, it’s the strategy wrapped around the model too.
Developing and maintaining a machine learning model – for good
Many companies find that machine learning projects are tedious. It is true that the scope of successful ML model implementations goes far beyond merely creating a model. However, by regularly maintaining and evaluating your model, it can amplify your company’s success and truly set it apart in a marketplace that is becoming more crowded by the day. Investing in an ML implementation does require resources and the right knowledge workers, but its ROI goes far beyond these investments.
Whether you have just recently implemented an ML solution, have been using one for years or are only considering one, you can benefit from the experience and knowledge of veteran technology consultants. At DecisivEdge, a global leader in business consulting and technology services with a seasoned data science practice, we focus on delivering the results you need to attain your business goals, grow your company, and allow you to innovate. Connect with me to start a conversation and learn how your ML implementation can drive your strategic vision.