Why Machine Learning Projects Fail

Start typing ‘artificial intelligence will change’ into a search engine and you will see suggested sentence endings like ‘the world’, ‘everything in your lifetime’ and ‘the face of business in the next decade.’ Search a little further and it will become clear that AI and machine learning projects are not only driving advancements, but are integral to their success. According to research from Accenture, 85% of executives in capital-intensive industries say they won’t achieve their growth objectives unless they scale AI.

At the same time, research from MIT Sloan suggests that the gap between organizations successfully gaining value from data science and those struggling to do so is widening. As we know, data science and machine learning are the engine behind AI applications, as it is through processing data that AI learns how to interpret our world and respond as we want it to. If AI is to make a real impact on companies and their customers, companies need a new approach to machine learning. As the MIT Technology Review concludes: ‘the way we train AI is fundamentally flawed.’

Many articles in publications like Towards Data Science and Open Data Science seek to pick apart exactly why machine learning projects fail with a fine tooth comb and a helping of technical jargon. These articles are great if you’re a data scientist, but not so helpful if you’re a company trying to work out why the conversational assistant or personalization campaign you spent thousands investing in never took off.

The reality is that your machine learning project most likely did not fail because you messed up your approach to data versioning or model deployment. Most machine learning projects fail simply because companies did not have the right resources, expertise or strategy from the start. McKinsey’s 2021 State of AI Report corroborated this, reporting that companies that see the biggest bottom-line impact from AI adoption follow both core and AI best practices and spend on AI more efficiently and effectively than their peers.

Five common AI mistakes businesses make

Through our work on ML projects for some of the world’s largest companies, Applause has identified a pattern of common mistakes that reduce efficiency, drive up cost, push back timelines — and ultimately are the reasons why machine learning projects fail.

Common mistake 1: Misjudging the resources needed to train ML algorithms

The first reason why machine learning projects fail is that companies are unprepared and ill-equipped to see them through. According to Dimensional Research, 8 out of 10 companies find machine learning projects more difficult than expected because they underestimate the work that goes into training models properly. This is why so few data science projects make it to production; without a clear understanding of the resources and expertise needed, companies end up either coming up against insurmountable obstacles or burning through their budget due to inefficiencies. One thing they misjudge the most is the effort required to obtain the right training data — which brings us to common mistake number two.

Common mistake 2: Relying on data brokers to supply one-size-fits-all training data

Companies do not struggle to obtain training data. After all, there are numerous data vendors that sell training data artifacts in huge volumes for low prices. The reason why machine learning projects fail is that companies struggle to obtain high-quality training data.

In purchasing one-size-fits-all data from vendors, companies do not obtain data specific enough for the needs of their machine learning project. To understand why, consider the example of an online exercise class provider building a digital personal trainer (PT). In order for the PT to be able to recognize poor form and recommend improvements, it needs to be trained with data that goes beyond images of individuals in different exercise positions. It also needs to know how to recognize individuals in different levels of exhaustion and perspiration, wearing different clothing and with different levels of fitness and expertise.

There are many other problems with pre-packaged training datasets, among them:

• There is no guarantee that the data represents the balance of ages, genders, races, accents, etc. needed to reduce bias

• The data has either not been annotated at all or not annotated in a way that makes sense for the algorithm

• The data has not been vetted for compliance to data standards required by global AI regulations like the draft European Artificial Intelligence Act (EU AIA)

• Companies cannot be sure that the correct data privacy and security measures have been observed, nor receive guidance on how to protect the data’s integrity moving forward

To execute truly successful machine learning projects, companies should think of training data as something they need to curate, rather than source.

Common mistake 3: Underestimating the extent to which AI development requires constant iteration

Buying data from vendors not only has ramifications for training data quality, but also renders the rest of the AI training process infinitely more difficult.

Training ML algorithms is not a one-and-done process. Once the training is underway, developers need to continually request changes to the data being collected as the needs of the data model become clearer. This is because training an AI algorithm is like trying to grocery shop and cook at the same time: you may think you have all the ingredients you need, but once you start cooking, you realize that you forgot an ingredient, one needs swapping out or the balance of ingredients isn’t right — and you need to keep amending your recipe accordingly.

In machine learning, it is difficult to know exactly which data you need until you initiate the algorithm training process. You may realize that the training set isn’t big enough or there was an issue with the way the data was collected. Many data brokers have stringent amendment policies — or offer no ability to amend orders at all — leaving AI developers with data they can’t use and no choice but to purchase another training set that meets their new requirements. This is a common bottleneck for many companies that drives up prices, pushes back timelines and reduces efficiency. Ultimately, it’s the main reason why machine learning projects fail.

Common mistake 4: Not integrating QA testing

Companies across all industries often fail to integrate QA testing at all stages of the product development process. It is falsely considered an add-on, as a formality to double-check that a product works correctly, as opposed to a tool that can be used to optimize the product in an iterative fashion.

One reason why machine learning projects fail is that this attitude towards QA testing is untenable given the realities of AI development. Unlike in traditional software development, you can’t sort out bugs with a simple software update; rather, errors discovered at the QA testing stage can only be fixed by re-doing the entire process. If your AI is not working as intended, it’s most likely because there was a problem with the training data, or the training data skewed the model in the wrong direction. Either way, this means going back to stage one and curating new training data artifacts.

Companies that don’t integrate outcome validation at all stages of the AI development process make more work for themselves. Rather than training the algorithm with one ginormous dataset and then testing the AI, companies need to train and test more iteratively. Taking an agile, ‘baked-in’ approach to testing will help drive down unnecessary spending, speed up timelines and allow for a more efficient allocation of resources.

Common mistake 5: Failing to schedule frequent reviews

The last reason why machine learning projects fail is that companies celebrate their success too early.

AI projects are never really finished. Even if an AI experience entirely meets accuracy and performance expectations, it still only has been trained on data that reflects society as it stands today. The algorithm has learned to make decisions based on opinions, dialogues and images that are already changing. Think about natural language processing (NLP) applications: these only know how to communicate because they were once trained on real conversations with people. Given that around 5,400 new words are created each year in the English language alone, NLP applications will wane in accuracy very quickly.

If AI experiences are to continue being useful to customers, they need to be re-trained on a rolling basis as social attitudes, developments in technology and terminologies change.

How to ensure successful machine learning projects

What companies need is a program approach to developing AI. Rather than thinking of each individual stage of the process as distinct projects, companies should consider uniting them as part of a holistic program. Developing AI is an iterative, agile process in which teams need to work in tandem, not silos, all governed by a program leader with responsibility for program success.

To learn more about how your company can implement a program approach to build AI experiences that are truly useful to your customers, download our whitepaper: Building a Global ML/AI Data Collection & Quality Program.