The Myth of a 0% Error Rate

Business stakeholders often have unrealistic, sky-high expectations when it comes to machine learning. They hear buzzwords like "AI" and "Deep Learning" and think we're on the brink of creating infallible machines. But here's the thing: Even today’s most advanced models have limitations. So, it’s crucial to manage stakeholder expectations early on to avoid disappointment down the line.

One way to do this is by gently reminding them that we humans are anything but perfect either. In studies conducted as part of the renowned ImageNet Large Scale Visual Recognition Challenge, researchers found that humans have an average error rate of 5.1% in image classification tasks. Even trained professionals only managed to reduce this to 1.7% [1]. But what if people would really give their absolute best? Maybe in a setting where actual lives depend on the accuracy of their work? During the Apollo program, they’ve found that 10.3% of components failed during vibration and thermal acceptance tests due to errors in workmanship [2]. This hopefully gives some context and helps with managing expectations. A model that gets to error rates below 5% on the test data can usually be considered on par with human performance.

Diminishing Returns of Increasing Model Complexity

True, we can increase performance and reduce error rates in our machine learning applications by creating more intricate models capable of learning more finely detailed generalizations. But this also results in much greater model complexity, which often comes with important trade-offs:

Cost: More complex models typically mean higher computational and operating costs.

Time: The size and intricacy of a model are directly proportional to the complexity of its deployment setup. This can drastically increase both time-to-market and operational efforts!

Explainability: Complex models, especially deep learning architectures, often lack interpretability. This can pose unique challenges in certain operational environments.

Revisiting Confidence Thresholds

So, are there ways to improve error rates without making the model more complex? Absolutely. My own favorite approach is to use Random Forest classifiers which are kind of like the Swiss Army knife of machine learning—versatile, reliable, fast, quite small, and relatively easy to deploy. They provide you with a probability value which can help filter low confidence predictions and hence reduce error rates.

For example, in the histogram below, I’ve plotted a model’s confidence levels for both correct and incorrect predictions using a movie sentiment dataset. The cumulative lines reveal that a majority of incorrect predictions fall within the low-confidence zone. By then setting a probability threshold at 0.58 (indicated by the red line), we can eliminate 65% of incorrect predictions at the cost of losing only 23% of correct ones. Although this strategy reduces the automation rate from 100% to 70%, we’ve managed to lower the total error rate from 17% to 9%, a significant drop.

Many models, including most deep learning architectures, offer confidence scores for their predictions, making this approach widely applicable.

As always, it's also worth revisiting fundamentals like gathering more training data and relevant input features which can provide the model with a richer information set, or 'signal', to learn from. Combining this with data preprocessing and feature engineering to incorporate domain knowledge can additionally strengthen the input signal, often drastically improving performance and model interpretability.

TL;DR

As attractive as the pursuit of a 0% error rate may sound, it’s simply not realistic given the current state of technology (if it will ever be). For that reason, it’s imperative to help our stakeholders form realistic expectations, clearly communicating the trade-offs such requirements have on both time-to-market and operational cost and complexity. This doesn’t mean that we shouldn’t look for ways to lower error rates, but I’ve found that, in most cases, simplicity trumps complexity, especially when we are talking about models used in productive environments.