Identifying Hidden Influences
≈ 2 min read
There is an increasingly wide variety of AI applications out there, and the most successful ones—like Recommendation Systems, Search Engines and Ad Placement Systems—usually have at least two things in common: They benefit from a straightforward, instant feedback loop, such as 'clicked' or 'not clicked,' and operate at a high frequency, generating abundant, high-quality data to continually refine the model.
But not all real-world scenarios afford these luxuries. When you have long feedback loops (e.g., long-term medical treatments), the chance of external variables affecting the outcome arises. This makes it much harder to establish a causal relationship between features and the target variable. Additionally, in low frequency use-cases such as fraud detection, you simply have a much higher chance of overfitting on the small set of available data points, which means the model may end up learning the noise rather than generalizing the underlying pattern.
Temporal Validation of Models
To reduce the risk of project failure, many Data Scientists—including me—rely on the creation of prototypes to identify if there are such unforeseen influences or delayed feedback loops in their data. My personal recommendation is to train a baseline model two times with different validation-splits, using temporal cross-validation. With “temporal” or time-based, I simply mean that all data points you use for training occur before any data point you use for validation. You don’t want to know how often a switch from random to temporal cross-validation killed the performance of models trained by clients. For the cross-validation, you want to use a validation-data split of at least 10% and then increase it by 20-30% for the second training. Similar results on the validation data for both trainings can give you a good indication of the strength of the signal in the dataset, and help you identify potential hidden influences or data drift. If the results look very different, the solution usually lies in integrating or engineering more representative input features, rather than adopting more advanced modelling techniques.