“I model,” he tells people. This is inevitably followed by a short pause. “And the looks I get,” he admits, “are priceless”.
Merijn van Koeverden is a data modeler. And his prediction model factory for ANWB, the Royal Dutch Touring Club, is impressive. It comprises over one hundred models for personalized marketing. Out on the catwalk, it turns heads.
ANWB is the largest association in the Netherlands, with a unique portfolio mix that aims to contribute to a more sustainable society. They don’t just provide travel advice to their five million members, but market over 250 products and services, from road assistance services, insurance products, packaged vacation deals, to travel-related products that can be purchased online and in physical stores.
The company uses personalized marketing to ensure that each of the five million ANWB members see only those products or services that are most relevant to them. And to determine which offerings are the most relevant, they use 100+ prediction models, which are scored, evaluated, and run on a daily basis by just two full-time employees. To put this into perspective, popular urban legend says that one full-time employee will be able to maintain a maximum of just four models in production.
In this interview with Merijn, we asked him to explain how they have successfully automated and scaled the modeling process.
How were you marketing your products and services to members before the prediction models and why did you decide to change it?
Before we started building our prediction model factory, target groups for most marketing campaigns were built based on a mix of ad hoc models and the knowledge of product marketeers or product experts. Basically this resulted in target groups like ‘members who bought the product before’ or ‘members who already own or experienced a specific product/service at our organization’. Without a model available, usually just one personal characteristic would determine whether a member would receive offerings from a marketing campaign or not. In terms of working in a data driven manner, the best we could do was evaluate the customer characteristics of the members who converted, and change target groups for the next campaign based on our learnings.
From the moment e-mail and online channels became more and more important, the demand for more targeted marketing campaigns grew within our organization. We realized that our way of working was no longer feasible. With multiple overlapping target groups between campaigns, we had to find a more efficient way to define them. We also wanted to keep the experience for our members as relevant as possible by making sure we always show the right product at the right moment to the right member.
Why did you decide to build prediction models?
First of all, we wanted to use prediction models to increase the efficiency of the task of creating our target groups. With so many different offerings and members within one organization, the time employees need to define a target group had to become manageable. Our wish was to make this process not just manageable, but also scalable, standardized and fast.
Second, in order to show the right offering at the right moment to the right member, we wanted our prediction models to tell us how likely a member would purchase a product or service anew each day and for every product. Prediction models give us the scalability to move from relying on only a few customer characteristics to computing our own multiple variables (we now have over 300).
Why did you decide to use KNIME to build your model factory?
Since we have so many products, but only two full-timers, it’s not possible to build a custom-made prediction model for every product. So we needed to find a tool that would enable us to streamline the model building process.
KNIME, as a low-code, no-code tool has a user-friendly interface which means we can build models quickly. KNIME Analytics Platform offers multiple components for machine learning and data mining which we use to build the model in a modular fashion, dragging and dropping nodes and components into a workflow. It’s fast, convenient, and a very efficient user of our resources. The best thing: You don’t even need all the skills or knowledge of a programmer to build a prediction model in KNIME.
Tell us more about the model building process.
First we define our target i.e., the kind of behavior we want to predict, and then we replicate this with KNIME. Together with our partner Personalyse, we designed our own blueprint in KNIME that enables us to build a solid classification model in half a day.
The template includes all the sampling, model training, testing, and validation steps. Even deployment is handled automatically. Every single model we build is based upon the same template, with no exception. All we need to do is to adjust it to take our target into consideration.
While a custom model would be close to perfect, as the saying goes, we didn’t want perfect to be the enemy of good: Our template-based models perform very well and are able to determine the most relevant product for an ANWB member. We almost always see that the higher the predicted score of a member for a product, the higher the click- and conversion-rates.
How do your prediction models contribute to marketing automation?
All of our prediction models are scored in an automated daily process. The output of this process is a giant table of the scores of every member for every model. These scores are then input for some formulas to determine what is relatively the most relevant product to show to every member in our channels.
This table is also used by database marketeers for selecting relevant members for their 1-on-1 marketing campaigns, in channels like email. This has proved to be extremely efficient. Instead of coming up with their own rules and statements to arrive at e.g. 10,000 relevant campaign recipients, they simply select the 10,000 members with the highest scores for a certain product.
How often do you retrain your models?
Our models are automatically retrained. But first, our prediction models are evaluated automatically in KNIME to monitor how they’re performing. We use the Area under the Curve (AUC) metric for this. By tracking the change in AUC based on new data, we can monitor performance over time and ensure the models’ reliability and effectiveness in making predictions on the latest data. Based on this evaluation plus a number of additional business rules, our KNIME workflow automatically determines whether a model needs to be trained again. And if it does, this is also done automatically in KNIME.
How accurate are your models?
As I mentioned before, our goal is to make multiple ‘good’ models that are automatically scored, evaluated and trained, rather than having ‘perfect’ models which require way more resources to maintain. Still, we’ve developed models that predict better than we had ever hoped for.
Not only do our nearly 100 models score an average AUC of 0.8, analysis shows that our models manage to correctly rank the right member for the right product at the right time. As a general guideline, a perfect AUC score is 1.0: A score between 0.7 and 0.9 demonstrates good to excellent performance. ANWB members that fall within the top 10% even show a conversion rate of over 5 times higher than average. With models that perform at this level, we can perfectly serve five million members daily, with only 2 full-time employees.