Email response modeling using decision trees – a practical example

Response modeling in most cases means mapping past reactions of your subscribers using statistical algorithms in a way that the outcome for each recipient (e.g. click or no click, open or no open, …) is explained by several explanatory attributes (e.g. age, email’s remote part, salutation, …). Such a view may help marketers, among other things, to …

• simply get a better understanding of what was going on under the surface of the last campaigns;
• predict the outcome of future email actions for specific segments.

Of course, open, click, and bounce rates are good and valuable performance indicators. However, they don’t tell you at once for example what groups of recipients showed more clicks and opens than others. And this would be a really interesting insight that holds great potential for optimizations. Decision tree models provide one easily interpretable representation of such mapped response behaviour. Let’s look at a practical example and explore how we could possibly make more of our data.

Example: Drill down into 30% open rate

Consider the following sample of response data. We define “response” as “has opened” our last email campaign. Email opens are measured by either hitting a tracking pixel or tracking a click at our email servers. (Although this may violate some standardization efforts.)

That being said, the sample data set contains 500 rows with 30% responders. So, conversely, 70% didn’t open our email. Furthermore, each row, which stands for one unique email recipient, also contains the attributes “age”, “lifestyle”, “zipcode”, “familystatus”, “car_value”, “sports_type” and “earnings per year”. Those fields come from our CRM solution and are needed to be able to describe differences between responders and non-responders.

Now, by looking at a typical list like this, it doesn’t tell us anything of interest:

This changes as soon as we model the response using some more sophisticated concepts, such as a decision tree algorithm. Decision trees are part of most statistical packages today. After some minor parameterization, we get the following picture out of the data above:

Interpretation

First, the class label “false” means “has not opened”, and “true” means “has opened” in the tree figure above. I will spare you all the details about what the algorithm did – there’s plenty of information available on the web concerning the different tree families, possible split-criterions, proper evaluation, general advantages, potential problems and extensions like pruning and boosting.

At this point, email marketers need to know that the tree has to be read from top to bottom. The root on top contains the most important criterion for splitting responders and non-responders. In our campaign it was the recipient’s age as indicated by the attribute “rec_age”. Let’s follow the tree structure to its leaf on the right. The majority class in each leaf is highlighted in gray. So on the right it says: in the group of “over 55s”, a phenomenal 84.4% opened our email. In addition, with a total of 125 rows (= 25% of our 500 sample), this segment seems also rather big and therefore very important. That’s good to know. But … who didn’t respond to our offer … and why …?

Let’s follow the node on the left side – the group of “under 56s”. The tree algorithm identified the recipient’s earnings as the second most important split criterion. In this group, only the rich ones, earning more than about $139,000 felt addressed by our email advertisement. And – going one level deeper – also those who earn less than $139,000 while living in zip codes less than or equal to 14756 responded.

Conclusion: Remarketing-campaign?

One customized follow up emailing addressing those 332 subscribers, who are …

1. younger than 56 years old, and
2. who earn less than $139,000, and
3. who live in zip codes greater than 14756

could possibly lift the overall result of the campaign. Note that this segment would have remained completely undetected if the email marketer just looked at his standard key performance indicators.

A general side note:

Don’t get me wrong: Response modeling isn’t as easy as it may seem, now. On the contrary. Most email response data is very noisy. Besides choosing the right parameterization, one must also have, find and deliver the right discriminatory input variables, which can split up the data into responders and non-responders. But after all I think we should get our little things done first before we turn on to trendy stuff like big data”. Here’s the clean data set for you to experiment with.