Benchmark reporters: show your data, please

From the introduction of a recent benchmark study:

“This study will help you to understand what metrics you should be analyzing when reviewing your email marketing program, what goals you can set based on similar organizations, and how your email marketing program is performing”

– sounds promising. Next, under “Key Findings” it comes up with a remarkable outcome saying that

“shorter subject lines continued to outperform their longer counterparts. Fewer than 10 characters achieved the best open rate at 58%.”

The group of mailings with super-short subject lines naturally stands out in a bar chart which illustrates average open rates across different length ranges (see figure right). So far, so good.

However, the question of what you can really make out of such numbers often catches me lately. Should I prune my newsletter subject lines down to 10 characters, now? I don’t think so. To anticipate my view: these finding are useless or even harmful, if presented this way. Why is that?

The 10’ers – are they possibly hiding something?

For fun, I just queried my email inbox for subject lines which contain less or equal than ten characters. The result lists ten subject lines out of 8772 emails (German left, English translation right):


subject line

We ♥ Sale
WSV Winter sale
Danke Thanks
Gewonnen! [You] won!

Well, none of them sounds convincing, do they? Nevertheless those should be the possible winning options? Oh, wait. What if we take a look behind the bar chart? Imagine, the results might look like this (left original, right our fictional (!) details):

Suddenly, the 10’ers have lost their sparkle. The huge eye-catching bar on the left is based upon the average of only a few observations. Why is that? Perhaps because of just one sender who delivers his quarterly employee newsletter? Also, several bins contain a different amount of spread. Using 50 to 59 character long subject lines provides for example a very high confidence in achieving 42% open rate. Are there some clusters in the latter bin which represent more groups? Et cetera.

Such details can put a completely new perspective on the outcomes. More information, less ink, and at the same time no loss in clarity. The informational value increases significantly. At the same time, it reduces the danger of making misguided decisions.

Two side notes

Of course, companies have reasons for spreading reports in such ways. The more details you disclose about your methodology, the more vulnerable you are to naggers. It raises new questions, takes additional effort to extract meaningful insights from data, and so on.

Also, note that IMHO writing effective subject lines is about front-loading your important keywords; it’s about choosing the right words and placing them in their best order. It’s a commonly accepted best practice to do so. Length itself on the other hand may be one of many test parameters for subject lines.


By mentioning this I want to increase awareness for the underlying problems of such studies, or rather the presentation of their outcomes. The web is more and more being drowned out by junk numbers and superficial pseudo-insights. They get retweeted and cited on and on – partly even more than the real nuggets, which then have difficulties ascending to the surface. And with the number of citations or quoting authorities, the initial junk numbers grow more and more into facts. In the end, companies have no incentive to show their data in a reasonable fashion. But this would really “help you to understand what metrics you should be analyzing when reviewing your email marketing program […]”.


Google for instance shows with their study “Our Mobile Planet” that open data doesn’t hurt. Survey data can be downloaded in a machine readable format. That’s examplary and people will demand for it more and more.

Enjoyed this one? Subscribe for my hand-picked list of the best email marketing tips. Get inspiring ideas from international email experts, every Friday: (archive♞)
Yes, I accept the Privacy Policy
Delivery on Fridays, 5 pm CET. You can always unsubscribe.
It's valuable, I promise. Subscribers rate it >8 out of 10 (!) on average.

4 Responses to Benchmark reporters: show your data, please

  1. Benchmark reporters: show your data, please

  2. Benchmark reporters: show your data, please

  3. Pingback: Email & data analysis: Does timing affect open rates? An analysis of variance (ANOVA) | E-Mail Marketing Tipps

  4. Pingback: comScore study might attach new importance to measuring open durations | E-Mail Marketing Tipps

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.