[Split-Test] Email signups doubled AND bounces reduced at the same time – see how

I want to share some testing results with you. For a couple of weeks, I ran a split A/B/C/D test on this website. The goal was to see, if I could convert more visitors to email subscribers. “Which test won”… can you guess it? The results are quite interesting.

Test candidates

The screenshots below show the original form (A) plus three challengers (B, C, and D). As you can see,

  • variation B contains a white headline on green background color (= blog theme style).
  • C has got a yellow background (text marker style).
  • And D contains a little animated gif that points to the form.

Your guess?

Now, before we dive into the details – which one of these outperformed the worst one by a factor of 2.5 (= 150% uplift)? What would you say spontaneously without looking at the results below? ;-) (Note that the test is still running today on 28 September 2012. Therefore, the form you’ll see on the right doesn’t necessarily have to be the winner.)

Pick one in the form below and click “vote”.

 

 

Actual results:

The figure below left shows a screenshot of the results in Google Analytics:

Yes, you saw that right: the ugly text marker style performed nearly twice as good (or “bad”, if you consider the low conversion rate) as the original headline. The test achieved a confidence level of 90% as the figure above right shows. I.e., we got significant differences between groups. The uplift is not solely due to chance. (Read this post to get more information on what this and the p-value mean.)

To be honest, I was a little sad when I first saw the table. That’s because my personal favorite, the animation, really lost. In fact, it did even worse than the original one. My guts told me something different when I launched the test.

But that’s not all. The text marker styled variation did not only convert more visitors. It also had the lowest bounce rate (99.9% significance level). That means, it somehow increased the likelihood that visitors stayed for more than one page view. C had the most page impressions per visit, whereas the animated gif provided the highest bounce rate.

Lesson learned: don’t trust your gut feeling. Do a test instead! it’s easy, and online marketing is full of surprises.

A/B tests using WordPress and Google Analytics

A last note goes to the test code. The reason for doing the test was originally because I instantly wanted to play around with this cool code snippet. I found the use of Google Analytics custom variables to be a creative, and yet straight forward way to do split tests in WordPress. Therefore, I also did not bother much about the variations themselves (if so, I’d probably not have tested different colors).

The concept of course has some drawbacks. Like e.g. it manipulates elements after loading the page, or like using asynchronous tracking. But it’s still ok for me.

I’m using the Google Analytics for WordPress Plug-in. In case you want to try it for yourself, the following code fitted perfectly into the “custom code” field:

_gaq.push(function () {
var a = _gat._getTrackerByName(),
b = a._getVisitorCustomVar(1),
c = Math.random();
if (typeof b == "undefined") {
c < 0.25 ? (b = "A") : c < 0.50 ? (b = "B") : c < 0.75 ? (b = "C") : (b = "D");
_gaq.push(["_setCustomVar", 1, "A/B Test (Button)", b, 1]);
}
var d = document.getElementById("signup_title"),
e = document.getElementById("signup_pointer");
switch (b) {
case "A":
d.innerHTML = "Weekly Email Signal";
break;
case "B":
d.innerHTML = "<span style=\"display:block;background-color:#478945;color:#FFFFFF;\">&nbsp;Weekly Email Signal</span>";
break;
case "C":
d.innerHTML = "<span style=\"display:block;background-color:#FFFF00;color:#000000;\">&nbsp;Weekly Email Signal</span>";
break;
default:
e.innerHTML = "<img src=\"http://epic.c0ol.de/gfx/pointer7.gif\">";
d.innerHTML = "Weekly Email Signal";
}
});

Any explanations why the yellow background color worked that well?

Testing is fun, right? :-) I'm always keen to find new interesting experiments on the web. Sign up for my Friday-news and I'll keep you posted: (archive♞)
Yes, I accept the Privacy Policy
Delivery on Fridays, 5 pm CET. You can always unsubscribe.
It's valuable, I promise. Subscribers rate it >8 out of 10 (!) on average.

10 Responses to [Split-Test] Email signups doubled AND bounces reduced at the same time – see how

  1. [Split-Test] Email signups doubled AND bounces reduced at the same time – see how http://t.co/fhsfPAqS via @LukeAnker

  2. [Split-Test] Email signups doubled AND bounces reduced at the same time – see how | E-Mail Marketing Tipps – http://t.co/A8NmAIkQ

  3. Split test. Simple and quick page change doubled signups http://t.co/sjx3sLMf also increased email address entry accuracy

  4. RT @KISSmetrics: [Split-Test] Email signups doubled AND bounces reduced at the same time – see how http://t.co/wUnIvavT #measure

  5. RT @KISSmetrics: [Split-Test] Email signups doubled AND bounces reduced at the same time – see how http://t.co/eiHKQ9I8 #measure

  6. RT @KISSmetrics [Split-Test] Email signups doubled AND bounces reduced at the same time – see how http://t.co/oHxIspPo #measure

  7. [Split-Test] Email signups doubled AND bounces reduced at the same time – see how – http://t.co/ky6A2ycH | #emailmarketing Tipps #marketing

  8. K. Buettner @newsletternews [Split-Test] Email signups doubled AND bounces reduced at the same time http://t.co/J6ll2WlL #emailmarketing

  9. My guess was right – try it yourself! RT @mayoris: Interessante E-Mail Split Test Resultate http://t.co/WQ4d9lzL

  10. [Split-Test] Email signups doubled AND bounces reduced at the same time – see how – http://t.co/ky6A2ycH | #emailmarketing Tipps #marketing

Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

All information is voluntary. Your email address will not be published. When commenting, you agree that your IP address will be processed and stored by Askimet in the U.S. for the purpose of recognizing comment-spam.