Do you understand Taguchi?
Ok there’s a storm coming to Internet Marketing in the form of Multi Track Generator from the stable of Armand Morin.
But what exactly is Taguchi. Some of the blog posts and websearches don’t actually reaveal how to do it, just what it is.
I got hooked into it after reading about it via Perry Marshall’s website… but I’m surprised he doesn’t mention applying it to finding the best adword ad???
So to save you searching - here’s both. What it is and how it can be calculated. (If you’re a bit scared of math then don’t worry I’ll try and keep things simple).
What is Taguchi, used for in terms of Internet Marketing and Website Tracking?
To understand this you have to first understand what split testing is. In essence, split testing is where you create two versions of an advert (i.e. your sales webpage) with a simple difference. One of them will convert more often than the other. By monitoring the conversion rate you can work out which page is more profitable.
Most people don’t do this. They throw up an ‘ecommerce’ website and never test out new headlines, new graphics, new fonts etc…. The ’smart’ marketers do test - repeatedly.
This is a tried and tested process directly from the father of Scientific Advertising - Claude Hopkins. Now the problem comes in two ways when you start testing multiple items.
For a split test of two headlines you need to get enough traffic to both to make a good estimate of which one works better. So if I show each one 200 times and headline A gets 2 sales and headline B gets 1 sale, then you’d think headline A was better, wouldn’t you?
Well in an ideal world it is. However if you then go and test two subheadlines, just using headline A, you are left asking the question would I have got more sales if I’d put either of these two subheadlines with headline B?
This gets more and more complicated both for the more variations of each item (say four headlines, three subheadlines, two font colours). As an aside this would take 4×3x2 different variations i.e. 24 different pages to test - at 200 hits a page that would be 4800 hits for a minimal guess at which one is best… probably even more. If you were driving traffic to the site during this optimisation time (which many people do) using Adwords it could cost $480 or more just to find the right page.
If we only used split testing we might miss that elusive combination that doubles or triples the conversion rate of the sales page. To make sure we find that combination, we would have to test out each and all the possible variations for the same amount of time, and it can take a very long time to find out which variation is best.
But I could have hundreds of variations and how do I manage and test them all?
The hard way is to run all of them for a set time or number of hits and then move on. Time consuming, boring and prone to error - even by the most meticulous of us (and I’m not!)
We could use software to display all the variations round-robin (one after the other) and track sales against that show. There are software products out there to do that… But there is a simplified way…
Taguchi!
Taguchi designs come from work done by Dr. Genichi Taguchi. He created methods that allowed engineers to find the best or optimal designs for their problems that had multiple variables and each variable had multiple values. (sound familiar???).
So in his world for example, an engineer would be designing a machine part and would have things like the tolerance, size, heat expansion and flexibility as variables. Each would have various possible values and the problem was to find the most optimal set of values by testing. Before using Taguchi (and other estimating methods), either a part was made and accepted as was - regardless of inefficiency and cost, or millions would have to be spent doing tests on each value/variation of the part.
Taguchi divides the problem down by associating an equal portion of each variable across a smaller number of tests. By using something called an orthogonal array you can reduce the number of tests and work out which is the best or optimal value. There are places over the internet that show these orthogonal arrays, as well of countless numbers of them in the Taguchi text books.
Here’s one for a simple example that allows you to test 3 different variables each with 2 values - using only 4 tests instead of 8!
For a web page we could test 1)Headline 2) Subheadline 3) Font
we’d have headline1, headline2, font1, font2, subhead1, subhead2
The Orthogonal Array (Called L4 but don’t worry about that now) is
This shows us what each test page should contain ( Test 1 has headline1, subhead1, font1; Test4 has headline2, subhead2 and font1)
Now we run the tests, same amount of hits for each one (or as close as we can get) for a while, recording the conversions for each test.
Now we work out the conversion ratio of the test page which is simple - conversions/hits. This is usually a small number so we multiply it by 100 to become a percentage.
i.e. if we had 200 hits and 1 conversion that would be 1/200= 0.005 ; 0.005x 100 = 0.5 % conversion ratio.
Now for the fun bit.. how to work out which ones are the best….
We work out the averages for headline1 by adding the conversion ratio for Test1 and Test 2 (where the headline appeared) and then we divide by 2 (two tests)
H1 = (CR1+CR2)/2
Do the same for headline 2 (results from test3 and 4)
H2 = (CR3+CR4)/2
Now the biggest number between the two is the better convertor.
We do the same for subhead and font.
S1 = (CR1+CR3)/2
S2 = (CR2+CR4)/2
Ouch you say, but doing this in a spreadsheet is easy and only takes a few minutes to set up…
This is part of the way there. The real amazing results come from the fact that you can do a further figure fiddling using a special math function that shows not only the best result, but how much effect that result had. So if two headlines were pretty close to each other in results then it could be that neither one wins but one has a much quicker result than another. Or another headline might prove better because there is not much to choose between these two.
(for math geeks only the one of the many formula is -10log(sigma(1/CR^2) for the maximum delta change - apply it to each of the percentages and recalc using abs on each.. to find the effect of selecting one or the other value i.e. max rate of change).
The end result from all this math is that you can work out with (in this case) between 1/2 to 1/3 of the total number of hits, which variation will give the biggest benefit.
Now I did this actual test with a couple of Adwords and the result was astounding - one ad went from a 0.4% Click through to 1.9% with just 29 clickthroughs to work out what ad to select.
Now the results are even more astounding for larger arrays - L8 - test seven pairs (usually 128 tests) with 8 tests i.e. ~6% of the time if done with split testing..
Or even better L12 - test 11 pairs (usually 2048 tests) with 12 tests and do it in 0.6 % of the time - saving months if not years…
L9 - test 4 things each of which has three options. (usually 89 tests) with 9 tests and do it in 10% of the time..
You see the benefit.
However doing it on a website proved a little tricky until Multi Track Generator came along. This software automates all the math, all the ad variation and automatically optimises the page over time. Yes, it takes a bit of time to set up to begin with but then it can be left to work out the best advert. The hard part for many people after it’s installed is actually deciding what copy to try out. It doesn’t make up for poor copywriting in the first place but it does allow for many variations and values that other techniques would not cover. It’s more than Taguchi!
Other products in this field… there are two I know of at the moment. The first is expensive at nearly $1000 called MR2 - you still have to manually create the tests. The second is free but will break if you put zero’s in and again you have to manually create the pages.
Of course if you want a spreadsheet created that does the manual calculations for you then drop me an email… I might be persuaded to create one.
But what exactly is Taguchi. Some of the blog posts and websearches don’t actually reaveal how to do it, just what it is.
I got hooked into it after reading about it via Perry Marshall’s website… but I’m surprised he doesn’t mention applying it to finding the best adword ad???
So to save you searching - here’s both. What it is and how it can be calculated. (If you’re a bit scared of math then don’t worry I’ll try and keep things simple).
What??? he’s giving away how Taguchi works!!!
What is Taguchi, used for in terms of Internet Marketing and Website Tracking?
To understand this you have to first understand what split testing is. In essence, split testing is where you create two versions of an advert (i.e. your sales webpage) with a simple difference. One of them will convert more often than the other. By monitoring the conversion rate you can work out which page is more profitable.
Most people don’t do this. They throw up an ‘ecommerce’ website and never test out new headlines, new graphics, new fonts etc…. The ’smart’ marketers do test - repeatedly.
This is a tried and tested process directly from the father of Scientific Advertising - Claude Hopkins. Now the problem comes in two ways when you start testing multiple items.
For a split test of two headlines you need to get enough traffic to both to make a good estimate of which one works better. So if I show each one 200 times and headline A gets 2 sales and headline B gets 1 sale, then you’d think headline A was better, wouldn’t you?
Well in an ideal world it is. However if you then go and test two subheadlines, just using headline A, you are left asking the question would I have got more sales if I’d put either of these two subheadlines with headline B?
This gets more and more complicated both for the more variations of each item (say four headlines, three subheadlines, two font colours). As an aside this would take 4×3x2 different variations i.e. 24 different pages to test - at 200 hits a page that would be 4800 hits for a minimal guess at which one is best… probably even more. If you were driving traffic to the site during this optimisation time (which many people do) using Adwords it could cost $480 or more just to find the right page.
If we only used split testing we might miss that elusive combination that doubles or triples the conversion rate of the sales page. To make sure we find that combination, we would have to test out each and all the possible variations for the same amount of time, and it can take a very long time to find out which variation is best.
But I could have hundreds of variations and how do I manage and test them all?
The hard way is to run all of them for a set time or number of hits and then move on. Time consuming, boring and prone to error - even by the most meticulous of us (and I’m not!)
We could use software to display all the variations round-robin (one after the other) and track sales against that show. There are software products out there to do that… But there is a simplified way…
Taguchi!
Taguchi designs come from work done by Dr. Genichi Taguchi. He created methods that allowed engineers to find the best or optimal designs for their problems that had multiple variables and each variable had multiple values. (sound familiar???).
So in his world for example, an engineer would be designing a machine part and would have things like the tolerance, size, heat expansion and flexibility as variables. Each would have various possible values and the problem was to find the most optimal set of values by testing. Before using Taguchi (and other estimating methods), either a part was made and accepted as was - regardless of inefficiency and cost, or millions would have to be spent doing tests on each value/variation of the part.
Taguchi divides the problem down by associating an equal portion of each variable across a smaller number of tests. By using something called an orthogonal array you can reduce the number of tests and work out which is the best or optimal value. There are places over the internet that show these orthogonal arrays, as well of countless numbers of them in the Taguchi text books.
Here’s one for a simple example that allows you to test 3 different variables each with 2 values - using only 4 tests instead of 8!
For a web page we could test 1)Headline 2) Subheadline 3) Font
we’d have headline1, headline2, font1, font2, subhead1, subhead2
The Orthogonal Array (Called L4 but don’t worry about that now) is
Test # Headline Subhead Font Conv Ratio
1 1 1 1 CR1
2 1 2 2 CR2
3 2 1 2 CR3
4 2 2 1 CR4
This shows us what each test page should contain ( Test 1 has headline1, subhead1, font1; Test4 has headline2, subhead2 and font1)
Now we run the tests, same amount of hits for each one (or as close as we can get) for a while, recording the conversions for each test.
Now we work out the conversion ratio of the test page which is simple - conversions/hits. This is usually a small number so we multiply it by 100 to become a percentage.
i.e. if we had 200 hits and 1 conversion that would be 1/200= 0.005 ; 0.005x 100 = 0.5 % conversion ratio.
Now for the fun bit.. how to work out which ones are the best….
We work out the averages for headline1 by adding the conversion ratio for Test1 and Test 2 (where the headline appeared) and then we divide by 2 (two tests)
H1 = (CR1+CR2)/2
Do the same for headline 2 (results from test3 and 4)
H2 = (CR3+CR4)/2
Now the biggest number between the two is the better convertor.
We do the same for subhead and font.
S1 = (CR1+CR3)/2
S2 = (CR2+CR4)/2
Ouch you say, but doing this in a spreadsheet is easy and only takes a few minutes to set up…
This is part of the way there. The real amazing results come from the fact that you can do a further figure fiddling using a special math function that shows not only the best result, but how much effect that result had. So if two headlines were pretty close to each other in results then it could be that neither one wins but one has a much quicker result than another. Or another headline might prove better because there is not much to choose between these two.
(for math geeks only the one of the many formula is -10log(sigma(1/CR^2) for the maximum delta change - apply it to each of the percentages and recalc using abs on each.. to find the effect of selecting one or the other value i.e. max rate of change).
The end result from all this math is that you can work out with (in this case) between 1/2 to 1/3 of the total number of hits, which variation will give the biggest benefit.
Now I did this actual test with a couple of Adwords and the result was astounding - one ad went from a 0.4% Click through to 1.9% with just 29 clickthroughs to work out what ad to select.
Now the results are even more astounding for larger arrays - L8 - test seven pairs (usually 128 tests) with 8 tests i.e. ~6% of the time if done with split testing..
Or even better L12 - test 11 pairs (usually 2048 tests) with 12 tests and do it in 0.6 % of the time - saving months if not years…
L9 - test 4 things each of which has three options. (usually 89 tests) with 9 tests and do it in 10% of the time..
You see the benefit.
However doing it on a website proved a little tricky until Multi Track Generator came along. This software automates all the math, all the ad variation and automatically optimises the page over time. Yes, it takes a bit of time to set up to begin with but then it can be left to work out the best advert. The hard part for many people after it’s installed is actually deciding what copy to try out. It doesn’t make up for poor copywriting in the first place but it does allow for many variations and values that other techniques would not cover. It’s more than Taguchi!
Other products in this field… there are two I know of at the moment. The first is expensive at nearly $1000 called MR2 - you still have to manually create the tests. The second is free but will break if you put zero’s in and again you have to manually create the pages.
Of course if you want a spreadsheet created that does the manual calculations for you then drop me an email… I might be persuaded to create one.
May 31st, 2007 at 2:28 pm
Really enjoyed this article. Would love a copy of the spreadsheet. Which arrays have you had the most succes with. Thank you for writting this article. Great Job! Dave