tag:blogger.com,1999:blog-18012046.post5573533763261379017..comments2024-03-25T23:56:43.770-07:00Comments on Digital Marketing and Analytics by Anil Batra: Significance of Statistically Significant Results in A/B TestingAnil Batrahttp://www.blogger.com/profile/10987449618439416854noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-18012046.post-16238502882146458752011-03-11T11:17:19.895-08:002011-03-11T11:17:19.895-08:00Hi Olga,
Yes this is from GWO. Will you be able t...Hi Olga,<br /><br />Yes this is from GWO. Will you be able to send me a copy of your chart?Anil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-6984622564818831012011-03-11T10:30:50.332-08:002011-03-11T10:30:50.332-08:00Thanks for the post. Is it Google website optimize...Thanks for the post. Is it Google website optimizer test? We had several tests with this tool & every time got the same charts: during 1st two weeks of testing there's a significant difference between combinations, in 2 weeks the chart shows that there's no difference between them at all.Olgahttps://www.blogger.com/profile/04128434628605517349noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-79679388021563355812010-09-20T16:25:48.406-07:002010-09-20T16:25:48.406-07:00I think a problem with the null hypothesis formali...I think a problem with the null hypothesis formalism is that can ironically paralyze decision making. You can get stuck worring that you are picking the 'wrong' answer. However, it might be better to think about the decision problem in terms of revenue maximizing (or minimizing regret) rather than winners and losers or right ones and wrong ones.<br /><br />A couple of things to think about before going the null hypotheses route:<br />1)Opportunity Costs - it costs you to learn, so you want to do it efficiently. Sig testing as discussed above does not account for the lost reward by playing suboptimal options.<br />2)The internet is real-time! There is often no need to arbitrarily pick a time (picking a confidence level is arbitrary) to force yourself to play a pure strategy from then on. Think in continuous terms rather than in discrete terms. There is no reason you can't keep all options available but decrease the frequency with which they are played as you move through time and learn more about them. So play a mixed (portfolio) strategy. Which brings me to the last point<br />3) The environment need not be stationary. More likely than not the environment that your application is making its decisions in is non-stationary, so the notion of a 'winner' may not make any sense. <br /><br />Often, these types of decision problems can be better modeled as a bandit problem rather than as a hypothesis test problem. <br />Of course it depends on what you want to learn and why. If you need to make generalizations about the world, than you prob want to get some confidence around your statements and should run more formal tests, but if you just want to optimize an online process, than I'm not sure that sig testing is always the way to go. I can guarantee that Google, Yahoo! etc are not running null hypothesis tests on Ad placements - they are running bandits. <br /><br />ThanksMatt Gershoffhttp://www.conductrics.comnoreply@blogger.comtag:blogger.com,1999:blog-18012046.post-76538043939848287752010-02-08T21:29:22.245-08:002010-02-08T21:29:22.245-08:00Jason, I agree with you.Jason, I agree with you.Anil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-8219111947203474112010-02-08T21:29:03.564-08:002010-02-08T21:29:03.564-08:00Suresh,
I always advice to wait but there are case...Suresh,<br />I always advice to wait but there are cases when the test runs too long without giving any conclusive results in that case you need to test a different variation. <br />In this case it seems like picking yellow might not have been a wrong decision but it would not have been the right decision either. (Also see my answers above).Anil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-46468153109362101102010-02-08T21:26:14.431-08:002010-02-08T21:26:14.431-08:00Giadascript, the test is still running. All the tr...Giadascript, the test is still running. All the treatments are getting equal traffic.Anil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-86368331416635235862010-02-08T21:25:23.427-08:002010-02-08T21:25:23.427-08:00Barbara,
Since the test has not completed yet I c...Barbara,<br /><br />Since the test has not completed yet I can not say I picking yellow would be a mistake or not. The point I am trying to make is that you have to wait for statistically significant results before declaring a winner (or looser) as results can change (as the chart shows).Anil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-2018349901129968712010-02-08T21:23:39.517-08:002010-02-08T21:23:39.517-08:00Suarbah, Once you reach statistical significance,...Suarbah, Once you reach statistical significance, the results should not change, assuming all other variables don't change.<br />This example shows that results can change early on and you should not make decisions in a hurry without getting statistically significant results. <br /><br />Google has a calculator that might help, https://www.google.com/analytics/siteopt/siteopt/help/calculator.htmlAnil Batrahttps://www.blogger.com/profile/10987449618439416854noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-57674426884927947252010-02-07T17:12:17.872-08:002010-02-07T17:12:17.872-08:00Great post!
The variations are trending so close...Great post! <br /><br />The variations are trending so closely - makes me think you need to go back to the drawing board and start over!<br /><a href="http://bit.ly/a2uVxU" rel="nofollow">http://bit.ly/a2uVxU</a>MicroArts Creative Agencyhttps://www.blogger.com/profile/15526388130673938702noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-24001485680651242332010-02-07T08:16:49.552-08:002010-02-07T08:16:49.552-08:00Nice post. I think even more important than letti...Nice post. I think even more important than letting your AB tool pick a winner for you, you should know the statistics behind the calculations so that your results are tool independent.Jason Thompsonhttp://emptymind.orgnoreply@blogger.comtag:blogger.com,1999:blog-18012046.post-50426159302252119372010-02-06T11:57:38.424-08:002010-02-06T11:57:38.424-08:00Anil, nice post! But some would argue that picking...Anil, nice post! But some would argue that picking yellow early wouldn't have a big negative impact (based on the graphs) and indeed you would be applying testing correctly - test, fail/succeed quickly, and then retest. It would be interesting to see the analysis around when testers must make a decision and when it pays to wait.Suresh Vittalnoreply@blogger.comtag:blogger.com,1999:blog-18012046.post-70070419120263831282010-02-04T16:58:43.110-08:002010-02-04T16:58:43.110-08:00This can also happen if your client has trend shif...This can also happen if your client has trend shifts in their traffic stream - of course you have to draw the line at some point or another or you'll be in perpetual testing mode forever. <br /><br />Having said that, there are SOME times when the test page completely tanks or smokes the original page - but in most instances I agree it's better to wait it out. <br /><br />I like the post. short and sweet.Loganhttps://www.blogger.com/profile/12921597903072079557noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-43861509878214539862010-02-03T13:56:11.770-08:002010-02-03T13:56:11.770-08:00Excellent post, Anil. It's so critically impor...Excellent post, Anil. It's so critically important we understand significance before just quickly making decisions on very early results. I wrote about this a bit in a post called "Wanna be better with metrics? Watch more poker and less baseball" (http://www.retailshakennotstirred.com/retail-shaken-not-stirred/2009/11/wanna-better-understand-metrics-watch-more-poker-and-less-baseball.html) and in that post included a free spreadsheet that can be used to determine statistical significance. I thought your readers might find it useful. <br /><br />Thanks for an excellent post!Kevin Ertellhttp://www.retailshakennotstirred.comnoreply@blogger.comtag:blogger.com,1999:blog-18012046.post-43084785990042897342010-02-03T11:58:35.583-08:002010-02-03T11:58:35.583-08:00While i agree with you conclusion, and prefer to h...While i agree with you conclusion, and prefer to have a stat.sig results, here is my question: would we have made a huge error by selecting yellow early (if we had no time to test any longer) as it seems to work as good as any of the other options...Unknownhttps://www.blogger.com/profile/18369066091802053827noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-35867756211467652822010-02-03T10:07:26.701-08:002010-02-03T10:07:26.701-08:00Not only do you not have a winner, you might incor...Not only do you not have a winner, you might incorrectly choose the "blue" as the loser. Possible scenario is that Blue is some new treatment that users need to get used to or some new treatments are "exciting" or novel to start with but that effect doesn't last.<br /><br />I completely agree with your conclusions here. <br /><br />Our paper http://exp-platform.com/hippo_long.aspx<br /><br />talks in depth about Novelty and Primacy effects.Randynoreply@blogger.comtag:blogger.com,1999:blog-18012046.post-39600304497055934392010-02-02T23:59:52.571-08:002010-02-02T23:59:52.571-08:00This has to be the best explaining article on stat...This has to be the best explaining article on statistically significant data for a/b testing. <br /><br />One of those cases when an image is worth 1000 calculations :)<br /><br />Thank you.Claudiu Murariuhttp://padicode.com/blog/noreply@blogger.comtag:blogger.com,1999:blog-18012046.post-72612756500340164622010-02-02T21:05:41.509-08:002010-02-02T21:05:41.509-08:00Between two sets in an A/B split test, if the popu...Between two sets in an A/B split test, if the population of each set is sufficiently large and the difference in response rates is statistically significant, how will you know that the results may change in the future and one should not take a call based on one test?<br /><br />How many days should you be running the test to be sure that the results are statistically significant and will not change in the future?Saurabh Nandahttps://www.blogger.com/profile/00867453089820169282noreply@blogger.com