Our analysis, taken from surveys given at the eMetrics Marking Optimization Summit, found that 52% of online marketing managers are currently engaged in A/B or Multivariate testing and that an even greater percentage plan to begin testing within the next year.
A high-level overview of the findings:
- A variety of testing platforms are considered before making a final decision
- Experiments vary in frequency, page type and page element
- Practitioners often use more than one platform to meet their needs
- Pretest hypothesis often fail
- Lack of Budget was cited as the main reason by those who currently do not do any testing
- Complexity of Tools and lack of best practices were two top challenges faced by those engaged in Testing
- A/B and Multivariate testing is “worth it”
- Behavioral targeting is a small but growing practice
Surprisingly lack of budget was one of the top reasons why companies were not involved in A/B or Multivariate Testing. But if you look at the chart below you will see that companies that do conduct A/B and Multivariate testing find it that it’s worth it. Majority of the respondents who engage in testing said that either it is worth it or can’t live without it. So if you are one of those who think because of lack of budget you can’t do A/B MVT, think again? Your Return will make it worthwhile to do testing. You will be able to move budget from other areas such as Ad spend to A/B testing. Do your ROI calculations. You will be surprised with the positive results you will get with developing a culture of continuous testing.
You can download the whitepaper from ZeroDash1 website.
--------------------------------------------------------------------------------
Looking to fill your Web Analytics or Online Marketing position? Try WebAnalytics Job Board
New Position
Web Data Analyst at Alzheimer's Association (Chicago, IL)
---------------------------------------------------------------------------------
Hey Anil, can you comment on the potential for sample bias in this research? When I see numbers like this -- more than 50% of companies doing some type of AB MVT I simply cannot reconcile that against my personal knowledge of the larger community doing web analytics. Is there any chance you may have accidentally sampled a more sophisticated set of practitioners/companies and are thusly seeing an outsized proportion of positive responses?
ReplyDeleteBased on conversations with other industry leaders I estimate the number of companies "actively" (e.g., regular attempts to optimize via testing) using this type of technology at (perhaps substantially) less than 1,000 worldwide. Now, this is just a guess, but it vets well with many other data points (known customer bases for Offermatica/Optimost/etc.)
Thoughts?
Otherwise this is great and valuable research. Thanks!
Eric T. Peterson
CEO, Web Analytics Demystified
http://www.webanalyticsdemystified.com
Eric brings up a very good point. We did have a discussion about this internally. We assumed that eMetrics gets all levels of attendees, novice as well as sophisticated.
ReplyDeleteThere is a possibility of a sample bias and further studies will be required to confirm that.
Thanks
Anil