2 A/B Testing Lessons Learned from Amazon Video

It is no secret that Amazon is a data driven organization.  The culture of testing is ingrained into pretty much everything Amazon does. However, a company like Amazon also makes mistake as I learned from my recent experience with Amazon Videos.

I have two Amazon Fire TV Sticks, one attached to each TV that I have. Both are tied to the same Amazon account.  Using my Amazon Video account, I started watching a movie on one TV, turned it off halfway through and then a day later tried to watch it on the other TV. Guess what? I could not find that movie in any of the obvious menu options. I was expecting it to show up in my stream or at least in the same place where I found it last time.  Nope.  Amazon changed the order of movies on me. It’s not like it added new movies to the lineup and this movie got pushed down. They just reordered the existing movie selection. It appeared to me that some kind of movie display sorting experiments was going on. However, that experiment, ruined the experience for me as I spent a lot of time looking for that particular movie.

In another instance on Amazon Video, I found the movie but as I soon as I clicked on it, it vanished and the movie list was reordered. Weird, right? Again, my suspicion is that some movie sorting/ordering algorithm experiment (A/B test) was going. So, what can you learn from this experience?  There are two A/B testing (experimentation) lessons that you can learn from this mistake by Amazon:

1. Keep version consistency across devices – Everybody uses multiple devices these days, multiple TVs, Tablets, Phones etc. Make sure you provide same version of your layout/algorithm to the same user across devices. Which means that you must do testing at the customer level instead of session or visitor level.
2. Do not change customer experience midstream – Make sure all touch points are in sync and you’re A/B test does not change customer experience as customer interacts with the interface. In this particular case, if the movie was right in front of me on the home screen then pressing the button on the movie should not have triggered a test that rearranged the order of the movies displayed on my screen. The ordering/sorting should have been done before it was presented to me.

Questions? Comments?

Need help with A/B testing and Personalization? Reach me at batraonline@gmail.com.

Difference between Web Analytics and Digital Analytics

Web Analytics and Digital Analytics are quite often used interchangeably.  I have been asked, by my students and some clients, about the difference in these two, so I decided to write this short post to clarify the terms.

As you can see from the Google Trends graph, Google searches for “Digital Analytics” were nonexistent till Web Analytics Association changed its name to Digital Analytics Association. Since then the term "Digital Analytics" has started to pick up.

In early days of internet, companies started to analyze website data such as users, visitors, visits, page views etc. and the term used to describe this analysis was called “ Web Analytics”.

Then came other forms of online (digital channels) such as email, search, social, mobile etc. and increasingly Digital Analytics folks were including this data and analysis of all these channels to provide a complete view of the “Digital” channels, marketing and customers. To fully include the scope of work of “Web Analysts” a new term “Digital Analytics” was coined.

“Web Analytics” companies like WebTrends, Omniture (now Adobe), Google Analytics etc. also started including data from other online channels and transformed from Web Analytics tools to Digital Analytics tools.

When I was on the board of “Web Analytics Association” from 2009 – 2011, we had several discussions regarding the name of the association. The general consensus was that our members were doing much more than traditional “Web Analytics” and association needs to change the name and scope to include the changing role of "Web Analytics". Association finally changed the name to "Digital Analytics Association" on March 5th, 2012.

So back to the original question - What is the difference between Web Analytics and Digital Analytics?

Web Analytics is analysis of the website data.

Digital Analytics includes analysis of data from all digital channels that includes websites. Data from search, display advertising, social, email, mobile etc. is included to provide a complete view of the digital marketing and customers.

Though usage of Digital Analytics is picking up, “Web Analytics” is still searched more often than “Digital Analytics” as shown in the following Google Trends chart

Thoughts? Comments?

2 Key Lessons From Facebook’s Video Views Metrics Fiasco

People have short term memory (or selective memory), when they can’t remember things they will resort to how they think something should be. Recently Facebook was in the hot seat because of this very reason. 

Facebook metrics definition issue

Facebook has a metrics called “Video View” for video ads.  In this metric they only counted the video as viewed if it was watched more than 3 seconds by the viewer.  In other words, if someone watches a video for 2 seconds then that video view won’t be counted as a view in this metric.

Facebook also has another metrics, called “Average duration of Video views”, the “standard” definition of it should be Total Time spent watching video divided by Total Viewers. However, that’s not how Facebook defined it.  In Sept Wall Street Journal reported that Facebook "vastly overestimated average viewing time for video ads on its platform for two years.”  This lead to an apology from Facebook

About a month ago, we found an error in the way we calculate one of the video metrics on our dashboard – average duration of video viewed. The metric should have reflected the total time spent watching a video divided by the total number of people who played the video. But it didn't – it reflected the total time spent watching a video divided by only the number of “views” of a video (that is, when the video was watched for three or more seconds). And so the miscalculation overstated this metric. While this is only one of the many metrics marketers look at, we take any mistake seriously.

As per DM News article, Facebook did state the definition when it rolled out this metric two years ago.  So it was not actually doing anything wrong.  It was a case of short term memory issue.

“The problem, as critics put it, is a problem of omission. While Facebook very clearly states that it's only counting views as any video-play event that lasts longer than three seconds, it does not go out of its way to explicitly beat readers over the head with the fact that this definition of a "video view" applies equally to the calculation of average duration of video views.”

If Facebook product team had read my posts from 2012 on “Creating a culture of analytics” then they might have likely avoided this “scandal”. The two issues that Facebook dealt with were the exact same ones I talked about in my posts. To recap, here are the gist of those two posts:

• Make sure everybody understands clearly how metrics are defined.  I talked about this in my post in 2012, Standard Definitions of Metrics: Creating a Culture of Analytics. In that post I said 

Lack of standard definitions for the metrics causes people to report different numbers for supposedly same metrics, leading to confusion and total lack of trust in data.  No trust in data means that nobody is going to use the data to make strategic decisions and there goes all your efforts to create a culture of Analytics.

Having standard definitions is not as easy as it sounds.  It starts from you and your team having a clear understanding on how to calculate various metrics.   Some seemingly simple metrics can be calculated in various different ways and all of those ways might be right but getting one standard way of calculating those removes any confusion and gets everybody on the same page.

• People have short term memory.  In my 2012 post, titled  Dealing with Short-Term Memory: Creating a Culture of Analytics,  I wrote:
  
  We all make assumptions from time to time; sometime we state them clearly and sometimes we just assume in our own head. We then operate under those assumptions.  In context of Analytics, one such assumption is that everybody knows what the goals and KPIs are.  We might have defined them on the onset of the program, campaign, beginning of month, quarter, year etc., but once those are defined we start to assume that everybody knows about them and is operating keeping those goals in mind.
  
  Well the truth is that people have short term memory. They do forget and then start to interpret the KPIs, defined based on those goals, in their own way.  As the Analytics head/analyst/manager, it is your job to constantly remind stakeholders of the goals and KPIs. 

Two Lessons

This fiasco provides two great lesson for all the Digital Analytics teams.

1.       Clearly define your metrics and make sure the underlying metrics and calculations are clear in your definition.

2.       Don’t make any assumptions, people have short term memory. Just because you stated a definition of a KPI in past does not mean everybody will remember it and know how tit was calculated. It is your job to make sure anybody using your metrics/KPI can get to the definition and calculations right away. 

Questions? Comments?

Not personalizing? You are leaving money on the table.

I started writing about personalization ever since I started this blog, back in 2006. I wrote extensively about privacy and how marketers should address it to engage in personalization. Consumers are now more at ease with online purchases, they have moved past initial privacy concerns of online tracking and now expect personalization.  Personalization is no longer optional.

According to an eConsultancy report, 94% of the companies realize that personalization is critical to current and future success.

However many can't move forward with personalization because of many barriers they face in implementing personalization.  The biggest being IT, Technology and Budget. (see below).

Charts from eConsultancy Reports - The Realities of Online Personalization

Many marketers don't realize that personalization does not have to be complex. A recent study be Accenture shows that consumer are likely to buy from a retailer who provides some level of personalization (see below):

As you see in the chart above, simply recognizing the customers by name will get them to buy more. So start there if you are already not doing it, that won't require big IT infrastructure or budgets or data. Don't be bogged down by the hype created by press releases or marketing presentation about the sophisticated personalization few companies are doing. Many of the personalization techniques don't require big budgets or IT infrastructures.

Start simple, understand the value personalization brings, show case the value to your internal stakeholders and make a case for getting more funding for more sophisticated personalization tools and technologies. I will be writing more about personalization so subscribe to my blog if you are interested. If you need help feel free to reach out to me. I would also love to hear from those who are currently involved with personalization or have a good story to share.

Article referenced in this post

• Behavioral Targeting 101
• Web's New Gold Mine Personalization

Personalization is no longer optional

I started writing about personalization ever since I started this blog, back in 2006. I wrote extensively about privacy and how marketers should address it to engage in personalization. Consumers are now more at ease with online purchases, they have moved past initial privacy concerns of online tracking and now expect personalization.  Personalization is no longer optional.

According to an eConsultancy report, 94% of the companies realize that personalization is critical to current and future success.

However many can't move forward with personalization because of many barriers they face in implementing personalization.  The biggest being IT, Technology and Budget. (see below).

Charts from eConsultancy Reports - The Realities of Online Personalization

Many marketers don't realize that personalization does not have to be complex. A recent study be Accenture shows that consumer are likely to buy from a retailer who provides some level of personalization (see below):

As you see in the chart above, simply recognizing the customers by name will get them to buy more. So start there if you are already not doing it, that won't require big IT infrastructure or budgets or data. Don't be bogged down by the hype created by press releases or marketing presentation about the sophisticated personalization few companies are doing. Many of the personalization techniques don't require big budgets or IT infrastructures.

Start simple, understand the value personalization brings, show case the value to your internal stakeholders and make a case for getting more funding for more sophisticated personalization tools and technologies. I will be writing more about personalization so subscribe to my blog if you are interested. If you need help feel free to reach out to me. I would also love to hear from those who are currently involved with personalization or have a good story to share.

Article referenced in this post

• Behavioral Targeting 101
• Web's New Gold Mine Personalization

Bootcamp and Courses for Digital Analysts

I have been working in Digital Marketing and Analytics fields for more than 10 years now. Started my career in the field by playing with HTML, developing website and looking at server log files to parse the data to calculate metrics such as sessions, users, page views and many other metrics and KPIs that you see in your tools like Google Analytics, Adobe Analytics etc.

All through my life I have been passionate about teaching.  As a kid I used to teach other kids as a tutor and then I taught as a teaching assistance in graduate school.  For past 10 years or so I have been involved in speaking, teaching and tutoring in Digital Marketing and Analytics at schools, colleges and events. Some of the places I have taught and spoken at are, University of British Columbia, University of Washington, Bellevue College and eMetrics.

Recently I have started developing courses that can be delivered on-demand online as well as immersive bootcamps to develop new Digital Analysts.

The purpose of this blog post is to connect with individuals and organization (for profit or non-profit) who are interested in helping shape these programs.  Specifically, I am looking for:

1. Input on what course should be developed? I have a long list of courses but would like to see which ones should be developed first, where are the gaps in what current programs teach. The idea is to augment what current university programs teach and not replace them.
2. Find likeminded folks who have a passion for teaching and training.
3. Connect with organization (for profit and non-profit) that are interested in partnering with me in providing real business problems and data for students to work on.
4. Connect with schools and educational institutions that want to enhance their Digital Marketing and Analytics programs.
5. Connect with organizations that are willing to provide internships and job opportunities for the students.

So if you are interested in helping develop new Digital Analysts then please contact me at batraonline at gmail.

Web Analytics Under the Hood

Do you understand the mechanism of Web Analytics? Do you understand how data is collected and translated into the nice reports that you see in Google Analytics, WebTrends, Adobe etc.? Now you can.

Signup for my online course - "Web Analytics Under the Hood".

This Online course will go under the hood and show you how the data gets generated, collected and processed to generate beautiful reports that you

see in your Web Analytics tools.

Having a good understanding of what happens behind the scene will provide you the confidence you need to support your understanding of the reports and provide a fresh new perspective on Web Analytics reports.

This course will be helpful for both newcomers as well as seasoned professionals. I will cover topics that I ask in interviews while hiring a web analyst. This course will help you:

• Understand how browser/server communication happens.
• Understand how data gets passed to server.
• Understand how data is collected ( Javascript, server logs) - Basics of Data Collection Javascript using Google Analytics as example.
• Understand how cookies are used (we will look at Google Analytics cookies)
• Understand how data is stored in the back-end.
• Understand how data is processed.
• Understand how data gets Converted into Visits, Visitors, Page Views, Referrer and various other reports.

Signup below to be notified when the course becomes available in Mid-July.  You can also pre-order this course for $50 (instead of $100). You will get a link for payment after you signup.

Signup to be notified of the availability of this course

As many of you already know, I teach few courses in Digital Marketing and Analytics at various universities and colleges including University of Washington and Bellevue College in Seattle area.

Last year I was asked if I can design a course on Data Visualization using Tableau. Since we were actively using Tableau at work I decided to do it.  I have taken that same course, that I taught, and developed an eBook. The audience of this eBook is beginners who want to learn the basic of Tableau and get familiar with various interfaces and terms used in Tableau.  If you are someone who wants to learn Tableau or have looked at Tableau but don’t fully understand it, then this eBook is for you.

I am giving away few copies of the eBook in return for the feedback.  If you are willing to provide me the feedback on this book within a week, then email me at batraonline at gmail.com or tweet me at @anilbatra and I will send you a copy of it.

—————————————————————————————————————————————

Want to learn Tableau but don’t have to provide me the feedback on eBook, here are few Tableau Books on Amazon

——————————————————————————————————————————————-

Subscribe to my mailing list

* indicates required

Email Address *

First Name

Last Name

What does 1st, 2nd and 3rd party data mean?

1st, 2nd and 3rd Party Data Demystified

I have referred to 1st party and 3rd party data in a lot of blog posts. Based on the queries I get, both via email and in the classes I teach, it is time to clarify what various data sources mean.

1st Party Data

1st party data is the data that you (brand/publisher/retailer) have collected about your visitors, customers, shoppers etc. You own the data outright and all the rights to it. You can use it for any purpose you want based on the agreements with your visitors, customers, shoppers etc. as specified in your data collection and use policies. Some examples of 1st party data are:

• Site registration data – name, email, address, gender etc.
• Visitors behavior data on your site – time of visits, minutes spent, products looked at, source of visits etc.
• Shoppers/customers purchase data – products purchased, transaction amounts, coupons used etc.
• Email data – emails sent, opened, clicked etc.
• It is most widely used data for the marketing purposes. Generally, you use the 1st party data for customer retention using email marketing, retargeting and onsite personalization.

2nd Party Data

2nd party data is the data that is collected by some other company and shared with you(brand/publisher/retailer), in other words it is their first party data. A strategic data sharing partnership between two brands/publishers can help both of them grow their customer base and monetize that customer base.
You can generally use 2nd party data to:

1. Augment the data you already have about your customers (or visitors) – for example, if you do not collect “Household Income” during customer registration/signup data but have a need for that data you can partner with another brand that collects that data to get that data to enhance user profile. Another example is Google Adwords sending the keyword/campaign data to enhance behavioral data collected on the site.
2. Add a list of new customers – for example, if you are hotel booking site, you can have a partnership with airline to share information about customers who recently booked. If a customer books a flight, then you can use the data from partner to reach those customers and offer them hotels. Similarly, the airline partner can reach the customers who have booked hotel on your site.

3rd Party Data

3rd party data is the data collected and aggregated by someone other than the 1st party (data collector). In other words, the data aggregator doesn’t directly collect the data from customers/shoppers/visitors but have relationships with several companies/sources that collect the 1st party data. Some examples of the 3rd party data provider are BlueKai, Acxiom and i-behavior. These data providers aggregate the data from different sources to build a comprehensive profile of a customer/person. These enhanced profile let you understand a visitor/shopper more than what a 1st party or 2nd party data sets can provide.

For example, if you are a Financial institution, it will be very helpful for you to know which of your customers travel frequently, this will help you offer them a credit card that provides added travel rewards and benefits. This is where 3rd party data becomes useful that can provide such information based on data collected from various data sources such as hotel booking sites, airlines, location based data on several other places, other credit card providers etc

-------------------------------------------------------------------------------------------------------------------------------------------

Subscribe to my mailing list

* indicates required

Email Address *

First Name

Last Name