It is no secret that Amazon is a data driven organization. The culture of testing is ingrained into pretty much everything Amazon does. However, a company like Amazon also makes mistake as I learned from my recent experience with Amazon Videos.
I have two Amazon Fire TV Sticks, one attached to each TV that I have. Both are tied to the same Amazon account. Using my Amazon Video account, I started watching a movie on one TV, turned it off halfway through and then a day later tried to watch it on the other TV. Guess what? I could not find that movie in any of the obvious menu options. I was expecting it to show up in my stream or at least in the same place where I found it last time. Nope. Amazon changed the order of movies on me. It’s not like it added new movies to the lineup and this movie got pushed down. They just reordered the existing movie selection. It appeared to me that some kind of movie display sorting experiments was going on. However, that experiment, ruined the experience for me as I spent a lot of time looking for that particular movie.
In another instance on Amazon Video, I found the movie but as I soon as I clicked on it, it vanished and the movie list was reordered. Weird, right? Again, my suspicion is that some movie sorting/ordering algorithm experiment (A/B test) was going. So, what can you learn from this experience? There are two A/B testing (experimentation) lessons that you can learn from this mistake by Amazon:
- Keep version consistency across devices – Everybody uses multiple devices these days, multiple TVs, Tablets, Phones etc. Make sure you provide same version of your layout/algorithm to the same user across devices. Which means that you must do testing at the customer level instead of session or visitor level.
- Do not change customer experience midstream – Make sure all touch points are in sync and you’re A/B test does not change customer experience as customer interacts with the interface. In this particular case, if the movie was right in front of me on the home screen then pressing the button on the movie should not have triggered a test that rearranged the order of the movies displayed on my screen. The ordering/sorting should have been done before it was presented to me.
Questions? Comments?
Need help with A/B testing and Personalization? Reach me at batraonline@gmail.com.
