The Great comScore Debate

comScore came out with a study saying cookie based system could be reporting 2.5 time more visitors.

I wrote a post with my view on the topic last week. My take was that we should not worry about this study for two main reasons
1. We don’t know the comScore methodology of this study
2. We know that two web analytics systems don’t report the same numbers so we were dealing with estimated numbers anyway. This is the reality of Web Analytics. We should make decision based on trends more than raw numbers.

Gary Angel from SEMphoice wrote a blog post where he disagrees with me and says he is worried about the accuracy of the data after this study.

Side note: Gary took my view and made them appear them as ZAAZ’s (my employer)view. One thing I want to make clear is that these views are mine only and do not represent the views of ZAAZ or any other employee of ZAAZ.

Back to the issue: Well, even before this study we all knew cookies were being deleted but we accepted this fact till comScore study came. We should have been worried all along but we were not. Why? I guess we had to work with whatever we had.

The point I was trying to make is that you have to take everything in context. Going to Gary’s example of a conference, let’s say conference A tell you they attract 5,000 visitors and the other conference B says they get 4,000 visitors. Next day a third party comes out and says that all the conferences numbers reported by any conference are inflated and actual number is 75% of what they state then what’s the net result? Well Conference A is still better than conference B. Only thing is that they each now have 3750 and 3000 visitors respectively. Every conference in the world will have the same issue, their rank is still the same. I don’t think based on this information conferences will start charging less for the booth. However the rate per visitor has gone up for you but you can’t do much, that’s the market rate. Same argument goes for sites that sell advertising based on how many users they reach.

I recently moved a customer from one web analytics tool to another, guess what? The unique visitors count was different (so was repeat visitor and visit count). They were using old system for 3 years, all their decisions were based on those numbers. We could have spent days to figure out which one was correct old or new. Was it worth it? No, I don’t think that would have made any difference to their business. We could have waited to get the perfect data but that was not going to happen no matter what. So what did we do? We verified the new system to make sure it is accurately configured. We have tagged all the pages, proper excludes and includes are there, Cookie is first party etc. and the accepted it, developed our baseline and started working from there.

I strongly believe in accuracy of data but accuracy is also within context. The system you are using has to be perfectly configured, has to be accurately measuring but accuracy is defined based on how that system works.

I am pretty sure that if you take logs from your server and process them with an old version of any tool of you choice and then with the latest versions you will get different numbers (DisclosureI used to work for a tool vendor). Which one is correct? Old or new? If new is correct than all your old decisions based on wrong numbers were wrong. How do I know for sure that new is good? All I know my trends were similar in both new and old system.

Now lets say based on cookies counting, one site reaches 2 times visitors than their competitors (assume both use same 3rd party cookie). Will this site be more or less valuable if both the sites were told they were over counting their visitors? They would still be the same. How they will compete is how much more visitors (or visits) they get. Panels have a potential to be faultier than cookies if panels are not true representation of the site’s visitors. Panel based counting is just like log file sampling, sampling at least is a true representation of your visitors but we all know that those numbers are not accurate.

I understand Gary’s issue about repeat users and new users. But again, if you use two different systems they will report different numbers so which one is correct?
As Jacques Warren pointed out as a response to Gary’s post, the right solution (at this time) is to provide a reason for users to not delete their cookie (or give a reason to login). If Gary care’s about repeat users then I am sure he has strategies to get them engaged and give them a reason to login (or not delete cookies). Give users a reason to be loyal and they will be. Then you won’t have to worry about cookie deletion and hence your numbers will be accurate. Till you get to that level any number is a close estimate weather it is panel based or cookie based; and is not worth loosing sleep over.

I am not going to be worried till I have concrete proof that 1st party cookie system is completely flawed. Right now I don’t have any basis to be worried. IAB has asked comScore and NNR to show their measurement process, let’s see what comes out of that