The Great comScore Debate

comScore came out with a study saying cookie based system could be reporting 2.5 time more visitors.

I wrote a post with my view on the topic last week. My take was that we should not worry about this study for two main reasons
1. We don’t know the comScore methodology of this study
2. We know that two web analytics systems don’t report the same numbers so we were dealing with estimated numbers anyway. This is the reality of Web Analytics. We should make decision based on trends more than raw numbers.

Gary Angel from SEMphoice wrote a blog post where he disagrees with me and says he is worried about the accuracy of the data after this study.

Side note: Gary took my view and made them appear them as ZAAZ’s (my employer)view. One thing I want to make clear is that these views are mine only and do not represent the views of ZAAZ or any other employee of ZAAZ.

Back to the issue: Well, even before this study we all knew cookies were being deleted but we accepted this fact till comScore study came. We should have been worried all along but we were not. Why? I guess we had to work with whatever we had.

The point I was trying to make is that you have to take everything in context. Going to Gary’s example of a conference, let’s say conference A tell you they attract 5,000 visitors and the other conference B says they get 4,000 visitors. Next day a third party comes out and says that all the conferences numbers reported by any conference are inflated and actual number is 75% of what they state then what’s the net result? Well Conference A is still better than conference B. Only thing is that they each now have 3750 and 3000 visitors respectively. Every conference in the world will have the same issue, their rank is still the same. I don’t think based on this information conferences will start charging less for the booth. However the rate per visitor has gone up for you but you can’t do much, that’s the market rate. Same argument goes for sites that sell advertising based on how many users they reach.

I recently moved a customer from one web analytics tool to another, guess what? The unique visitors count was different (so was repeat visitor and visit count). They were using old system for 3 years, all their decisions were based on those numbers. We could have spent days to figure out which one was correct old or new. Was it worth it? No, I don’t think that would have made any difference to their business. We could have waited to get the perfect data but that was not going to happen no matter what. So what did we do? We verified the new system to make sure it is accurately configured. We have tagged all the pages, proper excludes and includes are there, Cookie is first party etc. and the accepted it, developed our baseline and started working from there.

I strongly believe in accuracy of data but accuracy is also within context. The system you are using has to be perfectly configured, has to be accurately measuring but accuracy is defined based on how that system works.

I am pretty sure that if you take logs from your server and process them with an old version of any tool of you choice and then with the latest versions you will get different numbers (DisclosureI used to work for a tool vendor). Which one is correct? Old or new? If new is correct than all your old decisions based on wrong numbers were wrong. How do I know for sure that new is good? All I know my trends were similar in both new and old system.

Now lets say based on cookies counting, one site reaches 2 times visitors than their competitors (assume both use same 3rd party cookie). Will this site be more or less valuable if both the sites were told they were over counting their visitors? They would still be the same. How they will compete is how much more visitors (or visits) they get. Panels have a potential to be faultier than cookies if panels are not true representation of the site’s visitors. Panel based counting is just like log file sampling, sampling at least is a true representation of your visitors but we all know that those numbers are not accurate.

I understand Gary’s issue about repeat users and new users. But again, if you use two different systems they will report different numbers so which one is correct?
As Jacques Warren pointed out as a response to Gary’s post, the right solution (at this time) is to provide a reason for users to not delete their cookie (or give a reason to login). If Gary care’s about repeat users then I am sure he has strategies to get them engaged and give them a reason to login (or not delete cookies). Give users a reason to be loyal and they will be. Then you won’t have to worry about cookie deletion and hence your numbers will be accurate. Till you get to that level any number is a close estimate weather it is panel based or cookie based; and is not worth loosing sleep over.

I am not going to be worried till I have concrete proof that 1st party cookie system is completely flawed. Right now I don’t have any basis to be worried. IAB has asked comScore and NNR to show their measurement process, let’s see what comes out of that

Cookie versus Panel Based User Counting

comScore released the results of a study they did which claimed that cookie based measurement overstate Unique Users by as much as 150%.

I was not surprised by the results, not because I knew that cookies was over estimating because this is what I would have expected based on who was sponsoring the study. Since the study was conducted by comScore I wouldn't have expected them to come up saying panel data is worse than cookies based counting. If this study were done by Omniture or WebTrends we would have probably heard a different story. For example, if we were to compare wine and coffee and see which one is bad for health, wine company’s research will say wine is good for health while coffee is not. Coffee company will come with their research which will say coffee is good and wine is bad.

It is very likely that the users who participated in the study knew that they were being tracked (and they participated to get freebies) so they developed a tendency to clear cookies and hence skewed the results. There could be several reasons why comScore survey might be correct or not but comScore did not publish those so we can’t say for sure if panel based is really better than cookie based count.

It is likely that comScore is trying to solidify its position as a standard in audience calculation by releasing this study. In my article on Google and Behavioral Targeting I mentioned that Google is putting it’s cookie everywhere and could potentially get in the business that comScore is in currently (measuring audience size). This possibly could be a preemptive move by comScore. Don’t know for sure since I don’t have the details on how this study was done.

The crux of the matter is that no matter what system you use you will never get an accurate count of UU's. Even pages views are reported differently by different systems. Two panel based systems don't report the same numbers and two web analytics tool do not report the same number. To assume one system is not accurate or is better than other system is jumping to conclusions without basis. Both systems have their own advantages and disadvantages.

For individual site owners you need to pick one system, cookie based or panel based, and stick with it and accept the numbers. How does it matter if I told you that you actually get 3,000 users (or visits) and not 5000? What will change? What will you do differently? Have you ever moved from one web analytics system to another? You know they don't report same numbers. Say, your new system reports lower number than your old system than what are you going to do? Which one is correct, old or new? It is about trending and growth. You will probably use same growth and retention strategies no matter what the number is. If you goal is to increase the visitors by X% no matter what visitor count you use you will try to do X% of your current system weather that is cookie based or panel based.

If you are trying to compare sites then do the same thing, choose one system for comparison and stick with it. The same goes with advertising, if you are buying or selling ad inventory based on Unique Visitor count both buyer and seller will have to settle for one system and stick to stick to it.

However, one thing that surprised me in this study was a quote from Tacoda. Their whole business (Behavioral Targeting) is dependent on cookies and now they will validate with panel data because....(...they don't believe the size of their segment? Does that mean they will adjust the reach they claim they have?) Can someone from Tacoda please help me understand it?