March 26, 2010

The (un)importance of usage stats

Many of my friends and coworkers place a great deal of importance on web usage stats / web analytics. While the data can be useful, it has always been my opinion that the numbers are very easy to misinterpret. Usage reports provide perfectly accurate data that is very, very difficult to interpret, and should not be used by themselves to make decisions about web sites.

Value: do more hits = bigger value?

Not necessarily.

The Dilbert cartoon web page gets far more hits per day than all of my websites combined get in a year, but the world would be just fine if the Dilbert site went away. Likewise, maybe you have a page that only gets a few hits per month, but the information on it is used by someone to make an important decision.

When I look for something, I often go through several pages to get there, because of:
  • wrong guesses by me when navigating a site
  • search results that ended up being non-relevant 
  • simply clicking through a site to get to what I need (clicking through *to* something causes hits to the site and to pages that are on the way there)
 More hits on a site = users like the site and found it useful?

If I don't give up easily (and I don't, but some people do) I may click through dozens of your pages to find what I'm looking for, even if I never find it and leave unhappy

Page A gets no hits. Why?
  • It's useless? probably not, because people don't know that until they hit it. At most it will stop return visits by the same person.
  • It's hard to find? If it's misclassified, poorly linked, not in the menu, etc., is this the fault of the page, or the site?
  • Its content is not of much use? Maybe to most people, but what if one person visits it and uses the info on it to make a million dollar decision?
Page B gets lots of hits. Does this mean it's a good page?

A page hit does not mean the user liked the page (in the same way that going to a movie doesn't mean you liked the movie). Advertising and previews/trailers get you to movies.

Even repeat visits can't be interpreted as a satisfied user: I have contributed many repeat page hits at microsoft.com and other sites trying to find what I'm looking for, and going back to pages I've already seen because what I need must be there even though I didn't find it the first time; or I was lead back there by a search engine after clicking around the site failed to find it.

My site is getting less or more hits than a year ago, so my site is worth less or more than it was a year ago.

  • Has your site changed? (for example, was your site trimmed down by x%?)
  • Has your user population changed? (less or more of them than a year ago?)
  • Has your site become more well-known, and people are using favorites, shortcuts, links in email and documents, or keywords to get right to what they need instead of browsing in through the front door?
  • Did you move/rename something, making it harder to find (especially if you changed the path and broke the aforementioned Favorites/shortcuts/links)? This is one of the things I think web developers don't pay enough attention to: leaving things where they are for sake of those that *already* know where they are; if someone rearranges your grocery store to be more logical, or to fit some bigwig's whim, you're not being very nice to those that already know the layout of the store.
  • Did some news announcement cause a peak in user visits? Isn't that more a testament to the power of the media? (Don't movies do well at the box office because of the ad money poured into advertising them? Lots of bad movies have big numbers; seeing a movie doesn't mean the viewer liked it, in the same way that a web page hit doesn't mean that the user found the page useful).
So there's two major themes concerning why it is difficult to tell *anything* with usage stats:

  1. a user visit or page hit does not mean they are happy with what they found
  2. you can't ascribe value to a hit, because you don't know how valuable the found information is to the user
So what *can* you use stats for?

Well, they can be used in concert with
  • survey data (which itself is not especially known for being reliable, and which is more unreliable the longer the survey is); and
  • user testing (watching someone trying to perform a task); this is what the pros do, but it's every bit as time-consuming and expensive as you'd imagine it would be
I've found some of the following reports to be somewhat useful (but again, misinterpretation is just as easy to do here):
  • single-page visits: which pages do people come to via search result/link/favorite, and then leave?
  • entry pages: what pages do people start at?
  • exit pages: what is the last page people see before they leave?
  • most visited pages
But usually, once I start thinking of ways I might be misinterpreting the data, I end up taking nothing from it.