On bar graphs, buying guides and avoiding the tyranny of choice

Ah, the curse of the internet! Once upon a time, we would be satisfied just to get an okay taco in NYC. Now, unless you get the VERY best anything as rated by the internet, you’re stuck feeling like this:

Same goes for everything from chef’s knives to backpacks to whatever it is (I recommend The Sweethome as an excellent site with buying guides for tons of products). Funnily enough, I think we have ended up with this problem for the same reason that people whine on about bar graphs: because we fail to show the data points underlying the summary statistic. Take a look at these examples from this paper:

For most buying guides, they usually just report the max (rather than the mean in most scientific bar graphs), but the problem is the same. The max is most useful when your distribution looks like this:

However, reporting the max is far less useful a statistic when your distribution looks like this or this:

What I mean by all this is that when we read an online shopping guide, we assume that their top pick is WAY better than all the other options—a classic case of the outlier distribution I showed first. (This is why we feel like assholes for getting the second best anything.) But for many things, the best scoring item is not all that much better than the second best. Or maybe even the third best. Like this morning, when I was thinking of getting a toilet brush and instinctively went to look up a review. Perhaps there are some toilet brushes are better than others. Maybe there are some with a fatal flaw that means you really shouldn’t buy them. But I’m guessing that most toilet brushes basically are just fine. Of course, that doesn’t prevent The Sweethome providing me a guide for the best toilet brush: great, deeply appreciative. But if I just go to the local store and get a toilet brush, I’m probably not all that far off. Which is to say that the distribution of “scores” for the toilet brush are probably closely packed and not particularly differentiated—there is no outlier toilet brush.

While there may be cases where there is truly a clear outlier (like the early days of the iPod or Google (remember AltaVista?)), I venture to say that the distribution of goodness most of the time is probably bimodal. Some products are good and roughly equivalent, some are duds. Often the duds will have some particular characteristic to avoid, like when The Sweethome says this about toilet brushes:

We were quick to dismiss toilet brushes whose holders were entirely closed, or had no holders at all. In the latter category, that meant eliminating the swab-style Fuller brush, a $3 mop, and a very cheap wire-ring brush.

I think this sort of information should be at the top of the page, and so you buying guide could say “Pretty much all decent toilet brushes are similar, but be sure to get one with an open holder. And spend around $5-10.”

Then again, when you read these guides, it often seems that there’s no other rational option than their top choice, portraying it as being by far and away the best based on their extensive testing. But that’s mostly because they’ve just spend like 79 hours with toilet brushes and are probably magnifying subtle distinctions invisible to the majority of people, and have already long since discarded all the duds. It’s like they did this:

Now this is not to say those smaller distinctions don’t matter, and by all means get the best one, but let’s not kill ourselves trying to get the very best everything. After all, do those differences really matter for the few hours you’re likely to spend with a toilet brush over your entire lifetime? (And how valuable was the time you spent on the decision itself?)

All of this reminds me of a trip I took to New York City to hang out with my brother a few months back. New York is the world capital of “Oh, don't bother with these, I know the best place to get toilet brushes”, and my brother is no exception. Which is actually pretty awesome—we had a great time checking out some amazing eats across town. But then, at the end, I saw a Haagen Dazs and was like "Oh, let's get a coffee milkshake!". My brother said "Oh, no, I know this incredible milkshake place, we should go there." To which I said, "You ever had a coffee milkshake from Haagen Dazs? It's actually pretty damn good." And good it was.