Giving us what we want: Why the long tail is really growing and how it could break our society
To start, I haven't yet read Chris Anderson's The Long Tail, but I have seen him speak on the subject. So to some degree this is informed by Mr. Anderson's views, but if this feels like a chapter from the book, let me apologize in advance and claim independent invention.
As many of you probably are already aware, the 'long tail' refers to a section of the power law distribution which comes up in countless aspects of our world — very commonly specifically referring to media distribution, wherein a 'chosen few' make up a disproportionately large share of sales. The long tail is the huge number of items which each have a small number of sales. More specifically, the concept points to the fact that as modes of distribution change for largely technological reasons, the hits (think Britney and Star Wars) are becoming less important and indie pieces and cult classics out on the tail of the curve are selling more and becoming more relevant.
If you don't believe this, go see Mr. Anderson speak, or I would presume that you could just read his book as well. You could also find a quick introduction to the topic on Wikipedia. I think it's pretty clear that things are changing, and that most arguments on the topic will take place over the degree of change and its implications, not the presence thereof. Remember Tower Records?
Isn't this great?!?
Rosy-eyed and inspired by the promise of a new world of our own creation, in the beginning I saw only the upsides to this trend. Isn't it fantastic, I thought, now I can finally escape those lousy radio singles and hollow Hollywood action flicks and find media with real substance, something that really speaks to me. And if I can't find it, I can always just roll my own.
No really, isn't this great?!?
I still largely do think it's great actually, particularly the user-generated aspects, but I'm starting to see a big potential cultural downside. As we have more choices across the board, that means a denser distribution along almost any axis of view: more hardcore punk, more Gelugpa chanting, more documentaries about peanut farming, you name it — just more.
Again, isn't this great? Well, at first glance it is, at least through the idealist's lens that would tell us that given all of this wide and varied information that we will graze across it, gobbling up wide and varied cross-section of opinions, knowledge, and inspiration.
But is that really what we will do?
When I listen, I only hear myself
Signs point to no. In online communities, I can't see a lot of evidence that Air America fans are drifting over to Fox News or Ann Coulter for a little balance. They might, however, nominate their favorite liberal blog for an award. Or vice versa.
When we're given an all we can eat information buffet, it seems that we tend to just stuff ourselves on the same old meat and potatoes we're used to, while ignoring that wide diversity that brought us to the table in the first place. orgnet.com has an interesting piece called Political Books and Polarized Readers that analyzes the 'also bought' data from Amazon to show this effect in sharp relief.
But that's all I care to hear
And so then, instead of just measuring our increasing engagement in a broadening scope of opportunities, the growth in the long tail is actually fueled in large part by a narrowing of individual focus. When we read, hear, or watch something we like or agree with, we can now hunt down more of the same, almost effortlessly. And few, if any, of us can resist the temptation of being told over and over again that we are absolutely and completely right.
So the tail grows and grows, as we snatch up long lost import singles and director's cuts and books that express the same opinions as that last book we liked so much. And we are happy, but perhaps not fulfilled.
Now it's all I can find at all
Throw in an effective recommendation system of the future and now you've really got a problem, not because it won't work, but because it will. Given an infinitely long tail, you can find an infinite number of works that align with any narrow point of view (exaggeration to be sure, but within the scope of our media consumption capabilities, not excessively so). And how to browse an infinite catalog but through an innovative recommendation system? But then given that perfect system it will know that since I loved that Bill O'Reilly book so much that I must want nothing more than books by a selection of Bill O'Reilly clones. And I probably do — or at least I'll gobble them up happily if that's all I see.
But at least I know I'm right
With all of this confirmation of our viewpoints, what do we get but a polarized world where each side shares little but an adherence to our opinions that borders upon the religious? And think not of a two party system of disagreement, but of a hectagon where each side, though small, can be just as polarized and isolated from the rest.
The big three networks aren't enough either
At least at some ranges of scale, the value of media as a whole grows with the number of options presented. Television is worth much more with two channels with one, and more still with ten channels or fifty. I believe that in an ideal world this trend has the chance to continue onward to infinity. It's up to us as consumers and especially as technologists to attempt to continue to create and extract this added value.
There's a world of information out there — use it. Not just to read a rehash of that same blog post you just read six times in theme and variations, but instead to truly expand your horizons. We all would do well to expose ourselves to the other side from time to time. In the worst case, we are better informed and prepared to discuss or argue for our side, and in the best case we might learn something truly profound that shifts our viewpoint entirely.
On the technologists' side, Wikipedia comes to mind (as it so often does) as a good example of a structure that can encourage this kind of growth. Even ignoring the fact that it is user-generated, simply through diverse content and dense hyperlinking, I find it almost impossible to read about just one topic on any given visit, and often these journeys lead to surprisingly diverse content even after only a few links. Recommendation systems will need to be designed with these thoughts in mind and encourage us to learn, not just buy the same old comfortable materials and tired entertainment, while keeping enough comfort and familiarity to maintain market share and thus relevance. How about a reco system with a 'how crazy are you feeling today?' slider that lets you fine tune the amount of diversity that suits your current mood?
This is the end - this is only the beginning
I do believe that the exploding mass of content available online and otherwise is an inherently good thing at its core, but we need ways to manage it efficiently and most importantly we need to be of the mind to use it responsibly and effectively. Mental laziness disguised as a voracious appetite for learning (the same thing over and over again) is nothing strange in this new world, so let us not become victims of this masquerader.
Sorting by popularity so that the favorites of the group float to the top achieves little more than what the 'old media' has been doing for decades: letting the majority decide for the diverse minorities. Conversely, as collaborative filtering systems improve, there will be a point where increased absolute filtering performance will only serve to amplify the echo, as our individual past is projected forward to become our future, preventing us from growing and expanding mentally and philosophically. Between and along the edges of these regions lies a land of great need and opportunity.