With the annual publication of the AmLaw 100 and Second Hundred, it’s rankings season for BigLaw.

Think of it as our little industry’s equivalent of earnings season for public companies—we having become a “quasi public” industry, as one of my favorite managing partners has observed. The difference is that we report our financials annually as opposed to quarterly—as well as the distasteful detail that the financials we share with the press are, virtually without exception, not only unaudited per GAAP but not even perfunctorily attested to for veracity.

Across the pond we have comparable numerical tabulations to the AmLaw 100, such as The Lawyer‘s UK 200, Hot 100, and International Top 30. Here we also have the American Lawyer‘s own grab bag of elements making up their “A List,” and even the venerable category “Magic Circle” is a list of sorts, as well as a journalistic jackpot of a coinage if ever there were one.

But it’s high time, and past that, to ask whether the epistemological underpinnings of these kinds of broadly encompassing rankings can withstand analytic scrutiny. Because they can’t.

Even the narrowest, by far, of the lists we’ve enumerated—the Magic Circle—may be losing whatever fencing-in/fencing-out utility it once had. A lead story in the current Lawyer‘s comprehensive coverage of the Magic Circle as of 2015 is headlined, “Together but not the same,” and begins “The magic circle is often seen as a homogenous group of firms, but our analysis of their global investments throws up considerable differences.”

Simon Davies, head of Linklaters, has already made up his mind about the impact of market solvents on the category’s integrity as a substrate:

‘The magic circle isn’t what it used to be in the context of the US market or any market. I don’t think it’s going to gain power or momentum and I don’t think it’s a strengthening brand in itself,’ he says. ‘At the end of the day we see ourselves as one of the leading global law firms, not one of the leading magic circle firms.’

But what about the Granddaddy ranking scorecard, the #1 list in mindshare by far in the largest law firm market in the world—the AmLaw? Well, AmLaw itself has spent the last few years increasing the proportion of space devoted in its annual coverage to the theme that the 100 consist of at least three sharply distinct species—the “Super Rich” top 10% or so, vereins, and everyone else. (The Super Rich are somewhat arbitrarily but not irrationally defined as firms with RPL over $1.0-million and PPP over $2.0-million.)

It’s patently obvious why vereins deserve their own non-intersecting circle on the Venn diagram representation of the AmLaw, but what about the Super Rich? Aren’t they just law firms like all of us, only with the right mix of elite clients, practice areas, and geographic footprint?

Well, no.

Industry participants and observers from AmLaw itself to Citigroup Private Bank, ThomsonReuters Peer Monitor, managing partners of some of the firms themselves (anonymously and off the record), and yes, your faithful friends at Adam Smith, Esq., have all identified the phenomenon of the Super Rich’s pulling away from the rest, using such phrases as “accelerating segmentation,” an “unbridgeable” and “insurmountable” gap, and saying (AmLaw, in its May 2015 cover story on the 100), “the widening gulf in profits is even more extreme [this year than in the past].”

This makes one ask, “Why?” What are they doing that everybody else is not doing? What secret are they privy to that the rest of us are not?

My answer is simple; they’re following a different business model than everyone else; in a real sense, they are in a different business than everyone else. Unpacking all the elements of how exactly they’re different is a topic for another day, but here are some elements:

• Everyone claims to aspire to the high end, price-insensitive, bet the company work. The Super Rich are there and everyone else is, by and large, not.
• Everyone but everyone (I’d hazard 98 of the 100, and let me get back to you on who the two are) claims that their firm’s culture is “collegial” and “collaborative,” and that’s why they’re so special. The Super Rich usually are internally collaborative, but they also put a more weighty premium on high performance and a culture of excellence. If “collegial” to you means providing room for professionals spanning a wide range of competence and commitment, the Super Rich beg to differ. These are not cultures of optionality.
• And finally, yes, many were blessed to have started out in intrinsically dense and rich legal markets, particularly New York City, but don’t underestimate their exceptionalism, in not only surviving but rising to occupy the top sliver of the world’s most competitive markets. Far more firms have broken their lance trying to penetrate New York than there are Super Rich firms in total.

But the reality reveals a more profound ontological flaw at the root of lists like the AmLaw 100. (“Let me tell you about the very rich. They are different from you and me” [apologies, F. Scott].)

Nearly five years ago, Malcolm Gladwell published The Order of Things: What college rankings really tell us in The New Yorker, performing the high public service of subjecting the US News rankings of “Best Colleges” (and “Best” graduate programs, business schools, medical schools, and of course law schools) to a lacerating onslaught of attacks on its methodologicy and substance that reduced the “Best” lists to the intellectual stature of meow mix.

Before unleashing his broadside on US News, he uses the example of Car and Driver magazine to demonstrate the category error:

Car and Driver’s ambition to grade every car in the world according to the same methodology would be fine if it limited itself to a single dimension. A heterogeneous ranking system works if it focusses just on, say, how much fun a car is to drive, or how good-looking it is, or how beautifully it handles. The magazine’s ambition to create a comprehensive ranking system—one that considered cars along twenty-one variables, each weighted according to a secret sauce cooked up by the editors—would also be fine, as long as the cars being compared were truly similar. […]

A ranking can be heterogeneous, in other words, as long as it doesn’t try to be too comprehensive. And it can be comprehensive as long as it doesn’t try to measure things that are heterogeneous. But it’s an act of real audacity when a ranking system tries to be comprehensive and heterogeneous.

Now I realize that the AmLaw 100 is not a “Best” (judgment-freighted) list, merely an arithmetic ranking by gross annual revenues, which seems innocent enough. But as the AmLaw’s own admitted discomfort over the awkwardness of including vereins, not unitary integrated firms, in the list indicates, the implication of the list’s very existence is that all members are fundamentally similar in the ways that really matter. This is beginning to fray around the edges.

My argument is more radical than positing the vereins don’t belong on the list, although I would exclude them were I King. Where I might come down on vereins “in or out?” is that it’s of small moment since everyone knows they’re not comparable beasts.

Rather, my argument is that we deceive ourselves about how truly heterogeneous the business models within our industry are becoming by unconsciously assuming that the AmLaw 100—or the AmLaw Second Hundred, or the AmLaw 200 overall—constitutes a list of the members of one fundamentally indivisible species.

Shall we step back from Law Land for a moment to gain (one can hope) a wider perspective?

The Fortune 500 is another Granddaddy of lists, and it’s conceptually a near twin of the AmLaw 200, ranking US-based corporations by total annual revenue. Here are the top ten firms as of the 2014 list:

1. Wal-Mart
2. Exxon Mobil
3. Chevron
4. Berkshire Hathaway
5. Apple
6. Phillips 66
7. GM
8. Ford
9. GE
10. Valero Energy

I submit this tells you vastly more about the overall composition of our macroeconomy than it does about the business models, strategic acuity, or operational excellence of the companies on the list. It tells you (for example) that the energy and automotive sectors are big, as is mass-market retailing and high tech.

I suspect you are less than shocked at that late-breaking news.

Similarly, the AmLaw 100 would tell you that large firms with a global footprint tend to have more revenue than regional or more focussed firms—this is already sounding tautological, isn’t it? It does not provide a clue as to whether the Super Rich strategy trumps the “everywhere all the time” strategy or how that compares in turn to the “we know our niche and we’re stickin’ to it” or the “deer in the headlights/wake me when it’s over” strategy—any more than the Wal-Mart/Exxon gold/silver finish tells you whether Wal-Mart would be a better massively integrated petroleum producer than Exxon or Exxon would be a better massively integrated retailer than Wal-Mart. To pose the hypothetical is to expose its transparent absurdity.

In other words, the AmLaw 100 can provide us a one-dimensional data series, period: An irresistibly fascinating one to pore over, to be sure, but ultimately uninformative about the firms’ relative performance as against their chosen strategy. Different strategies will drive firms to intrinsically different band positions.

Consider Gladwell’s example of two of the measures US News incorporates in its college quality rankings: One is “student selectivity,” and another is “graduation rate performance.” Selectivity is self-explanatory; in the most recently published statistics, Stanford was #1, accepting only 5.7% of applicants. (For those of you keeping score at home, Harvard was #2 at 5.8%, Columbia and Yale tied for #3 at 6.9%, and Princeton came in #5 at 7.4%.)

“Graduation rate performance” measures how a college’s actual graduation rate compares with the predicted rate given socioeconomic status and test scores of its incoming freshman class. Think of it as a measure of the school’s effectiveness. Penn State, for example (appealing to students from a very wide range of backgrounds with a high “standard deviation” of resources available to them when they were growing up), had an expected graduation rate of 73% but an actual rate of 85%: Admirable. Yale, by contrast, admitting student all-stars, had a projected rate of 96%, meaning the highest its score could possibly be on this measure was +4 (it was actually +2).

Both statistics—selectivity and “efficacy”—are important, but compiling a ranking requires the compiler to choose weights. US News weights selectivity twice as important as efficacy which prompted Graham Spanier, president of Penn State at the time of Gladwell’s article, to say:

“If you look at the top twenty schools every year, forever, they are all wealthy private universities. […] Do you mean that even the most prestigious public universities in the United States, and you can take your pick of what you think they are—Berkeley, U.C.L.A., University of Michigan, University of Wisconsin, Illinois, Penn State, U.N.C.—do you mean to say that not one of those is in the top tier of institutions? It doesn’t really make sense, until you drill down into the rankings, and what do you find? What I find more than anything else is a measure of wealth: institutional wealth, how big is your endowment, what percentage of alumni are donating each year, what are your faculty salaries, how much are you spending per student. Penn State may very well be the most popular university in America—we get a hundred and fifteen thousand applications a year for admission. We serve a lot of people. Nearly a third of them are the first people in their entire family network to come to college. We have seventy-six per cent of our students receiving financial aid. There is no possibility that we could do anything here at this university to get ourselves into the top ten or twenty or thirty—except if some donor gave us billions of dollars.”

Now we’re approaching the nub of the problem.

Lists of law firms which do contain a judgment-freighted component (the AmLaw’s “A List,” a “Top” or “Hot” almost-anything) are forced to determine in advance just exactly what their measures of quality are, and which matter most. The US News rankings notoriously exclude price (tuition) from their school and university rankings. And now that you mention it, I’ve never seen a list of “Top” law firms that includes price as a metric, either.

This brings us back to the Super Rich and “everyone else.” Here are a few firms from each decile of the latest AmLaw 100. Ask yourself what firms in these groupings have in common (hint: not that much):

• 1st decile: Latham, Kirkland, Jones Day
• 2nd decile: Gibson Dunn, Morgan Lewis, Sullivan & Cromwell, Cleary, Greenberg Traurig, Reed Smith.And as we go up the deciles, the differences are even more patently obvious.
• 6th decile: Vinson & Elkins, Cravath, Wilson Sonsini, Bryan Cave
• 10th decile: Boies Schiller, Fox Rothschild, Fenwick, Baker Donelson

I’m not so naive, or idealistic, as to imagine that pointing out flaws in aggregate listings will deprive them of their guilty-pleasure appeal. US News has transformed itself from a Time/Newsweek also-ran to a List Factory, methodological and conceptual flaws notwithstanding. Some publications in Law Land seem to the innocent eye to be tending in the same direction, and without question it’s a strategy with demonstrable marketplace popularity.

All I’m asking is that you not mistake lists or rankings for intellectual analysis.

The Super Rich are playing a different game than everyone else. For them and their clients, price (cost) is asymptotically close to immaterial. The rest of us aren’t so fortunate.

Or maybe we have to make our own fortunes, on a non-intersecting plane of reality in this four-dimensional marketplace we are all playing in. One where price matters.

Either way, it’s high time to stop assuming lists can be comprehensive and heterogeneous.