Tilt and tilted

Data is not the level ground that we act like it is.

Source: Boston Globe.

Last week, in an midnight order declining to block a Texas law banning most abortions, five justices on Supreme Court of the United States revealed themselves to be what they’ve always been, no matter how hard we’ve tried to pretend otherwise: People, with opinions. 

When we talk about the Supreme Court, we place the institution outside—and often above—the political fray. We’re taught that the judgements of the court come from careful readings of the law, deliberate considerations of facts and precedent, and the country’s sharpest legal minds. Though justices ascend to the bench from prior careers as political appointees and lawyers, campaign managers, and members of activist organizations, we act as though those histories—not to mention their upbringings and religious convictions—are purged the moment they take their lifetime seat on the court.

Sometimes this is explicit; during his confirmation hearing, Chief Justice John Roberts said his job was to be an umpire, calling balls and strikes. But more often, this framing is implicit, a reflexive deference to the court and those who sit on it. Just last week, the New York Times’ David Leonhardt commented that justices usually “solicit briefs, hold oral arguments and spend months grappling with their decision” (emphasis mine). While we may disagree with what the justices decide, we want to believe in the complexity of the decisions they make, the good faith from which they make it, and the legal ground on which they rest. And many people, from civics teachers to op-ed columnists to the justices themselves, are eager to sell this vision. 

Last week’s ruling tears the mask off. 

The five justices in the majority showed that Leonhardt was half right—justices do need time to issue rulings. But that time isn’t for figuring out what to do; it’s for finding a plausible argument for how to do it. In this case, evidently so eager to overturn fifty years of precedent, they didn’t bother.

The Texas law, known as SB8, has two significant elements. The first bans nearly all abortions in the state. The second delegates enforcement of the law to the citizenry. Under SB8, individual citizens can sue for and collect $10,000 from anyone involved in providing an abortion, including doctors, clinicians, people who pay for procedures, and people who drive patients to clinics. The bizarre and dystopian scheme, devised in plain sight, was designed to confuse legal challenges to a plainly unconstitutional law—or at least give a potentially friendly court that wants to ban abortions an excuse to do it.

As Justice Sotomayor said in her dissent, “the gambit worked.” The Supreme Court declined to block the law because it raises “complex and novel antecedent procedural questions.” As should be obvious to everyone, this is a brazen lie. The court is corrupt and corrupted. The question at hand was not complicated. This was not a careful and unbiased reading of the facts. This was not an opinion rooted in laws; it is now a law, rooted in opinions. This was, per Adam Serwer, five justices invalidating a previously upheld constitutional right to abortion “because they wanted to, because they could, and because fuck you.”

Of course, justices are people too, swayed by their own opinions, in ways that may be invisible even to them. Good LSAT scores and long resumes don’t make people immune to personal desires. That justices would rule on SB8 in accordance with their own preferences isn’t a surprise. The surprise is how audaciously they did it.

But, even if they had been more delicate—or attempted their best good-faith reading of the law and prior precedent—the ruling still wouldn’t be impartial.

For our laws are not natural ones. They aren’t fundamental truths that we didn’t design and cannot escape. They’re synthetic, “entirely self-referential and made up,” spun out of layer upon layer of prior opinions—opinions warped by the same preferences and prejudices that stained this ruling. Even if Roberts could be a neutral umpire, the game he’s refereeing is itself a deeply flawed human creation, contrived by decidedly non-neutral architects—which is to say, mostly white men with money.

None of this implies the law isn’t valuable, or that the Supreme Court has no validity. It is, and it must.¹ But we can’t pretend that the court is something that it’s not. It is not a legal and moral calculator, or an impartial juror who weighs evidence “without respect to persons, and do equal right to the poor and to the rich.” It is—as it has always been—a political body, manned by political operators, with political opinions no different than Congresspeople, TV talking heads, and unhinged uncles on Facebook. The only difference is that justices are usually better at hiding it. 

To pretend otherwise is not only foolish, but dangerous. By assuming the court always acts in good faith, we surrender to legal sleights of hand that hide personal and political agendas behind manufactured arguments over “antecedent procedural questions.” In a world where the audience refuses to acknowledge the possibility they’re being deceived, the deftest magicians are king.

The reverence for quantitative rhetoric

In some circles—in “data-driven” companies, in much of Silicon Valley, in Nate Silver’s Twitter feed—data is also a form of magic. Arguments made with data are celebrated as unimpeachable level-minded science; arguments without it are the shrill opinions of a hysteric. Walk into a conversation without a supporting graph, and you’re hit with a Goodread link to W. Edwards Deming quotes: “Without data, you’re just another person with an opinion.” “In God we trust; all others must bring data.”

Making arguments from data, like interpreting the law through legal deliberation, isn’t inherently problematic. Quite the opposite, in fact—to the extent that it’s possible, data should be foundational. But, also like the law, it’s a foundation built on less level ground than we often admit.

Though we think of data as irrefutable ground truth, it is, in fact, also almost “entirely self-referential and made up.” Often, data—and its computational cousin, the metric—isn’t an abstract representation of an innate natural quality we’re attempting to quantify; it’s an accounting identity. The equation for GDP, for instance, is GDP. Change the equation and the definition of GDP changes. 

Nearly everything we measure, from inflation to Google Doc word counts, depends on similar identities—identities that are built on precedent, personal preferences, and arbitrary decisions.

Consider a corporate example: sales quota attainment. On the surface, this is easy and objective to compute. How much, in hard dollars and cents, did a sales rep sell? But, as with so many quantitative questions, there are lots of devils in the details. How much credit do reps get for products that were co-sold with partners? It depends on how much the company values their partners and how much they want to promote this sort of sale. How do we account for multi-year contracts? It depends on the biases of the company’s leadership, and whether they prefer aggressive growth or customer retention and satisfaction. If a customer cancels an annual agreement after four months, does the rep who sold the initial deal get full credit? It’s likely based on arbitrary industry standards and conventions. 

There are no right or wrong answers to these questions. There’s not a natural “sales quota attainment” out there that we’re trying to capture, if only our instruments were precise enough to detect it. Quota attainment exists only as the identity that defines it—an identity, by the way, that is created and maintained by fiat. If a leadership team wants to change how quotas are tallied—because of shifts in the business, because reps are selling in ways that aren't sustainable, because the company is struggling to hit its targets, or because the Sales VP is convinced a burning bush told them to do it—they can. 

Moreover, just as well-cited legalese and court letterhead doesn’t absolve a judge of shoehorning a political opinion into legal argument, tacking spreadsheets and probabilities on a personal preference doesn’t make it unbiased. In domains awash with data, any half decent analyst can make a compelling case for nearly anything. And critically—as is the case when usually shrewd Supreme Court justices hide their preferred outcomes behind legal technicalities—most people can’t always tell the difference. 

For an example of how this manifests, consider the recent debate about crime in San Francisco. Despite crime being a highly complex and nuanced issue, despite many of the San Franciscans engaging in the debate seeing themselves as free-thinking and rational, and despite everyone arguing with data, the two sides are well sorted by their political bent. Prior beliefs and biases determine how we see and present data, much more than data changes what we believe. 

In short, everything can be tilted: the data, the analysis, and the people analyzing it.

But, in spite of this malleability, our conceit for data (see: Deming, W. Edwards) protects those who wield it from accusations of bias. To use data is to be level-headed. The surest sign of fairness is to support your claims with numbers; the surest sign of prejudice is to fail to do so. Data is both a sword and a shield: It is a weapon for prosecuting your point, and a defense for protecting yourself as reasonable and impartial.

In the best cases, when data is used with the best intentions, this advantages those who are empirically comfortable. Deming’s demand to “bring data” doesn't necessarily make us better at seeking the truth; it just compels us toward a particular form of debate. It favors those who can make clumsy arguments with data over those who make clever ones without it.² It convinces analysts that we’re both the best at understanding stuff and the only people reasonable enough to be objective about it.

In the worst cases, data can be outright abused. This goes well beyond the usual complaints about misleading statistics and visualizations. The best quantitative illusionists don’t trim y-axes or talk in absolutes rather than ratios; they use data to hide their opinions behind sober analyses, pointing to their inside voices as proof of their neutrality. 

We can’t escape these flaws. For those of us who work with data, the solution isn’t to make our data or analysis more objective. We can’t. Our raw material is too tilted, as are we. We can't insulate our interpretations of data from our opinions any more than justices can pretend to insulate their legal reasoning from their political lives or the influences of their environments. 

Our responsibility, then, is to stop pretending otherwise. We should be humble about our own ability to be objective. We should pay more attention to others who aren’t as skilled in quantitative rhetoric. We aren’t inherently better or more unbiased than those who argue from emotion and experience; we simply take a different tact. And most of all, we shouldn’t elevate others who hang numbers on things to make a point. Data is persuasive, and plenty of sly con men are willing to exercise that power in irresponsible ways. As the people who are supposed to understand data, we, like those who understand the law, should make clear what all of us are: Humans, with opinions. 

1. In its current form, however, it is a profoundly undemocratic institution that should be reformed.

2. It’s also worth noting that this view, combined with other toxic prejudices, also favors white men.