Tilt and tilted
Data is not the level ground that we act like it is.
Last week, in an midnight order declining to block a Texas law banning most abortions, five justices on Supreme Court of the United States revealed themselves to be what they’ve always been, no matter how hard we’ve tried to pretend otherwise: People, with opinions.
When we talk about the Supreme Court, we place the institution outside—and often above—the political fray. We’re taught that the judgements of the court come from careful readings of the law, deliberate considerations of facts and precedent, and the country’s sharpest legal minds. Though justices ascend to the bench from prior careers as political appointees and lawyers, campaign managers, and members of activist organizations, we act as though those histories—not to mention their upbringings and religious convictions—are purged the moment they take their lifetime seat on the court.
Sometimes this is explicit; during his confirmation hearing, Chief Justice John Roberts said his job was to be an umpire, calling balls and strikes. But more often, this framing is implicit, a reflexive deference to the court and those who sit on it. Just last week, the New York Times’ David Leonhardt commented that justices usually “solicit briefs, hold oral arguments and spend months grappling with their decision” (emphasis mine). While we may disagree with what the justices decide, we want to believe in the complexity of the decisions they make, the good faith from which they make it, and the legal ground on which they rest. And many people, from civics teachers to op-ed columnists to the justices themselves, are eager to sell this vision.
Last week’s ruling tears the mask off.
The five justices in the majority showed that Leonhardt was half right—justices do need time to issue rulings. But that time isn’t for figuring out what to do; it’s for finding a plausible argument for how to do it. In this case, evidently so eager to overturn fifty years of precedent, they didn’t bother.
The Texas law, known as SB8, has two significant elements. The first bans nearly all abortions in the state. The second delegates enforcement of the law to the citizenry. Under SB8, individual citizens can sue for and collect $10,000 from anyone involved in providing an abortion, including doctors, clinicians, people who pay for procedures, and people who drive patients to clinics. The bizarre and dystopian scheme, devised in plain sight, was designed to confuse legal challenges to a plainly unconstitutional law—or at least give a potentially friendly court that wants to ban abortions an excuse to do it.
As Justice Sotomayor said in her dissent, “the gambit worked.” The Supreme Court declined to block the law because it raises “complex and novel antecedent procedural questions.” As should be obvious to everyone, this is a brazen lie. The court is corrupt and corrupted. The question at hand was not complicated. This was not a careful and unbiased reading of the facts. This was not an opinion rooted in laws; it is now a law, rooted in opinions. This was, per Adam Serwer, five justices invalidating a previously upheld constitutional right to abortion “because they wanted to, because they could, and because fuck you.”
Of course, justices are people too, swayed by their own opinions, in ways that may be invisible even to them. Good LSAT scores and long resumes don’t make people immune to personal desires. That justices would rule on SB8 in accordance with their own preferences isn’t a surprise. The surprise is how audaciously they did it.
But, even if they had been more delicate—or attempted their best good-faith reading of the law and prior precedent—the ruling still wouldn’t be impartial.
For our laws are not natural ones. They aren’t fundamental truths that we didn’t design and cannot escape. They’re synthetic, “entirely self-referential and made up,” spun out of layer upon layer of prior opinions—opinions warped by the same preferences and prejudices that stained this ruling. Even if Roberts could be a neutral umpire, the game he’s refereeing is itself a deeply flawed human creation, contrived by decidedly non-neutral architects—which is to say, mostly white men with money.
None of this implies the law isn’t valuable, or that the Supreme Court has no validity. It is, and it must.1 But we can’t pretend that the court is something that it’s not. It is not a legal and moral calculator, or an impartial juror who weighs evidence “without respect to persons, and do equal right to the poor and to the rich.” It is—as it has always been—a political body, manned by political operators, with political opinions no different than Congresspeople, TV talking heads, and unhinged uncles on Facebook. The only difference is that justices are usually better at hiding it.
To pretend otherwise is not only foolish, but dangerous. By assuming the court always acts in good faith, we surrender to legal sleights of hand that hide personal and political agendas behind manufactured arguments over “antecedent procedural questions.” In a world where the audience refuses to acknowledge the possibility they’re being deceived, the deftest magicians are king.
The reverence for quantitative rhetoric
In some circles—in “data-driven” companies, in much of Silicon Valley, in Nate Silver’s Twitter feed—data is also a form of magic. Arguments made with data are celebrated as unimpeachable level-minded science; arguments without it are the shrill opinions of a hysteric. Walk into a conversation without a supporting graph, and you’re hit with a Goodread link to W. Edwards Deming quotes: “Without data, you’re just another person with an opinion.” “In God we trust; all others must bring data.”
Making arguments from data, like interpreting the law through legal deliberation, isn’t inherently problematic. Quite the opposite, in fact—to the extent that it’s possible, data should be foundational. But, also like the law, it’s a foundation built on less level ground than we often admit.
Though we think of data as irrefutable ground truth, it is, in fact, also almost “entirely self-referential and made up.” Often, data—and its computational cousin, the metric—isn’t an abstract representation of an innate natural quality we’re attempting to quantify; it’s an accounting identity. The equation for GDP, for instance, is GDP. Change the equation and the definition of GDP changes.
Nearly everything we measure, from inflation to Google Doc word counts, depends on similar identities—identities that are built on precedent, personal preferences, and arbitrary decisions.
Consider a corporate example: sales quota attainment. On the surface, this is easy and objective to compute. How much, in hard dollars and cents, did a sales rep sell? But, as with so many quantitative questions, there are lots of devils in the details. How much credit do reps get for products that were co-sold with partners? It depends on how much the company values their partners and how much they want to promote this sort of sale. How do we account for multi-year contracts? It depends on the biases of the company’s leadership, and whether they prefer aggressive growth or customer retention and satisfaction. If a customer cancels an annual agreement after four months, does the rep who sold the initial deal get full credit? It’s likely based on arbitrary industry standards and conventions.
There are no right or wrong answers to these questions. There’s not a natural “sales quota attainment” out there that we’re trying to capture, if only our instruments were precise enough to detect it. Quota attainment exists only as the identity that defines it—an identity, by the way, that is created and maintained by fiat. If a leadership team wants to change how quotas are tallied—because of shifts in the business, because reps are selling in ways that aren't sustainable, because the company is struggling to hit its targets, or because the Sales VP is convinced a burning bush told them to do it—they can.
Moreover, just as well-cited legalese and court letterhead doesn’t absolve a judge of shoehorning a political opinion into legal argument, tacking spreadsheets and probabilities on a personal preference doesn’t make it unbiased. In domains awash with data, any half decent analyst can make a compelling case for nearly anything. And critically—as is the case when usually shrewd Supreme Court justices hide their preferred outcomes behind legal technicalities—most people can’t always tell the difference.
For an example of how this manifests, consider the recent debate about crime in San Francisco. Despite crime being a highly complex and nuanced issue, despite many of the San Franciscans engaging in the debate seeing themselves as free-thinking and rational, and despite everyone arguing with data, the two sides are well sorted by their political bent. Prior beliefs and biases determine how we see and present data, much more than data changes what we believe.
In short, everything can be tilted: the data, the analysis, and the people analyzing it.
But, in spite of this malleability, our conceit for data (see: Deming, W. Edwards) protects those who wield it from accusations of bias. To use data is to be level-headed. The surest sign of fairness is to support your claims with numbers; the surest sign of prejudice is to fail to do so. Data is both a sword and a shield: It is a weapon for prosecuting your point, and a defense for protecting yourself as reasonable and impartial.
In the best cases, when data is used with the best intentions, this advantages those who are empirically comfortable. Deming’s demand to “bring data” doesn't necessarily make us better at seeking the truth; it just compels us toward a particular form of debate. It favors those who can make clumsy arguments with data over those who make clever ones without it.2 It convinces analysts that we’re both the best at understanding stuff and the only people reasonable enough to be objective about it.
In the worst cases, data can be outright abused. This goes well beyond the usual complaints about misleading statistics and visualizations. The best quantitative illusionists don’t trim y-axes or talk in absolutes rather than ratios; they use data to hide their opinions behind sober analyses, pointing to their inside voices as proof of their neutrality.
We can’t escape these flaws. For those of us who work with data, the solution isn’t to make our data or analysis more objective. We can’t. Our raw material is too tilted, as are we. We can't insulate our interpretations of data from our opinions any more than justices can pretend to insulate their legal reasoning from their political lives or the influences of their environments.
Our responsibility, then, is to stop pretending otherwise. We should be humble about our own ability to be objective. We should pay more attention to others who aren’t as skilled in quantitative rhetoric. We aren’t inherently better or more unbiased than those who argue from emotion and experience; we simply take a different tact. And most of all, we shouldn’t elevate others who hang numbers on things to make a point. Data is persuasive, and plenty of sly con men are willing to exercise that power in irresponsible ways. As the people who are supposed to understand data, we, like those who understand the law, should make clear what all of us are: Humans, with opinions.
In its current form, however, it is a profoundly undemocratic institution that should be reformed.
It’s also worth noting that this view, combined with other toxic prejudices, also favors white men.
An analogous case is the current flap over gender issues.Executives and legislators have declared that "there are only two genders," and that all the attempts to explain things such as gender dysphoria are just "woke dementia" masquerading in the form of serious medicall and psychological knowledge.
The first thing to be noted is that the media has, as it all to often done, miscommunicated about scientific matters by, e.g., saying that "science proves such-and-so" when in fact science deliberately rules out proving anything. The reason is that you may have a large body of confirming evidence, but when you turn the next corner a contrary case shows the claim isn't true. "All swans are white" was seemed to be true until the first explorer brought back news from Australia of black swans. That's why philosophy of science people still frequently refer to black swans. With regard to gender, the mass media have frequently taken it as a "safe" term to use instead of using that terribly nasty word "sex."
Until sometime around 1950, "gender" was a term that was used solely in grammatical and language discissions. In German, what is the gender of a pencil? It depends on whether grammatically it is sppoken of as a he or a she, much as in English ships are traditionally ccalled "she."
John Money of Johns Hopkins University began his study of what is going on when someone declare that despite having male genitalia, the individual feels that the genitalia are inappropriate. Such an individual is spoken of, since Money's innovation, as having male sex but female gender. Then the question becomes, "How in hell is this sipposed to be explained?"
People have been working on this general problem for about a century, and they have been working on it as scientists, which can mean scientific papers that oppose each other, followed by more research, followed by eliminating some conclusions that turned out not to be as well-founded as the investigator supposed. People do not remember it, but the discoverer of https://en.wikipedia.org/wiki/Kuru_(disease) was treated with extreme contempt in some circles (not a proper scientist at all), but in the end won out and received a Nobel Prize in Medicine. The same fighting occurs all the time if you are editing something on Wikipedia. All of which is to say that it's very unlikely that thousands of people who have worked in the field just decided to fabricate a plausible "urban myth" to fob off on the people. Instead, as I understand it, a huge amount of research has been done, the research findings have been tried against each other, amd a consensus position has emerged regarding the general nature of the phenomena. (Details are still being hashed out and might change some aspects of that consensus.)
The consensus of a field of specialists who have made it their life's work has emerged, and anybody who choses to go beyond the mass media can find it just by checking with Google Scholar. (Lots of that stuff is behind a pay wall, but you can get at least one of these consensus reports in the first couple of pages of https://pure.knaw.nl/ws/portalfiles/portal/5305590/FunctNeurol09Swaab.pdf
Studies that trace out fetal development also point to possible explanations for autism, homosexuality, etc. It turns out that genitalia develop in the first half of gestation, and many brain structures are laid out in the second half of gestation and are subject to all kinds of interferences from uterine environmental factors.
Certain politicians may not even know that this research exists. To them the matter is simple. If you have a penis and testicals you are a male and you must behave in what I the leader declare is the proper masculine way.If you don't, we will come after you. The same treatment should handle individuals with female genitalia. You will behave like Ricky Nelson's TV mom, or you are in for it.
I agree that often times data is used to create a sense of authority where none rightly exists. I will go a bit farther to say that it often is used to create the illusion that a particular decision is the only plausible one. People use it to claim they have no agency over a policy or a decision when in fact they are swimming in it.
I do think in most of the cases where data is being used to create such a false sense of objectivity, the practitioner is rarely attempting to actively deceive the audience. The are either taking shortcuts to serve some greater point or they are pulling a Medawar. "Its author can be excused of dishonesty only on the grounds that before deceiving others he has taken great pains to deceive himself."
But we data folk do have tools that help us to keep from fooling ourselves. I think the piece focuses on cases where the truths under consideration are more "analytic" in the very old sense of the word. https://www.oxfordbibliographies.com/view/document/obo-9780195396577/obo-9780195396577-0044.xml
But data humans also deliver synthetic truths, and in many cases we can undertake activities that can check the underlying veracity of some of our claims. We can look for testimony from primary sources, we can do external validation. We can use statistical distributions to see if your synthetic metrics are producing stochastic variations or whether they can be attributed to some consistent cause. We can run a damn experiment.
I am not sure that the law has any such tools to ground itself in something other than motivated reasoning. Or I am not sure they are any good. I will leave that discussion to others.
It is true that there is to some degree, opinion all the way down in the data. You can't be perfectly objective. But I think that the implicit inference that we should despair of the idea that some opinions are more objective than others is a rhetorical conceit we would not accept if we were not already feeling catastrophic about the subject. We would not accept an argument that a heuristic is no good just by knowing it can't seperate cases perfectly.
(I also feel it is worth pointing out that pessimism also tends to lend an argument a deceptive feeling of objectivity in much the same way that meaningless enumeration can.)