I agree that often times data is used to create a sense of authority where none rightly exists. I will go a bit farther to say that it often is used to create the illusion that a particular decision is the only plausible one. People use it to claim they have no agency over a policy or a decision when in fact they are swimming in it.
I do think in most of the cases where data is being used to create such a false sense of objectivity, the practitioner is rarely attempting to actively deceive the audience. The are either taking shortcuts to serve some greater point or they are pulling a Medawar. "Its author can be excused of dishonesty only on the grounds that before deceiving others he has taken great pains to deceive himself."
But data humans also deliver synthetic truths, and in many cases we can undertake activities that can check the underlying veracity of some of our claims. We can look for testimony from primary sources, we can do external validation. We can use statistical distributions to see if your synthetic metrics are producing stochastic variations or whether they can be attributed to some consistent cause. We can run a damn experiment.
I am not sure that the law has any such tools to ground itself in something other than motivated reasoning. Or I am not sure they are any good. I will leave that discussion to others.
It is true that there is to some degree, opinion all the way down in the data. You can't be perfectly objective. But I think that the implicit inference that we should despair of the idea that some opinions are more objective than others is a rhetorical conceit we would not accept if we were not already feeling catastrophic about the subject. We would not accept an argument that a heuristic is no good just by knowing it can't seperate cases perfectly.
(I also feel it is worth pointing out that pessimism also tends to lend an argument a deceptive feeling of objectivity in much the same way that meaningless enumeration can.)
I don't think I disagree with this. I don't think data is worthless (nor is the law). In some instances, data isn't opinions all the way down; there are certainly cases where data *is* an abstract representation of some natural quality. Even in cases when it's not, people can, as you said, run experiments, or use different sources, from ask questions from different angles, and so on. I think it's fair to say those are both ways in which the legal analogy breaks down, save asking for a second and third and fourth appeal (which still doesn't really work, since we just go with what the last person said, rather than trying judge them together like we might with analysis).
Moreover, even if it were all opinion, it's still better than nothing, and some opinions are more defensible than others. "Analytic" truths or rules are better than Calvinball.
Still, I think we overweight these rules, and in particular, the people who use them. And that latter point is my bigger gripe.
With the law, take the abortion issue. If we were to ask people what they think of SB8, we'd correctly assume that most people's position is rooted in what they personally think of abortion. But if they suddenly toss in some legal citations, that preference goes somewhere from being a secondary concern to entirely off-limits, an ad hominem attack in which the person accusing them of having an opinion is the biased one.
Data is similar. If you don't use it, you're a shill; if you do, and someone disagrees with you, they're the shill. Even in cases when everyone's trying their best, that's a big problem. It invalidates some arguments (and people who aren't skilled in those arguments) on their face, and validates others. The issue here isn't that people will always abuse it, though some will. It's that it incorrectly sorts arguments and people. It also perpetuates itself by partly undermining this idea: "Its author can be excused of dishonesty only on the grounds that before deceiving others he has taken great pains to deceive himself." The more society elevates your arguments, the fewer pains you actually have to go to deceive yourself. As analysts or judges, you get shoddy when you get smug.
Nate Silver's too easy of an example here, but he really checks all the boxes. Things like his primary forecasting is junk science; he has no particular expertise in anything to do with covid. But because he tacks numbers onto his opinions, it's treated as much more than punditry. And he's convinced himself he's the only one who's seeing things clearly because he's the numbers guy, and unless you're also a numbers guy, you shouldn't be taken seriously.
Now, most of us aren't that conceited. But shades of that attitude are pretty prevalent in analytics.
maybe it's unrepresentative, but whenever I see Nate Silver's name trending on twitter it is 1000 people screaming at him, so there are probably better examples.
True. But, if telling actual epidemiologists they're hacks for 18 months in the middle of a global pandemic is what it takes for a data analyst to fly too close to the sun, we're getting an awful lot of leeway.
I agree that often times data is used to create a sense of authority where none rightly exists. I will go a bit farther to say that it often is used to create the illusion that a particular decision is the only plausible one. People use it to claim they have no agency over a policy or a decision when in fact they are swimming in it.
I do think in most of the cases where data is being used to create such a false sense of objectivity, the practitioner is rarely attempting to actively deceive the audience. The are either taking shortcuts to serve some greater point or they are pulling a Medawar. "Its author can be excused of dishonesty only on the grounds that before deceiving others he has taken great pains to deceive himself."
But we data folk do have tools that help us to keep from fooling ourselves. I think the piece focuses on cases where the truths under consideration are more "analytic" in the very old sense of the word. https://www.oxfordbibliographies.com/view/document/obo-9780195396577/obo-9780195396577-0044.xml
But data humans also deliver synthetic truths, and in many cases we can undertake activities that can check the underlying veracity of some of our claims. We can look for testimony from primary sources, we can do external validation. We can use statistical distributions to see if your synthetic metrics are producing stochastic variations or whether they can be attributed to some consistent cause. We can run a damn experiment.
I am not sure that the law has any such tools to ground itself in something other than motivated reasoning. Or I am not sure they are any good. I will leave that discussion to others.
It is true that there is to some degree, opinion all the way down in the data. You can't be perfectly objective. But I think that the implicit inference that we should despair of the idea that some opinions are more objective than others is a rhetorical conceit we would not accept if we were not already feeling catastrophic about the subject. We would not accept an argument that a heuristic is no good just by knowing it can't seperate cases perfectly.
(I also feel it is worth pointing out that pessimism also tends to lend an argument a deceptive feeling of objectivity in much the same way that meaningless enumeration can.)
I don't think I disagree with this. I don't think data is worthless (nor is the law). In some instances, data isn't opinions all the way down; there are certainly cases where data *is* an abstract representation of some natural quality. Even in cases when it's not, people can, as you said, run experiments, or use different sources, from ask questions from different angles, and so on. I think it's fair to say those are both ways in which the legal analogy breaks down, save asking for a second and third and fourth appeal (which still doesn't really work, since we just go with what the last person said, rather than trying judge them together like we might with analysis).
Moreover, even if it were all opinion, it's still better than nothing, and some opinions are more defensible than others. "Analytic" truths or rules are better than Calvinball.
Still, I think we overweight these rules, and in particular, the people who use them. And that latter point is my bigger gripe.
With the law, take the abortion issue. If we were to ask people what they think of SB8, we'd correctly assume that most people's position is rooted in what they personally think of abortion. But if they suddenly toss in some legal citations, that preference goes somewhere from being a secondary concern to entirely off-limits, an ad hominem attack in which the person accusing them of having an opinion is the biased one.
Data is similar. If you don't use it, you're a shill; if you do, and someone disagrees with you, they're the shill. Even in cases when everyone's trying their best, that's a big problem. It invalidates some arguments (and people who aren't skilled in those arguments) on their face, and validates others. The issue here isn't that people will always abuse it, though some will. It's that it incorrectly sorts arguments and people. It also perpetuates itself by partly undermining this idea: "Its author can be excused of dishonesty only on the grounds that before deceiving others he has taken great pains to deceive himself." The more society elevates your arguments, the fewer pains you actually have to go to deceive yourself. As analysts or judges, you get shoddy when you get smug.
Nate Silver's too easy of an example here, but he really checks all the boxes. Things like his primary forecasting is junk science; he has no particular expertise in anything to do with covid. But because he tacks numbers onto his opinions, it's treated as much more than punditry. And he's convinced himself he's the only one who's seeing things clearly because he's the numbers guy, and unless you're also a numbers guy, you shouldn't be taken seriously.
Now, most of us aren't that conceited. But shades of that attitude are pretty prevalent in analytics.
maybe it's unrepresentative, but whenever I see Nate Silver's name trending on twitter it is 1000 people screaming at him, so there are probably better examples.
True. But, if telling actual epidemiologists they're hacks for 18 months in the middle of a global pandemic is what it takes for a data analyst to fly too close to the sun, we're getting an awful lot of leeway.
This is the first post where I think I have beef. I don't disagree with the overall point but I do think it is incomplete in a way that is important.