"Nearly every news story, every blog post, every analyst report, and even every email that references some corporate statistic follows the same pattern..."
Without any contextless data to back up this claim, what am I to do? Simply take Benn at his word? Of course.
Not to respond with undue seriousness or in too optimistic of a tone, but I *think* that (at least inside of organizations) this problem can be fixed by two things:
1. citations that include reproducibility / provenance. tell me how you got to that number! could include a link to a data catalog, dashboard, whatever.
2. the ability of any reader to start a comment thread (as in gdocs/notion/etc) on top of any stat you post.
100% agree that this may not be soluble in the public internet. but with these two things, while everyone isn't going to be able to challenge every line of reasoning, _some people_ inside of an organization will. and they'll have the ability to socialize that in line with the original statement.
we have the opportunity to create an information context that lets more speech cure misleading speech! the structure of the communication channels and structure of the knowledge creation / dissemination tooling makes all the difference.
One thing I didn't think about here is how this dynamic differs on the internet vs inside companies. Even putting aside tooling possibilities, I think there are potentially two ways in which each world is very different:
1. The obvious one is that people in companies generally operate in better faith. I assume Twitter is full of people looking to make a point with data; inside companies, I'd guess that people are more inclined to look for the objective truth.
2. But, I'd guess that in companies, people are potentially more susceptible to fooling themselves. As you've pointed out before, there are definitely results you want to see in the data. And since most questions are only answered by one person, it's pretty easy to get a number like "20% of our users use this feature," think that feels high, and then go with the headline "Feature adoption is as high as 20%," without having any real check against it.
Which, put together, feels like I've talked myself into agreeing with you. In bad faith arguments, "more speech" often means clever people win, even if they're arguing from the wrong position. But in good faith ones, especially where the problem is not enough conversation, providing ways to facilitate a bit more of one feels like it'd go a long way to making this better.
Thanks! And not at all - people can and will very much mislead with statistics in all of the traditional ways. (And, for what it’s worth, while some people do do this in bad faith, it’s far more common for people to mislead themselves with statistics, and make good faith but badly reasoned arguments with them.)
But that doesn’t mean people don’t use data as a rhetorical tool too (again, sometimes unintentionally; it’s easy to fool yourself too). But I don’t think we’ve trained ourselves to be as on alert for this.
Analysts have already lost the battle there is nothing that can be done now. We are all paid shills.
"Nearly every news story, every blog post, every analyst report, and even every email that references some corporate statistic follows the same pattern..."
Without any contextless data to back up this claim, what am I to do? Simply take Benn at his word? Of course.
If I put no statistics in my arguments, I can't be accused of misusing them.
"experts say...." Sometimes I'm amazed at what there are experts in.
Not to respond with undue seriousness or in too optimistic of a tone, but I *think* that (at least inside of organizations) this problem can be fixed by two things:
1. citations that include reproducibility / provenance. tell me how you got to that number! could include a link to a data catalog, dashboard, whatever.
2. the ability of any reader to start a comment thread (as in gdocs/notion/etc) on top of any stat you post.
100% agree that this may not be soluble in the public internet. but with these two things, while everyone isn't going to be able to challenge every line of reasoning, _some people_ inside of an organization will. and they'll have the ability to socialize that in line with the original statement.
we have the opportunity to create an information context that lets more speech cure misleading speech! the structure of the communication channels and structure of the knowledge creation / dissemination tooling makes all the difference.
One thing I didn't think about here is how this dynamic differs on the internet vs inside companies. Even putting aside tooling possibilities, I think there are potentially two ways in which each world is very different:
1. The obvious one is that people in companies generally operate in better faith. I assume Twitter is full of people looking to make a point with data; inside companies, I'd guess that people are more inclined to look for the objective truth.
2. But, I'd guess that in companies, people are potentially more susceptible to fooling themselves. As you've pointed out before, there are definitely results you want to see in the data. And since most questions are only answered by one person, it's pretty easy to get a number like "20% of our users use this feature," think that feels high, and then go with the headline "Feature adoption is as high as 20%," without having any real check against it.
Which, put together, feels like I've talked myself into agreeing with you. In bad faith arguments, "more speech" often means clever people win, even if they're arguing from the wrong position. But in good faith ones, especially where the problem is not enough conversation, providing ways to facilitate a bit more of one feels like it'd go a long way to making this better.
Thanks! And not at all - people can and will very much mislead with statistics in all of the traditional ways. (And, for what it’s worth, while some people do do this in bad faith, it’s far more common for people to mislead themselves with statistics, and make good faith but badly reasoned arguments with them.)
But that doesn’t mean people don’t use data as a rhetorical tool too (again, sometimes unintentionally; it’s easy to fool yourself too). But I don’t think we’ve trained ourselves to be as on alert for this.