Benn, I share your concern about engagement optimization corrupting OpenAI's mission - I watched the same thing happen to Facebook through Steven Levy's book. Mark Zuckerberg really did start out wanting to connect the world, but engagement metrics drove everything.
But your question "what's the plan?" has answers at three levels:
As individuals: We're already seeing the vanguard figure this out. Gen Z's social media time dropped 10% from its 2022 peak - the first time that line has ever gone down (https://appedus.com/gen-z-social-media-decline-great-unplug/). 44% of teens cut back in 2024. The newsletter boom you're sitting on shows people wanted a different way to consume information. Some will fall for the drug, but the ones who make society move are the ones who don't. In the AI age, that means continuously educating yourself - even using AI to help with that - to maintain your own agency.
As builders: This is where I'd challenge you. You and I aren't spectators here - I've been building software since 1981, you've been writing about data and AI for years. We need to figure out what we can build that acts as a counterweight to the drug. I'm trying to build tools that help people be lifelong learners. As you find your next mission, the responsibility is the same - build something that pushes back against the addiction.
Societally: Every technology generates a moral panic. TV would make us passive (it didn't). The web would make us stupid because all the answers were there (it didn't). We're seeing real harms from social media, yes. But most technologies have enhanced our lives more than detracted from them. The question isn't whether AI will cause problems - it will. The question is whether we're building the counterweights.
We've been here before. We're starting to move away from social media's worst excesses. We'll do it again with AI.
I've been reassessing the moral panic around TV, and I think anyone panicking in the 1940s was right, for their social world did change.
Walk into a pre-war house. What's the first room inside the house? It's the sitting room. Outside the front door is also likely a sitting porch. These were practical rooms. These were rooms for visitors and visitation.
Fast forward 80 years, what is the sum of modern day affliction? Loneliness. The sitting rooms are empty. The data centers are full. Like Derek Thompson and Charlie Werzel recently said, the phone is just TV (see shorts, TikTok, reels). "The phone is now more television than television."
The "plan" as Benn lays out here is to have the phone be more heroin than heroin. Automated dopamine. An infinity loop of addiction.
Being existential about it isn't quite practical, but it is honest.
I do like that a fraction of the population is looking for screen-free anything. My wife and I garden. We are planning to share that garden experience with more people who want to bury their phone.
I have no pretense this is the majority of the world.
I'm not terribly optimistic of what we'll do as individuals (I mentioned this in another comment, but I think we are, on average, more or less the same as we'll ever be, and there is). I think it takes a few people (be it people building stuff, people setting social norms, or people setting policy) to create systems that encourage something better.
In this case, though, I'm not sure builders work that well. To some degree, for sure, and maybe in ways that establish some norms for what people expect out of the technology. But (and this is what I think happened with a lot of social media), if people are allowed to sell the drugs, the drugs are gonna sell pretty well. "Healthy" social media companies never really took off, because they were selling vegetables when someone else was selling candy.
The TV thing is interesting to me though, where yeah, on one hand, all new tech creates these questions, and maybe asking this questions is just being a "get off my lawn" old man. But on the other hand, sometimes these things do go badly (for me, social media, very much so went bad). And that's the thing about crying wolf - eventually, *there was a wolf.* So, is this a wolf or not? I don't know, though internet scale machines that optimize themselves to make people like them don't have a terribly great track record, in my mind.
And we keep getting the drugs at a younger & younger age. I'm seeing more & more toddlers playing on a smartphone in their stroller. It's gonna be pretty hard to break that habit later on...
It's the heroin metaphor. Definitely. GPT-bots should be in research labs only for now; they are clearly not safe for human consumption, and the performance gains are relatively minor. Some friction removed from goggling; replacing Substack or Reddit with Chat GPT; instant answers instead of having to ask and maybe waiting a while.
But the replacement of that grind comes at such a huge cost and relatively minor upsides...tis simply not worth it.
Unfortunately, I think at this point it's too late
I think I'm a more of a believer in the capabilities of the underlying technology (which I really do think is properly revolutionary, and strikingly so). I think it's the chatbot that's the problem. If you want to make Claude code super engaging, or a AI bot that reads and summaries legal docs, by all means. That seems mostly harmless to me, like trying to make a kindle enjoyable to read, or a knife as ergonomic as possible. But the uncontained engagement that comes with a general chatbot, where the evolutionary mandate is basically, make it as addictive as possible, through whatever means you can think of, seems not so great. But, to your point, I'm not sure that that cat is going back in the bag.
I'm a bit more in favor of the latter two, but even then I think they corrode something important in the developing of knowledge among people not already there; akin to the effect of how reading a book tends to imprint ideas better than watching a movie, because the book requires active effort to understand.
For those already experts it's something of a force multiplier, but for those who grow to rely on it from the beginning...I worry.
A force multiplier which will make it even more difficult to keep up with the top performers and result in even more inequality. Great. As if we didn't have enough of that already. More & more of the world will continue to be captured by fewer & fewer people. The slope is only getting steeper. And now we can get rid of paralegals, customer support agents, and countless other jobs, which will be replaced by.... nothing. When one person can do the work of ten, what happens to the other nine? Goodbye middle class. I hope they enjoy serving a shrinking number of ever richer people. But how much can one rich person possibly consume? Much less than a thriving middle class. The demand portion of the economy is about to go bye-bye. The wealthy will be bathing in luxury while everyone else can hopefully afford the bare essentials, until the whole thing breaks down.
It's like a rocketship is about to depart for Elysium -- who will be on board? And everyone else will be left behind to fight for the scraps.
And/or we all drown in slop created by people trying to get there. As if we weren't drowning in entertainment already. We don't need more (of anything, really). We're already saturated. This isn't like during the Industrial Revolution, when the world was in a period of exponential growth and in need of so much.
Anyway, it'll be great! Utopia definitely on the way.
Yeah, there's kind of interesting question in there to me (which I'm a million miles outside of my depth to be able to answer) about how much learning with ChatGPT is real learning. We can do way more, while knowing way less. Which, is that bad? Or is that great, that we can now do so much without the burden of having to spend a ton of time to learn all the ins and outs first? I'm not actually sure.
That seems right, though I guess my partial counterpoint would be Google. In a way, Google did the same thing, and made us some degree of an expert in lots of things - or at least made getting there a lot easier - and I think I'd say that's, on net, a good thing? Or at least I'd say I'm glad it gave me what it gave me (of course, everyone here may disagree with that...).
The main difference though is that what Google did is allowed you to really efficiently search through the corpus of 'anything a person had written online'. And prior to the GPTs, you could reasonably trust anything you read was written by a human, which meant you generally had to contend with 'Is this person actually an expert or some nerd who thinks they know more than they do?'
It wasn't perfect but even the nerds /usually/ weren't making stuff up out of whole cloth. Like, you weren't going to find them earnestly telling you it was safe to drink bleach (except satirically)
Whereas like, the GPTs have plenty of tendencies to do exactly that. And it's more insidious since usually Expert and Angry Nerd write certain ways that could let you tell them apart. Usually. but GPTs are like body snatchers behaving convincingly human until that -one- moment, and the danger isn't to someone like you who will go 'hnn thats wrong' its to someone who won't realize the AI is now saying its safe to drink bleach.
But does the world *need* to be saved? Saved from *what*, exactly? It's specifically from that very drug dealer that we need to be saved, so why do we keep going back to him for more, while he keeps using all the money to create even more powerful drugs? Where does it end?
The world has already been saved, but most people do not want that Savior. They prefer the drugs.
I do think there's sort of an interesting question there of, "if you accept the premise that this goes kinda bad - which could be totally wrong, I don't know - what's at the root of the problem?" Or, like, what would you have to change to avoid things going bad?
I don't think it's openai, or any company in particular. They seem to be responding to incentives in fairly normal ways, and I'm not sure it'd be useful for us to say "well, be better, and don't do that."
For better or for worse, it seems the issue is deeper than that, and is basically something like internet scale capitalism. Where things can reach everyone on the planet so quickly and so cheaply, "drugs" can overwhelm us before we have any idea what is happening.
It goes even deeper than that. Sin (selfishness/covetousness/greed/evil/etc.) is in our gut and in our bones. We're all slaves to it ("incentives", like you said; "normal", like you said). There's no escape other than being rescued. We'll never get completely out on our own (because the world is saturated with it). It's like survival instinct gone bad (and wealth/power secures survival like nothing else), and it's all being supported/normalized by our peers. And that's precisely the final enemy to defeat: death (and our fear of it, and our survival instinct to avoid it). Basically "I'd rather die than do wrong / harm others". Our Savior showed us how.
Systems (like capitalism) often do not help, but it's not about the systems; it's about the people. Same system + different people = different outcomes. Different system + same people = same outcomes (although some systems do put limits on how bad things can get, fortunately). There's no point in changing the rules when the same people are in power. They'll have their way one way or another. We need to change the people. There is no perfect system or set of rules that will turn ordinary people into angels. Game theory studies the systems, but we'll never find the answer that way.
Basically, how do we get people to the point where they are able to say "I won't do that"? No system can accomplish this. We have to change the people.
Heaven & Hell are not different systems -- they are different groups of people.
Ahh, see, I think I have the exact opposite view there. To me, it's the Ta-Nehisi Coates bit from this post (https://benn.substack.com/p/it-is-the-internet). We're all more or less the same (not individually, but groups of us all average to the same thing), and so all that matters is the system. There is no changing people; you can only change the systems in which they live.
I actually agree with you on this. Put people in a different system, and they will happily start killing Jews (or dissenters/heretics) in a heartbeat. We've seen it many times before, especially in the 20th century. But how do you change the system? Who knows the secret recipe to create a utopia? How do we bring Heaven on Earth? Which enlightened being will birth this new structure? That's not how power works. It doesn't end in the hands of the benevolent (save for a few miraculous exceptions). Even *if* we knew the way, do we really expect not to meet any resistance? People don't want change. They just want things to stay as they are (or to go back to the way they were). Maybe they will welcome change if it benefits them personally (i.e. more power). And isn't the obvious solution to persecute (or kill) those we perceive to be the problematic people?
That's why Jesus said "without me, you can do nothing". We need a higher power, because as natural brute beasts, we suck. We can't truly change the system. But He can, if we trust in Him. The only Way is to be transformed by the Love of God, which has been demonstrated to us through Jesus Christ. And then through faith in Him, we can contribute to changing the system (usually through sacrifice, which we rarely get to see the fruits of). That's basically what Western civilization is. But we're backsliding.
That's how we put an end to slavery. How and why would we ever have done it otherwise? It's a very profitable system (for those in power). Bobby Carter did what he did because of his faith in God. As he said himself: "My plans and advice have never been pleasing to the world" (but pleasing the world is what most people try to do). And he didn't see the fruits of his work.
The entire Bible is basically a guide against tyranny (if we understand it properly, rather than use it as a weapon or a tool of control, which is the very tyranny we're supposed to stand against).
I guess we could say that believing in a higher power *is* a change to the system/reality that we live in, because we no longer live in the same reality as everyone else. And our intimate knowledge of that higher power changes our reality even further -- especially once we come to understand that He is Love, and the enemy of tyranny.
Yeah, I wouldn't claim to have any idea how to answer the question of how you change that system. It seems like social pressure - either positive or negative - works about as well as anything though, where if you feel like you can't do something because other people will judge you for it, you won't; if you feel like you're supposed to do it, you will. So, anything that changes how you believe other people will perceive you is probably pretty effective. The internet has definitely done that, making a lot of stuff acceptable in ways it wasn't before. And religious faith certainly has too - probably in ways that are more effective than anything else humanity has done - in that it also creates those sorts of social pressures.
Well said, Benn.
Benn, I share your concern about engagement optimization corrupting OpenAI's mission - I watched the same thing happen to Facebook through Steven Levy's book. Mark Zuckerberg really did start out wanting to connect the world, but engagement metrics drove everything.
But your question "what's the plan?" has answers at three levels:
As individuals: We're already seeing the vanguard figure this out. Gen Z's social media time dropped 10% from its 2022 peak - the first time that line has ever gone down (https://appedus.com/gen-z-social-media-decline-great-unplug/). 44% of teens cut back in 2024. The newsletter boom you're sitting on shows people wanted a different way to consume information. Some will fall for the drug, but the ones who make society move are the ones who don't. In the AI age, that means continuously educating yourself - even using AI to help with that - to maintain your own agency.
As builders: This is where I'd challenge you. You and I aren't spectators here - I've been building software since 1981, you've been writing about data and AI for years. We need to figure out what we can build that acts as a counterweight to the drug. I'm trying to build tools that help people be lifelong learners. As you find your next mission, the responsibility is the same - build something that pushes back against the addiction.
Societally: Every technology generates a moral panic. TV would make us passive (it didn't). The web would make us stupid because all the answers were there (it didn't). We're seeing real harms from social media, yes. But most technologies have enhanced our lives more than detracted from them. The question isn't whether AI will cause problems - it will. The question is whether we're building the counterweights.
We've been here before. We're starting to move away from social media's worst excesses. We'll do it again with AI.
I've been reassessing the moral panic around TV, and I think anyone panicking in the 1940s was right, for their social world did change.
Walk into a pre-war house. What's the first room inside the house? It's the sitting room. Outside the front door is also likely a sitting porch. These were practical rooms. These were rooms for visitors and visitation.
Fast forward 80 years, what is the sum of modern day affliction? Loneliness. The sitting rooms are empty. The data centers are full. Like Derek Thompson and Charlie Werzel recently said, the phone is just TV (see shorts, TikTok, reels). "The phone is now more television than television."
The "plan" as Benn lays out here is to have the phone be more heroin than heroin. Automated dopamine. An infinity loop of addiction.
Being existential about it isn't quite practical, but it is honest.
I do like that a fraction of the population is looking for screen-free anything. My wife and I garden. We are planning to share that garden experience with more people who want to bury their phone.
I have no pretense this is the majority of the world.
Most addicts don't know they are addicted.
And it's even harder to recognize their addiction when it is being supported/normalized by all of their peers.
Well put.
I'm not terribly optimistic of what we'll do as individuals (I mentioned this in another comment, but I think we are, on average, more or less the same as we'll ever be, and there is). I think it takes a few people (be it people building stuff, people setting social norms, or people setting policy) to create systems that encourage something better.
In this case, though, I'm not sure builders work that well. To some degree, for sure, and maybe in ways that establish some norms for what people expect out of the technology. But (and this is what I think happened with a lot of social media), if people are allowed to sell the drugs, the drugs are gonna sell pretty well. "Healthy" social media companies never really took off, because they were selling vegetables when someone else was selling candy.
The TV thing is interesting to me though, where yeah, on one hand, all new tech creates these questions, and maybe asking this questions is just being a "get off my lawn" old man. But on the other hand, sometimes these things do go badly (for me, social media, very much so went bad). And that's the thing about crying wolf - eventually, *there was a wolf.* So, is this a wolf or not? I don't know, though internet scale machines that optimize themselves to make people like them don't have a terribly great track record, in my mind.
And we keep getting the drugs at a younger & younger age. I'm seeing more & more toddlers playing on a smartphone in their stroller. It's gonna be pretty hard to break that habit later on...
It's the heroin metaphor. Definitely. GPT-bots should be in research labs only for now; they are clearly not safe for human consumption, and the performance gains are relatively minor. Some friction removed from goggling; replacing Substack or Reddit with Chat GPT; instant answers instead of having to ask and maybe waiting a while.
But the replacement of that grind comes at such a huge cost and relatively minor upsides...tis simply not worth it.
Unfortunately, I think at this point it's too late
I think I'm a more of a believer in the capabilities of the underlying technology (which I really do think is properly revolutionary, and strikingly so). I think it's the chatbot that's the problem. If you want to make Claude code super engaging, or a AI bot that reads and summaries legal docs, by all means. That seems mostly harmless to me, like trying to make a kindle enjoyable to read, or a knife as ergonomic as possible. But the uncontained engagement that comes with a general chatbot, where the evolutionary mandate is basically, make it as addictive as possible, through whatever means you can think of, seems not so great. But, to your point, I'm not sure that that cat is going back in the bag.
I'm a bit more in favor of the latter two, but even then I think they corrode something important in the developing of knowledge among people not already there; akin to the effect of how reading a book tends to imprint ideas better than watching a movie, because the book requires active effort to understand.
For those already experts it's something of a force multiplier, but for those who grow to rely on it from the beginning...I worry.
A force multiplier which will make it even more difficult to keep up with the top performers and result in even more inequality. Great. As if we didn't have enough of that already. More & more of the world will continue to be captured by fewer & fewer people. The slope is only getting steeper. And now we can get rid of paralegals, customer support agents, and countless other jobs, which will be replaced by.... nothing. When one person can do the work of ten, what happens to the other nine? Goodbye middle class. I hope they enjoy serving a shrinking number of ever richer people. But how much can one rich person possibly consume? Much less than a thriving middle class. The demand portion of the economy is about to go bye-bye. The wealthy will be bathing in luxury while everyone else can hopefully afford the bare essentials, until the whole thing breaks down.
It's like a rocketship is about to depart for Elysium -- who will be on board? And everyone else will be left behind to fight for the scraps.
And/or we all drown in slop created by people trying to get there. As if we weren't drowning in entertainment already. We don't need more (of anything, really). We're already saturated. This isn't like during the Industrial Revolution, when the world was in a period of exponential growth and in need of so much.
Anyway, it'll be great! Utopia definitely on the way.
Yeah, there's kind of interesting question in there to me (which I'm a million miles outside of my depth to be able to answer) about how much learning with ChatGPT is real learning. We can do way more, while knowing way less. Which, is that bad? Or is that great, that we can now do so much without the burden of having to spend a ton of time to learn all the ins and outs first? I'm not actually sure.
I would say not great, because Hallucinations are a thing and without domain expertise, how can you tell it's giving you medicine or poison?
That's the main problem with relying on it - eventually your ability to tell its full of BS degrades to nothing.
That seems right, though I guess my partial counterpoint would be Google. In a way, Google did the same thing, and made us some degree of an expert in lots of things - or at least made getting there a lot easier - and I think I'd say that's, on net, a good thing? Or at least I'd say I'm glad it gave me what it gave me (of course, everyone here may disagree with that...).
The main difference though is that what Google did is allowed you to really efficiently search through the corpus of 'anything a person had written online'. And prior to the GPTs, you could reasonably trust anything you read was written by a human, which meant you generally had to contend with 'Is this person actually an expert or some nerd who thinks they know more than they do?'
It wasn't perfect but even the nerds /usually/ weren't making stuff up out of whole cloth. Like, you weren't going to find them earnestly telling you it was safe to drink bleach (except satirically)
Whereas like, the GPTs have plenty of tendencies to do exactly that. And it's more insidious since usually Expert and Angry Nerd write certain ways that could let you tell them apart. Usually. but GPTs are like body snatchers behaving convincingly human until that -one- moment, and the danger isn't to someone like you who will go 'hnn thats wrong' its to someone who won't realize the AI is now saying its safe to drink bleach.
Oh man, that ending!! 👌🔥
But does the world *need* to be saved? Saved from *what*, exactly? It's specifically from that very drug dealer that we need to be saved, so why do we keep going back to him for more, while he keeps using all the money to create even more powerful drugs? Where does it end?
The world has already been saved, but most people do not want that Savior. They prefer the drugs.
I do think there's sort of an interesting question there of, "if you accept the premise that this goes kinda bad - which could be totally wrong, I don't know - what's at the root of the problem?" Or, like, what would you have to change to avoid things going bad?
I don't think it's openai, or any company in particular. They seem to be responding to incentives in fairly normal ways, and I'm not sure it'd be useful for us to say "well, be better, and don't do that."
For better or for worse, it seems the issue is deeper than that, and is basically something like internet scale capitalism. Where things can reach everyone on the planet so quickly and so cheaply, "drugs" can overwhelm us before we have any idea what is happening.
It goes even deeper than that. Sin (selfishness/covetousness/greed/evil/etc.) is in our gut and in our bones. We're all slaves to it ("incentives", like you said; "normal", like you said). There's no escape other than being rescued. We'll never get completely out on our own (because the world is saturated with it). It's like survival instinct gone bad (and wealth/power secures survival like nothing else), and it's all being supported/normalized by our peers. And that's precisely the final enemy to defeat: death (and our fear of it, and our survival instinct to avoid it). Basically "I'd rather die than do wrong / harm others". Our Savior showed us how.
Systems (like capitalism) often do not help, but it's not about the systems; it's about the people. Same system + different people = different outcomes. Different system + same people = same outcomes (although some systems do put limits on how bad things can get, fortunately). There's no point in changing the rules when the same people are in power. They'll have their way one way or another. We need to change the people. There is no perfect system or set of rules that will turn ordinary people into angels. Game theory studies the systems, but we'll never find the answer that way.
Basically, how do we get people to the point where they are able to say "I won't do that"? No system can accomplish this. We have to change the people.
Heaven & Hell are not different systems -- they are different groups of people.
Ahh, see, I think I have the exact opposite view there. To me, it's the Ta-Nehisi Coates bit from this post (https://benn.substack.com/p/it-is-the-internet). We're all more or less the same (not individually, but groups of us all average to the same thing), and so all that matters is the system. There is no changing people; you can only change the systems in which they live.
I actually agree with you on this. Put people in a different system, and they will happily start killing Jews (or dissenters/heretics) in a heartbeat. We've seen it many times before, especially in the 20th century. But how do you change the system? Who knows the secret recipe to create a utopia? How do we bring Heaven on Earth? Which enlightened being will birth this new structure? That's not how power works. It doesn't end in the hands of the benevolent (save for a few miraculous exceptions). Even *if* we knew the way, do we really expect not to meet any resistance? People don't want change. They just want things to stay as they are (or to go back to the way they were). Maybe they will welcome change if it benefits them personally (i.e. more power). And isn't the obvious solution to persecute (or kill) those we perceive to be the problematic people?
That's why Jesus said "without me, you can do nothing". We need a higher power, because as natural brute beasts, we suck. We can't truly change the system. But He can, if we trust in Him. The only Way is to be transformed by the Love of God, which has been demonstrated to us through Jesus Christ. And then through faith in Him, we can contribute to changing the system (usually through sacrifice, which we rarely get to see the fruits of). That's basically what Western civilization is. But we're backsliding.
That's how we put an end to slavery. How and why would we ever have done it otherwise? It's a very profitable system (for those in power). Bobby Carter did what he did because of his faith in God. As he said himself: "My plans and advice have never been pleasing to the world" (but pleasing the world is what most people try to do). And he didn't see the fruits of his work.
The entire Bible is basically a guide against tyranny (if we understand it properly, rather than use it as a weapon or a tool of control, which is the very tyranny we're supposed to stand against).
I guess we could say that believing in a higher power *is* a change to the system/reality that we live in, because we no longer live in the same reality as everyone else. And our intimate knowledge of that higher power changes our reality even further -- especially once we come to understand that He is Love, and the enemy of tyranny.
Yeah, I wouldn't claim to have any idea how to answer the question of how you change that system. It seems like social pressure - either positive or negative - works about as well as anything though, where if you feel like you can't do something because other people will judge you for it, you won't; if you feel like you're supposed to do it, you will. So, anything that changes how you believe other people will perceive you is probably pretty effective. The internet has definitely done that, making a lot of stuff acceptable in ways it wasn't before. And religious faith certainly has too - probably in ways that are more effective than anything else humanity has done - in that it also creates those sorts of social pressures.