So-called AI is built on hype and its widespread adoption promises to dehumanize us and wreak environmental havoc. Emily M. Bender, computational linguist and co-author of The AI Con, describes how AI's 'boomers' and 'doomers' are two sides of the same coin - trying to convince us of AI's all-powerful inevitability. Emily points toward a path of refusal of this harmful emergent technology being pushed on us by the world's richest and most powerful, and she offers a vision of how we can resist and begin creating the future we actually want. Highlights include:
Why the phrase 'artificial intelligence' is just a marketing term used to hype everything from chatbots to image processing systems to facial recognition;
Why the concept of 'intelligence' itself is loaded with eugenics, race science history, and a stunted view of biological beings;
Why large language models (LLMs) in the form of chatbots are devoid of the relational context of subjectivity, care, and conflict which humans use to negotiate their social contexts and which give human language meaning and accountability;
How AI hype plays on FOMO (fear of missing out) and has led everyone from business, government, education, entertainment, and media to thoughtlessly embrace so-called AI at a rapid rate;
How AI hype is used by both AI boomers and doomers to oversell the potential of their technologies and hide the serious environmental and labor harms that AI use is causing right now;
Why so-called AI will likely lead not to a job apocalypse but increasing work gigification and dehumanization;
How effective altruism, which is backed by Silicon Valley money and power, replaces relationality and accountability with cold mathematical notions of optimization;
Why the ridiculous idea of “AI sentience” is pulling attention away from expanding rights to all humans and nonhuman animals;
Why there's ultimately no responsible use of so-called AI and why it has no place in education where it atrophies critical thinking skills and damages the learning community;
How individuals can resist AI in their everyday lives by refusing chatbots, automated search, automated doctor scribes, airport facial screening - and collectively push back through labor activism and citizen resistance to data centers.
MENTIONED IN THIS EPISODE:
Podcast: Mystery AI Hype Theater 3000 with cohosts Emily M. Bender and Alex Hanna
Podcast episode: The Dangers of Effective Altruism with guest Alice Crary on the OVERSHOOT podcast
Documentary Film: Ghost in the Machine: A Documentary About AI by Valerie Veatch
Book: Enshittification: Why Everything Suddenly Got Worse and What to Do About It by Cory Doctorow
-
Emily M. Bender (00:00:00):
Part of the hype narrative about so-called artificial intelligence is the boosters say AI is a thing, it's inevitable, it's imminent, and it's going to solve all of our problems, including the climate crisis somehow. And the doomers say AI is a thing, it's inevitable, it's imminent, and it's going to kill us all. And that kill us all narrative is not coming from the environmental ruin associated with these systems, which actually is an existential risk, but instead these weird thought experiments of, well, if we had a all powerful artificial intelligence and if it didn't like us for some reason, well then what would follow? So the anthropomorphizing language trick there is to talk about it as if the software has goals, desires, the ability to plan something on its own rather than being a designed, built artifact that somebody designed and built to do a certain thing.
(00:00:53):
The first thing I want to say is that those two positions are the same and there's plenty of room outside of that coin for positions that understand this technology for what it is and the harms that are happening.
Alan Ware (00:01:04):
That was linguistics professor Emily Bender. In this episode of OVERSHOOT, we'll talk with Emily about her recent coauthored book, The AI Con: How to Fight Big Tech's Hype and Create the Future We Want.
Nandita Bajaj (00:01:28):
Welcome to OVERSHOOT, where we tackle today's interlocking social and ecological crises driven by humanity's excessive population and consumption. On this podcast, we explore needed narrative, behavioral, and system shifts for recreating human life in balance with all life on Earth. I'm Nandita Bajaj, co-host of the podcast and executive director of Population Balance.
Alan Ware (00:01:53):
I'm Alan Ware, co-host of the podcast and researcher with Population Balance. With expert guests covering a range of topics, we examine the forces underlying overshoot - the patriarchal pronatalism that fuels overpopulation, the growth- obsessed economic systems that drive consumerism and social injustice, and the dominant worldview of human supremacy that subjugates animals and nature. Our vision of shrinking toward abundance inspires us to seek pathways of transformation that go beyond technological fixes toward a new humanity that honors our interconnectedness with all of life. And now on to today's guest.
(00:02:32):
Emily M. Bender is a professor of linguistics and an adjunct professor in the School of Computer Science and the Information School at the University of Washington, where she has been on the faculty since 2003. Her research interests include multilingual grammar engineering, computational semantics, and the societal impacts of language technology. She's the coauthor of recent influential papers such as Climbing Towards NLU: On Meaning, Form, and Understanding in the Age of Data and On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? and the book with sociologist Dr. Alex Hanna, The AI con: How to Fight Big Tech's Hype and Create the Future We Want. In her public scholarship, Bender brings linguistic insights to lay audiences to cut through the hype about so-called artificial intelligence and facilitate understanding of the actual functionality of the systems being sold under that name. And now on to today's interview.
Nandita Bajaj (00:03:34):
Hi, Emily. Welcome to the OVERSHOOT Podcast. We are thrilled to have you here.
Emily M. Bender (00:03:39):
This is, I think, going to be a really fun discussion, and I love the space to think about how scale is harmful, especially because scale is so central to this project called artificial intelligence.
Nandita Bajaj (00:03:51):
Yes, indeed. And we came across your excellent work, debunking the hype surrounding the technologies being sold as AI, in the recent documentary Ghost in the Machine by Valerie Veach. And a number of years ago, we started noticing that Silicon Valley and big tech were increasingly amplifying some of the very ideologies that we challenge on this podcast, which include pronatalism, human supremacy, technological fundamentalism, and growthism, which as you identified, it has to do so much with scaling everything up. And with so much of our collective imagination being hijacked by the hype around so-called artificial intelligence, we were thrilled to read your book, The AI Con, that you coauthored with Alex Hanna, and which we will be discussing today to help us better understand these large language models and their chatbots that are being sold as "intelligent" and even sentient and how we might best resist the tech industry that's pushing it.
(00:04:59):
So as a linguist, you examine how language works and how people work with language. You've described artificial intelligence as a marketing term that encompasses several very different forms of technology. What are some of those technologies and why do you argue that none of them constitute "intelligence" in the way we normally understand the term?
Emily M. Bender (00:05:24):
So the marketing phrase artificial intelligence has been applied to everything from chatbots to the algorithms for predicting the folded shape of a protein based off of a sequence of amino acids, to image processing systems, both image generation, the filters where you can put a cat face on yourself, facial recognition technologies, but also sometimes it's just used to zhuzh up what's actually a spreadsheet with some logistic regression in it. It really is a marketing term. And I think it is really important to be specific when we're talking about technology because if you call all of that artificial intelligence and you also think about things that are called artificial intelligence in science fiction, it's pretty easy to believe that we are on our way towards something like C3PO from Star Wars or Lieutenant Commander Data from Star Trek or the malevolent computer from 2001 A Space Odyssey and so on and so on.
(00:06:18):
And the illusion of language technology, which we can talk about some more, paired with the idea that that's the same thing as the image processor, as the thing that scientists are using in protein research and so on, makes it seem like there is one maybe even evolving kind of in quotes "intelligence" that we have to be concerned with as opposed to a bunch of different specific technologies that work to varying degrees. And you asked, is any of this intelligence? I think we have to back up and say, what do we mean by intelligence? Because that, especially when you talk about IQ as these guys do, is very much embedded in eugenics and race science. And it is fundamentally the idea that you can line up all of humanity along one metric called intelligence that is furthermore a way of evaluating how worthy people are. And then with the idea of artificial intelligence, you can say, well, we're also going to be able to put people on the same scale as machines.
(00:07:15):
And you see that scale come out in the discourse when people talk about, I think it was Dario Amodei who says we're going to have a whole country of geniuses in a data center. Genius is being a bot on the IQ scale, but you also see it where people say, well, the previous models were only good as a high school student and now these ones are like PhDs. It's like, okay, why are you making a scale again? Or you'll also get it putting animals and people together in the scale. We've made something that is in quotes, as smart as a cockroach and the next challenge is as smart as a mouse or whatever. And all of this relies on this narrative of there being a scale, that there is one way to use our minds that is the worthy way to do it and people are better or worse at that.
(00:07:59):
And furthermore, the minute you look into the literature on this, you find that there's this idea that this is somehow racially determined as if race were something that's biologically real. So none of this has anything to do with intelligence because intelligence is not a sensible concept, but also do we think of hammers as superhuman at pounding in nails or cars as superhuman at traveling on highways? There's this weird desire to anthropomorphize this technology in particular and talk about it in comparison to people instead of as a set of tools and products that may or may not be effective, functional, a good choice for what we're trying to do. So the phrase artificial intelligence was coined in 1955 by John McCarthy to try to get some money. And in this case, the money was in the form of a research grant to support McCarthy and his buddies to do basically summer camp for nerds in 1956.
(00:08:57):
I'm sure they had a great time and they wanted to work on a collection of different things and they needed to make it sound cool. And there was an existing term around cybernetics that they didn't want to use because there was one dude, Norbert Wiener, who they specifically wanted to exclude, and he was associated with that. So they had to get a new term. And in Valerie Veach's film, there's this wonderful scene where she's found archival footage of McCarthy basically laughing about how he made this up to get some money. So it was a catchphrase for a while and there was an enormous amount of money that came from basically the defense budget in the US and similarly in the UK to try to build artificial intelligence and the promises couldn't be kept. And so then it fell out of fashion again as a term, and we have what people call the AI winter when there wasn't funding for something that was being billed as AI.
(00:09:44):
And then instead of talking about everything as AI, well, some people went to work on what they called computer vision, that's an anthropophizing term, so I prefer image processing and people working on natural language processing and people working on planning systems. A bunch of different things sort of became separate. And yeah, there were probably conferences where there's folks talk to each other, but they wouldn't introduce themselves as AI researchers. And then in the late 2010s, it sort of comes back into vogue and all of this stuff that was previously described as separate gets called AI again.
Nandita Bajaj (00:10:17):
Right. Yes. And I appreciated also what you said about the harmful use of the word intelligence in this. And biologists and ecologists might say intelligence is an evolved adaptation of biological organisms in response to interaction with their environment that also requires internal embodied goals that the organism is seeking to achieve. And as you so wonderfully captured, all beings, all different animals and humans have different types of adaptations that they've evolved to have to respond to their geographies. And so this way of measuring intelligence and how we always look at animals as if an animal can count or if an animal can do the things that humans do, but not even all humans, some humans who, as you say, have been classified on the IQ scale. So this classification just seems to be a never-ending goal of a lot of these tech bros in identifying who's at the very top.
(00:11:27):
And of course they identify themselves as at the very top. So yes, and LLM chatbots are neither evolved nor have their own embodied goals, which we are going to be debunking really throughout the whole episode today.
Emily M. Bender (00:11:42):
Yeah. And I think it's worth pointing out too that intelligence as encapsulated in IQ is about a certain kind of mental activity that we can do, but it is, I don't think the defining characteristic of being a human or maybe even being a biological entity in the world, partially because it is so abstracted away not only from embodied experience, but also social relational experience. And that is part of what's insidious about the LLMs wrapped up in this chatbot design is that they mimic that social relational thing without actually having a subjectivity, the ability to care, the ability to have different goals that we have to negotiate. It is all a fake sort of verisimilitude of this thing that's very, very important to us as people in our interactions with each other. And it is extremely disheartening to see folks turning to these instead of towards each other.
Alan Ware (00:12:37):
Right. And you've written about the problem of using anthropomorphizing language when we talk about so-called AI. And what do you think that language lead us to misunderstand about these systems and how would you recommend we speak differently about technology sold as artificial intelligence?
Emily M. Bender (00:12:54):
Well, first of all, you're doing a great job now. Instead of saying about AI, technology sold as AI. I think that in many cases, the anthropomorphizing language oversells what the technology can do. So when someone says, and this goes back to Drew McDermott, coined this term 'wishful mnemonics', where if you say my computer function is named 'understand', well, you've named it that. That doesn't mean that that's what it's doing. And then you lose track of what the actual functionality is. So we want to be precise in our language. We want to talk about what are we automating? What's the input? What's the output? What's actually happening here? So if we're talking about a particular technology, we can say automatic license plate readers. And reader is already a bit anthropomorphizing. So automatic license plate classification or automatic license plate detection and classification. Let's get really specific.
(00:13:45):
If we want to talk about chatbots, my proposed alternative phrasing there is conversation simulator because it's not actually chatting with you. It's something that simulates a chat. Bot is okay because that shows that it's artificial. But another thing that happens a lot with the anthropomorphizing language is that it puts the software into the agent position. So we've had all kinds of reports for the last few weeks of AI agents going rogue and hacking into whatever. That's not what happened. What happened is that OpenAI and the rest have been developing and running malware and using their malware they have hacked into other specific companies, government institutions and so on. And it's not that OpenAI, for example, was trying to hack Hugging Face, but they were trying to write software that is malware that could break into other people's systems and then they run it and lo and behold, it breaks into someone else's system.
(00:14:42):
So the anthropomorphizing language trick there is to talk about it as if the software has goals, desires, the ability to plan something on its own rather than being a designed, built artifact that somebody designed and built to do a certain thing. So when you anthropomorphize the software, you tend to cut the people who are actually accountable out of the frame.
Alan Ware (00:15:02):
And as you've talked about, artificial intelligence would ideally be called maybe probabilistic automation. And as a computational linguist, I've heard you describe the large data sets that they're drawing for probability of one word, two word, three word, the quick brown fox, you use that example, what is the probability each of those words follows each other? And then the words get weighted up or down given the likelihood that they will be found probabilistically in the next word in the training data. And that's really what these so-called chatbots are doing, right?
Emily M. Bender (00:15:34):
Yeah. So that's what the large language models are doing. The phrase probabilistic automation is sort of one attempt at a cover term for lots of different technology. Technologies sold as AI also works for that so long as they are being sold as AI. But large language models are sort of further development in a long line of technology called language modeling, which is already a bit of a misnomer because what's being modeled is not the whole language, but actually whatever words happen to have been in the corpus of text that was used to build the system. And early on, language models were very, very simple. And by early on, I mean like the 1940s where you basically could take some training corpus or some input corpus and just count up probabilities. This is called a unigram frequency. How frequent are each of the different words? And I think many of your listeners will have experienced working with a unigram language model if they ever had a cell phone with the T9 texting setup.
(00:16:32):
So this is text on nine keys. I don't know how many different languages it was set up for, but in English we have three to four letters on each of the number keys. And the idea is that first with the cell phones to do texting or SMS, you would have to push each number multiple times if you wanted to get to the second or third or fourth letter there. And the idea behind T9 is you push each number once and then it looks up in its database, what are the possible English words that could have been made out of that sequence of letters ranked by how frequent they are in the training corpus. And when people were using T9 in the 2000s, it was still just a unigram language model. Now we had much better language models in the world, but they require more storage space and the cell phones were small devices.
(00:17:15):
And so the best you could have was a unigram language model, which meant that when you typed 'I am on my way' and then the sequence of numbers that should give you home, what came up first was always 'good' because 'good' was more frequent in that original text that they used to build the system. A bigram language model would immediately fix that. If you are ranking all of the words based on just one word before them, 'I am on my way' and you're choosing between 'home' and 'good', it's got to be 'home', not 'good'. Immediately obvious. So far so good. This is useful technology. We use language modeling in this way of like, okay, here's a couple of things that might have been. Which one looks the most like what was in the training corpus as part of automatic transcription, as part of machine translation, as part of spell check and grammar check and so on.
(00:17:59):
That's all fine. Technologies that should be used with care, but they make sense as technologies. What's happened with the large language models is some advances in how the distribution of words are modeled. There was some technical advances in the 2010s. And then also the idea that we should really do with these is use them to repeatedly answer the question, what's a likely next word? And that's where you get synthetic text. And this is what you were describing before, Alan, of like, okay, which word comes next? We're going to look at the probabilities based on what's happened so far. And the problem with that is it's very fluent-looking, very plausible-looking, and nobody has any accountability for it.
Nandita Bajaj (00:18:38):
And meanwhile, we are made to believe that these are agential bots or that there's nobody with intention training these so-called agents with their own biases. So we're made to believe that we are chatting with then a someone, not a trained machine to give us predictable outputs.
Emily M. Bender (00:19:02):
Yeah. And the machines are set up as sort of, if not all knowing, then encompassing all of the knowledge on the internet, which if you just step back for a moment and think, Ew, if you think about all what's out there on the internet, no thank you. I would like to be accessing a much more curated set of documents than everything, as my co-author Alex puts it, that wasn't nailed down on the internet. So they're set up as extremely knowledgeable if you don't stop and think about what that knowledge is, even though what they're doing is actually making paper mache of that input data. It's not that they store knowledge, they store information about which bits of words go where, and they are absolutely designed artifacts. There's choices that go into what is the training data. There's choices that go into all of the further steps. This is what sometimes gets called alignment research by the people who claim to be aligning these systems to human values as if they're in a position to know what human values are. And the accountability problem sort of is at two levels.
(00:20:02):
We make sense of things that we read and see based on who we think is actually publicly committed to having said those words, where if it turns out not to be true, then they either have to own up to being misinformed or to lying. And there's nothing like that in the text that comes out of these systems. And we have somehow failed to also hold the companies accountable for the outputs of their systems, which is a failure that we could fix.
Alan Ware (00:20:27):
Yeah. You've talked about when someone's communicating, speaking with language, there's a speaker who has a reputation and a certain amount of expertise and they're socially situated, whereas these so-called chatbots are just a disembodied sort of answer machine that has no repercussions of reputation, accountability, any of that.
Emily M. Bender (00:20:47):
And that's a huge problem. And we have over and over again school districts saying, Well, we've got to incorporate these in the classroom so that the kids, first of all, they can get this wonderful personalized education, which is really like here, let's give the kids bespoke misinformation and cheat them out of being part of a learner community, cheat them out of more relationships with their instructors and their peers. And also, well, this is the way of the future, so the kids have to know how to use that. It's very heartening that we're starting to see changes in that. So Norway recently put in something of a ban. New York City put in a ban on this technology in K-8 and limited exposure in high school. So it's not everybody who believes that, but it is astonishing to me how many folks jumped on that bandwagon.
Nandita Bajaj (00:21:34):
And hype has so much to do with that, right? You co-host a podcast, a lovely name, Mystery AI Hype Theater 3000. And in the podcast you describe how the AI industry hype relies on heightening the sense of FOMO, the fear of missing out. Could you elaborate on how this manufactured urgency shapes the way businesses, governments, and the public respond to these technologies being sold as AI, as you just described what's happening with educational institutions?
Emily M. Bender (00:22:06):
Yeah. So part of the hype narrative is this is inevitable. We are on the verge of experiencing this brand new technology that's going to change everything. And then the people selling that can say, So you better get on board or you're going to miss out. And I've had people say to me, If you don't use this in your classroom, aren't you afraid of your students getting left behind? To which my answer is usually, Where's everybody going? Even if that were a possible future, I don't like it. I don't have to contribute to it. But if you honestly believed that we were on the verge of having an everything machine that could do any kind of work, that would have all of the answers, that can do it tirelessly, then you would be very ill-advised not to have part of that, not to invest in it, not to make sure that your educational institution is shaped around it, not to investigate how it can help you deliver better healthcare.
(00:23:00):
There's this promise of it's going to be everything, it's going to happen really fast, but then it's like, okay, but what do you mean left behind? If you think about the medical system, it seems to me that the wiser choice is, okay, maybe there's some promise there. There isn't, but I could imagine someone thinking maybe there's some promise there. Let's see how that plays out. Let's keep an eye on it. Not, we have to buy this immediately.
Nandita Bajaj (00:23:24):
And so much of the rhetoric is about if you don't buy it, that means you're not committed to improvement in patient health or you're not committed to students succeeding in schools. There are so many ways in which the language of equity and justice is being co-opted to promote exactly the opposite outcomes - inequitable, unjust, unsustainable. And hype is used because it works, right? It gets more users, it increases the market share, it attracts more investment, higher stock valuations, and there's a playbook to this hype. It's to get users trapped in these products. It creates a dependence, and then the tech company can, as Cory Doctorow describes it, engage in the enshittification process. They increase the fees, they start placing ads, you have kind of premium tiers now you need to sign on to, and then they can sell data to third parties. So it's a very solid strategy that's worked for these guys.
Emily M. Bender (00:24:25):
Yeah. I think the one difference here is the steps of the enshittification process are definitely applying. Here's this thing for free. We're going to give it to you at a loss. I mean, nobody's actually making money on these things yet, although maybe Anthropic has figured out how to get the rest of the tech companies to pay it for use of Claude Code. There's starting to be stories of Amazon going, Oops, we spent too much on Claude. And backing away from their leaderboards where they were tracking to what extent Amazon employees were using these systems or tokenmaxxing. It's like, Yeah, maybe that's actually not fiscally responsible. But in terms of the sort of general chatbot use, they lose money on those, even for the people who are paying 20 bucks a month or whatever, because it's expensive to build and expensive to run. So we're going to flood the market with this, get people attached to it, and then start charging more.
(00:25:15):
And this is one of the lessons that I hope people will hear in school districts, in medical systems, which is if you build your systems around this technology instead of around your people, then what happens the minute that the providers of the systems raise their prices? Not a safe choice. It's not sustainable. The place where I think the enshittification story falls apart a little bit is that Cory Doctorow coined it for things like the way Amazon worked, the way Facebook worked, the way Uber worked, where early on there's real genuine value for the users to get them hooked so they become the thing that could be sold to the other side of the equation. And I know that there are many people who experience working with chatbots as the chatbot being useful, but I really think it's generally a bad idea to do it. So it's sort of like this false usefulness in a way that being able to summon a cab on your phone was an enormous change in people's mobility.
(00:26:15):
And yes, there were harms there, right? We're going to undercut the taxi industry. There's lots of labor exploitation that happens, but that initial interaction as a user I think was more clearly positive than what happens when someone goes to ChatGPT or Claude or Gemini for an answer and therefore is deprived of the opportunity to go find information in its actual context where they can do better sense making.
Alan Ware (00:26:40):
And as we've heard other commentators talk about so-called AI being the frontier of growth for the tech industry, just like they had crypto or NFTs or the metaverse that we were all supposed to be part of by now. So they're always looking for a new growth frontier and that requires hype. And as you write in the book, the predictions of both boom and doom with so-called AI are really two sides of the same hype coin. And we'll talk about the extinction later, but looking at the job apocalypse warnings, how do you respond to those hype boomers and doomers when it comes to job apocalypse talk?
Emily M. Bender (00:27:18):
I mean, I think there too, it really is just the same story with slightly different audiences. What's being sold to companies is, hey, you can replace those pesky people who might have needs. They might need to take breaks in the day and not work more than eight hours a day. They might have opinions about how to get things done. You can replace those pesky people with our shiny automation. We're going to call it artificial intelligence. And furthermore, because of the overall environment, when you say you're doing that, investors love it and your stock price goes up. So that's the positive spin on the story. The negative spin is, oh no, all the people are going to lose their jobs. And I think in the negative spin version, they make it sound like this is just a natural consequence of the existence of this technology in the world where it really doesn't have to be.
(00:28:08):
Any given employer could be making different decisions. They could be saying, you know what? Yes, people will eventually demand more pay, but it's a little bit easier to negotiate with employees than it is with one of these few tech firms where they could very easily be colluding and price fixing. And even if we say, okay, fine, whatever coding product isn't as good as Claude Code, but now Anthropic's made that too expensive, so let's try the other one. Well, that could quickly get expensive too. Your negotiating partner across the table is actually much more powerful in that case. You could have companies that say, yes, maybe we could automate some of the tasks that our junior employees do and therefore not have to hire junior employees. Where do you think senior employees come from?
Nandita Bajaj (00:28:52):
Yes.
Emily M. Bender (00:28:52):
So there's different decisions that could be being made. We are seeing increased gigification. So as companies buy into this hype and say, yes, I do want to save money that way, or yes, I do want to juice my stock price by saying I'm saving money that way, then you do still need people on the other side annotating the training data for the so-called artificial intelligence systems or being the call center workers who are the fallback. There was just reporting that Meta's new Muse thing that's supposed to be able to go take action for you, that's also agentic, sometimes involves routing to a person who calls on your behalf, unbeknownst to the user of the system. The self-driving cars that we do have, so the Waymos and whatnot, have both remote workers. I think for Waymo, it's in Mexico who are on call to help the car out if it gets stuck, number one.
(00:29:44):
Number two, there's specific protocols for law enforcement in the communities where Waymos are for how to deal with these cars if they get stuck because there's no person in there. And then number three, there's all these crews who have to clean the cars and make sure they are charging when they go back to their base, and there is horrific stories about what kind of stuff they find in those cars, what people, the passengers will do, including defecating in the cars because they think it's private. Anyway, all that to say that you still need people to do this work. I think the big economic risk here is not mass unemployment, but mass gigification and much less employment security, much degraded working conditions, and that is absolutely worth fighting against. Remembering that it is not a natural consequence of the existence of this technology in the world, but actually a series of business decisions and regulatory decisions that could be otherwise.
Alan Ware (00:30:38):
Yeah, where concentrated capital in the form of these tech platforms have greater and greater power over people's work and gigification. I think you've also written about the enshittification of jobs. I heard somebody recently who comes into work now in a software company and controls multiple AI agents, and they've basically been turned into a sort of manufacturing foreman of shop floor workers. He doesn't have a lot of autonomous professional latitude as much as babysitting, checking the mistakes of the LLMs, and then being held accountable for the mistakes of the LLMs that are spewing out this enormous amount of content that is also, I wonder if it's creating an attentional arms race where there's just so much content pouring out - language, images, music, everything that probabilistic automation is creating - that some companies are saying don't write emails with any AI program because you're just creating too much work for your coworkers. You have all these bright ideas that we have to go through and analyze and then they put their chatbot on answering that email.
Emily M. Bender (00:31:44):
Yeah, something called work slop, which is basically if your coworker is running these systems instead of doing the thinking themselves, then you are basically having to do the verification step that should have been theirs to do, but it looks kind of polished, right? That's called work slop, and that is one way that the line from our book, which is actually due to Alex, is that AI won't take your job, but it will make your job shittier. It's these kinds of interactions where the people that you are working with where you used to have much more sort of mutual accountability are encouraged to quickly send you the output of these systems. You can get into these loops where it's all going across the email system, but it's one person's LLM usage output being processed by another person using an LLM and back and forth and back and forth, and it's like, where's the thinking?
(00:32:29):
Meanwhile, the CTO of Zoom had this fantasy that we will be able to train our own replacement avatars that could attend Zoom meetings for us, and then eventually you just have avatars talking to avatars and supposedly everyone's off at the beach, which raises several questions. One is, gosh, the beaches are going to be really crowded and who's going to be doing the work of cleaning up after the beach goers that are probably not being gentle on the environment? But also, what do these people think meetings are for?
Nandita Bajaj (00:33:02):
Yes. And that question can be asked really of any job. What is this for? What is education for? What are podcasts for? Now there's podcasts that are being automatically generated based on papers we never gave consent to have converted to podcasts. There's people who more and more sound like real people in terms of the affect. The question is why? Why are we replacing humans with human-like technologies?
Emily M. Bender (00:33:32):
I think it's an intensely dehumanizing move. The move here is that the tech companies are saying the work that you do and the relationships that you have consists only of the words that you say in those contexts. And because we've now built a system that can say words that sound like words you might have said, we can replace you.
Nandita Bajaj (00:33:50):
Yes. And it's basically just to promote this capitalist march toward more growth because as you're saying, all of us are being reduced to our productivity, our efficiency, our ability to produce information, knowledge and stuff for the market, not a relational being that interacts with other beings and their geography and their environment and have evolved for hundreds of millions, if not longer, for other organisms to operate in very slow and intentional timelines, decadal timelines, not millisecond timelines.
Emily M. Bender (00:34:30):
Right, right. Exactly. Human scale and biological scale, and our scale might be different from other creatures on the planet. That's fine. We interact together, but something that struck me recently is one of the selling points is this will free you up. This will free you from the drudgery of blah, blah, blah, blah. That argument is basically the opposite of mindfulness and being present in the moment, and this idea that we should be able to escape what it is to be alive on this planet. I do not enjoy grading papers. That is my least favorite part of my job, but I don't automate it because that is part of my interaction with the students, part of understanding what's working for them, what's not working for them, and also for them getting direct feedback from me also matters. Automating that would break that connection.
Nandita Bajaj (00:35:21):
Totally. Emily, some of the most attention-grabbing warnings about the so-called AI safety are coming from the AI industry itself, including concerns that rogue AI could become misaligned with human values and lead to possible human extinction. How do you view these warnings, particularly coming from an industry that is simultaneously promoting AI as enormously powerful and transformative? I'll note that you are laughing, as are we.
Emily M. Bender (00:35:52):
Yes. It is so absurd. The first level of the answer is this is another case where the boosters and the doomers are singing the same song, just maybe in this case in a different key. The media would have us believe that the doomers and the boosters are the far ends of how you could possibly feel about so-called artificial intelligence, and anybody who's not at one of those far ends has to line up in the middle somewhere. But in fact, these are two sides of the same coin. The boosters say AI is a thing, it's inevitable, it's imminent, and it's going to solve all of our problems, including the climate crisis somehow. The doomers say AI is a thing, it's inevitable, it's imminent, and it's going to kill us all. That kill us all narrative is not coming from the environmental ruin associated with these systems, which actually is an existential risk, but instead these weird thought experiments of, well, if we had a all powerful artificial intelligence and if it didn't like us for some reason, well then what would follow?
(00:36:53):
The first thing I want to say is that those two positions are the same and there's plenty of room outside of that coin for positions that understand this technology for what it is and the harms that are happening. The second thing to know is that this narrative, which seems to be coming from several independent voices, actually has an ideological home and the people who might seem like they're independent are not. There's a well-documented now set of ideologies that Dr. Temnit Gebru and Émile Torres dubbed the TESCREAL bundle of ideologies. So transhumanism, extropianism, singulatarianism, cosmism, rationalism, effective altruism, and long-termism. Those are all the names. They trace back to eugenics and this idea of making better people and eventually making something that transcends the human, that's the transhumanism, which makes it interesting because when you talk about this being embedded in ideologies of human supremacy, it is also sort of a very anti-human movement.
(00:37:55):
But I think you're right that it is a kind of human supremacy. It's this idea that not all people are equal and the ones who are the best on their scales are going to go on to become even better, even better, and that's where the supremacy sits. These stories come out of a philosophical tradition from Oxford, so Nick Bostrom and others, and they are really distressing. They are also intensely silly. They become distressing because people take them seriously. It is laughable, and unfortunately, there is a lot of money behind it because the folks who are proselytizing for these ideologies spent a bunch of time in the last 10, 15 years recruiting at elite universities, especially in the US and the UK. Those people that they recruited have gone on using their elite educations to make a lot of money, especially in Silicon Valley. Now there's a lot of money behind these ideas, which means a lot of access to power.
(00:38:51):
We just had at the UN General Assembly, several tech CEOs addressing the UN Security Council about the dangers of rogue AI. It's like, is this real? If somebody wrote this in a movie script, you'd be like, Nah, okay, a little hard to suspend disbelief there.
Nandita Bajaj (00:39:07):
The very fact that we're talking about this being an existential risk is laughable, right? When I said it's hijacked our collective imagination, there are so many existential risks that are causing real harm to the billions of humans, trillions of animals. There is this completely, as you're saying, nonsensical belief system that's making us draw all the resources and attention away from community-based strategies to address those issues to now every other news headline is about this doomerism or this boosterism. We've had Émile Torres and we've had Adam Becker and Alice Crary talk about these ridiculous narratives, and yet, as you're saying, with so much money behind it, with the Oxford elite University titles behind it, these guys just keep getting more and more. The economist on its front page had effective altruism.
Emily M. Bender (00:40:08):
One of the things about effective altruism is that on the face, it looks appealing, right? We should be altruistic and we should do that in a way that's effective. Where's the problem? The problem is that it takes the spot where community and connection and accountability and relationality should be, and in that spot puts these cold mathematical notions of optimization and then also mathiness, the idea that you can quantify all of this and that furthermore, what we're quantifying over isn't just the people and animals and the rest of our ecosystem that are currently here now, but we should be concerned for the future. Again, okay, so far so good. Concerned for the future makes some sense, but it's not concerned for the future in general, but concerned for this specific future that they have imagined that I'm imagining your previous guest talked about with these simulated minds uploaded to live on computers, circling all the stars in the galaxy. I mean, that whole thing is absurd, and I like to say, have these people never worked in tech support? Who's going to keep those computers running, right?
(00:41:14):
It's clearly science fiction, but the adherents don't see it that way. Then they're in The Economist, they're talking to the UN, they're talking to policymakers, and it's really unfortunate. But I think a really good rule of thumb is to say, where are the relationships? If this is the ideology of how we're going forward, where is community? Where is mutual care? If those things aren't present, then this probably isn't as good as it is selling itself to be.
Nandita Bajaj (00:41:44):
Yes, and you're probably familiar with the work of Peter Singer who became popular with his book in the '70s on animal liberation. He is known as this kind of father of the contemporary animal rights movement, but also he's the one who popularized effective altruism. Today, he is kind of at the helm of this new movement that he's founded called Sentient Futures, believing that the so-called technologies of AI are sentient and are going to lead us into a better way for recognizing the real sentience of animals. Finally, animals can be recognized as biological sentient beings because we can give rights to this machine.
Emily M. Bender (00:42:34):
I mean, wow. The first thing is, why would we ever give rights to machines when we haven't done it for all people and animals and the natural world? That's just so backwards. I think it also shows the way in which the notion of artificial intelligence is in some ways a blank canvas that anybody can paint their own fantasies onto because it's not real technology. Because it starts from these thought experiments that a thought experiment you can use to reason about something, but you can't accept the premises of the thought experiment as real. That's the sort of like you can prove anything from false. If you take a false premise in, you can get to any conclusion you want, and I think that's what's happening here.
Alan Ware (00:43:16):
Yeah, and as you've noted, we spend so much time debating the boom or doom discussion and not enough time looking at the real world harms it's causing right now. I wonder if you could talk about some of the damaging uses of it going on right now.
Emily M. Bender (00:43:32):
Yeah, so there's harms both in the production of these technologies and their use, and then also in what they're displacing. The harms in production include environmental harms, so we're not yet doing enough to slow global warming emissions, and now a lot of that has been reversed with this big rush to build data centers or will be reversed if the data centers get constructed. There's actually wonderful resistance and it's not happening as fast as the companies would like it to, which is great. There's also environmental harms in terms of the minerals and chemicals required to produce the chips that are in the data centers and the water usage and what happens to already water-stressed communities. If you want to talk about existential risks, like the recent catastrophic flooding in the Himalayas driven by glacial collapse from climate change, that's existential on the environmental front and all of the people who have been dying due to excess heat waves and so on. There's real things that are existential to the people involved here.
(00:44:31):
In terms of the other aspects of the production of the technologies, you have, first of all, massive theft of creative work, also normalization of surveillance, the idea that all your data belonged to us. If it's digitized and I can grab it, but it's okay, that's not okay. Then because these are systems that are built up out of absolutely indiscriminate data collection, the output has to be cleaned up. I mean, if you're Elon Musk and you're making Grok, then there's this thing about not cleaning up the output, and then you've created a system that is used to produce child sexual abuse material. There's a harm, but for Anthropic, OpenAI and others, I think Meta's in this boat too, they would like the output of the chatbots to not be obviously bigoted, right? It's still subtly so, but they want to keep the worst of it out, and so they have to hire people to produce additional training data to clean it up, and that means looking at the bad output, sometimes even trying to cause the bad outputs and then rating them.
(00:45:33):
This work is psychologically damaging, it is traumatic, and it is done by people who are precariously employed in places with struggling economies without psychological supports, so there's damage there. Then we get to situations where people are using probabilistic automation in places where it's really not a fit for purpose. Thinking about people being killed in accidents with so-called self-driving cars, thinking about people being denied due process at the border in asylum cases. You get stories of people using machine translation systems to talk to asylum seekers where they aren't even translating from the right language, but the output of the machine translation system is going to look fluent to the person who's looking at it, and so they can just assume that that actually was what the person they were talking to said. You've got many police departments using facial recognition technologies to try to find people who are suspected of crimes, you get false matches, and then you get people who are traumatized on the basis of that false match being arrested in front of their children, the children are also traumatized and so on.
(00:46:40):
There are plenty of actual harms. One recent story CNN reported that sometime this past spring, a US military analyst was investigating the contents of a Chinese vessel, asked some chatbot, we don't know which one, got back a response that indicated nuclear weapons materials on the ship. The US mobilizes this whole attack plan. Planes are in the air and it gets called off at the last minute because somebody thankfully decided to check the source of the information and found that it came out of this chatbot. The harms come from the way we create the systems, both environmental and sort of labor harms, and then the way they get used when they are actually not fit for purpose. Sometimes it's used in the context of things like military intelligence, the so-called AI scribes in doctor's offices are doing a bunch of harm, but then also you get people who form a kind of dependency with these systems and end up being isolated from friends and family and in some cases doing themselves grievous harm.
Alan Ware (00:47:40):
Right. Yeah, we get concerned about, well, the externalization of capitalism and externalizing the costs, and with this use of probabilistic automation, you're externalizing the cost to the Kenyan workers who have to be traumatized by checking the models or due to neoliberal austerity, governments, I worry, are going to be looking for probabilistic automation to simplify and make their services less expensive or the refugees, the asylum seekers. These costs gets externalized under those who are least able to push back against it, and then of course the cost externalized to the natural world.
Emily M. Bender (00:48:18):
Right, and all of us who live in the natural world, which is all of us, despite what the tech bros say. One of the arguments that you'll get is like, well, we have to use the LLM for this. We have to use a chatbot for this because it's better than nothing. And in many cases, it's not actually better than nothing, but the other question to ask is why was the alternative nothing? Why can't we do better by each other?
Nandita Bajaj (00:48:39):
Yeah, and we see that tech that is being sold as AI is being adopted for a lot of these reasons, right, it's better than nothing, at lightning speed by major institutions of society, whether it's business, government, education, media, entertainment, et cetera, and we hear these institutions increasingly being reassured or they're reassuring us that it can be used responsibly or ethically. I hear a lot of rhetoric around women in AI, just AI. There's just a lot of, as I was saying, the language of justice and equity and sustainability being co-opted. What's your response to that?
Emily M. Bender (00:49:21):
I have a piece that I wrote basically saying that there is no such thing as responsible AI, and there's people who try to maintain this uncomfortable, untenable middle ground of, Yes, I see the harms, but we can do this responsibly. Part of that piece says, You don't have to keep carrying water for the tech companies. You can set down that bucket and come step over onto firmer, more comfortable ground. Step one is to basically disaggregate, right? If you're trying to say this is a responsible AI, then you are picking up all of the baggage of everything that's sold as AI. If instead you want to say, I am interested in helping communities build technological systems that actually help those communities with their goals, then sure. The wonderful example of Te Hiku Media in Aotearoa (New Zealand), which is an initiative for doing speech technology, so both speech to text and text to speech for Aotearoa Māori, indigenous language of Aotearoa (New Zealand), and it is built up out of a community transcription project where people donated their voices and their time to produce a collection of text that is spoken language matched with a transcription, and that's an important ingredient for building these systems.
(00:50:38):
At the same time, they have created a license that keeps that data there for the community. All these big tech companies have wanted it and they said, No, this is ours. We're not giving it to you to profit off of. So there are ways to build technology built by and for communities, and you can work on that, but as soon as you start calling it ethical AI, responsible AI, then you're picking up all the baggage from all the other stuff, and why? Set it down. You don't have to.
Nandita Bajaj (00:51:04):
Yes, and there's been a very strategic move from these tech industries to target schools first with a lot of their free software and to get the initial buy-in and to sell it as pedagogically essential. Yeah, I just wanted to see if you have any thoughts on how you view the role of LLMs in the form of chatbots and education. How are you encouraging your students to resist this?
Emily M. Bender (00:51:33):
Definitely, it's a question of resistance. The first thing I want to share is that I got to speak at UNESCO's Digital Learning Week last year, which was a great experience. One of the things that I said there is that these systems have no place in education. I was talking to educators and education leaders from around the world, this involved many, many places where the tech companies are pushing into the education systems, not through contracts, but through philanthropy. We're going to give you this for free. I was encouraging these educators and education leaders to see their education systems not as sites of need, but as sites of wealth, that there's wealth in the sort of learning potential of all of the students, and there's wealth in the connections within a classroom, and there's wealth in the expertise of the teachers. We don't want to give that up just because somebody has said, Here's some shiny tech that we're going to give you for free. One of the things that I said is, If they put pressure on you to make the decision quickly, then the answer should be no.
(00:52:35):
If these philanthropists really are there with the best interest of education at heart, then they should be willing to be patient and let the educators go work through it and come to agreement, which I don't think they should, but in this alternate world, come to agreement that, yes, this is beneficial and not do it in a pressurized fashion. In terms of my own students, I teach in a professional master's program in computational linguistics. That is the field that this technology comes out of, and so my own disciplinary lens is you're going to learn how these things work and why they don't actually do what they're being sold to do. I think that every educator has a disciplinary lens that they can bring to why it doesn't make sense for their discipline. The other thing is that I say to the students, I am not interested in being set up in an adversarial relationship with you where I have to police what you're doing.
(00:53:26):
I would much rather be working collaboratively and cooperatively, and as a result, I am much less concerned about students cheating and much more concerned about what students are getting cheated out of. We ask you to write essays, not to keep the world's supply of essays topped up, but also part of the selling point to students of the large language models in particular is you don't write well enough, this will help you sound professional, this will help you sound educated. That's horrific. As a linguist, I want to say language variation is amazing, it's wonderful, and as an educator, I don't want to see polished first drafts. I want to see messy first drafts because part of my job is to help students find and hone their own voice. There's absolutely no value in my spending time reading something the student didn't write to give them feedback because that's not feedback they can learn from, and I certainly don't want to spend the time reading it.
(00:54:23):
The tech companies would love to be able to mediate every single one of our interactions with each other and also replace many of those interactions, and we don't have to acquiesce to that. We can keep turning towards each other instead.
Alan Ware (00:54:35):
Yeah, those are good arguments for educators beginning to resist in different ways, and you make some arguments in the book for how individuals as collectives can push back. So let's start with individuals. What are some practical steps we can take to resist using so-called AI?
Emily M. Bender (00:54:51):
Just very, very practically, you can say no. You can pick a search engine that doesn't give you an AI overview, or if you're stuck with one that does, there's usually ways to go around it. So for Google, for example, if you add minus AI to the query, and at least historically, if you add a swear word to the query, you don't get the AI summary, AI overview. I don't fully recommend the swear word one because that probably also changes what results you get back. You can not use ChatGPT when you have a question. The next time you have an information need, you can think, Who in my life might help me figure this out? Who would I turn to? One of the possible answers to keep in mind is your local librarian at your local library. That's what librarians are all about. They've been working on information access for millennia, really.
(00:55:36):
There's a long tradition in library information science. When you go to the doctor's office, if they ask if they can record you so the AI scribe can take a note, you can say no. That's hard because you don't want to be a troublesome patient, and I understand that it's not always going to be something you're comfortable doing, but if you are comfortable, you can say no. Similarly, with the so-called facial recognition technologies at US airports, there is this thing for the TSA. You can opt out. It is not required. For some people, it is easier to opt out than others. Those of us who are less surveilled, and I mean white people, we therefore can and should keep exercising that option to keep it open. This sort of what it is. All of these steps are basically every time I say no to the thing, that means I'm saying yes to something else.
(00:56:24):
I'm keeping practices and ways of relating and community connections stronger than they would've been if I turned to the technology instead. We can also make fun of it. The Mystery AI Hype Theater 3000 is what Alex dubbed 'ridicule as praxis', and we can all do that, and that's in the book too. At a slightly more collective level, we have a series of questions in the book that we outline about what you should ask when you are considering adopting automation somewhere. What's being automated? What's the output? Does that even make sense? If it makes sense, how is it evaluated? Do the results of that evaluation actually map onto what we're doing here, or are we talking about a very different use case where it needs to be reevaluated? So language technology that is evaluated for one population that speaks in one way is not going to work equally well for another population, so we might need to redo the evaluation.
(00:57:14):
Why are we automating it? Who's benefiting? If we automate it to speed things up, are we actually allocating enough resources to handle the cases where it doesn't work, or are we going to make people stand in line because we've sped things up with automation? If it works as intended, who's going to be harmed? If it fails, who's going to be harmed? These are things that we can always think through, and it's worth keeping in mind that policy is made at many different levels. You've got international, national governments, state governments, city governments, school districts, companies, organizations. So keeping an eye on that policy and participating where we have power to participate. These are all things that we can do from a more individual point of view. And then collectively, I think it's about organized resistance. So in the workplace, strengthening our labor unions where we have them, building them where we don't, so that there's a strong voice to say, No, you can't lay us off and try to automate us, or, No, we're not going to accept that work slop.
(00:58:11):
We need good working conditions. But there's also organizing that's being done fantastically around opposing data centers, and these are things that we can contribute to as individuals, but we cannot do on our own. We have to do it in an organized fashion.
Alan Ware (00:58:23):
Right. I think I heard you mention it gives people a focal point, a physical space to resist AI in a way that nothing else does. I'm not sure if it's you that made the analogy to the Vietnam computer labs on campuses that were being used.
Emily M. Bender (00:58:38):
You're thinking of a piece by Sonja Drimmer, who was basically saying that in the protests that were happening in the Vietnam War era, the computer labs on campuses were great focal points, and she was saying, We don't have that now. I said, Yes, we do. It's just not on campus anymore, and she very much agreed.
Alan Ware (00:58:55):
Yeah, and the fact that it's bipartisan and that the methods big tech is used by having public policymakers sign these non-disclosure agreements, to stay mum about it for months before they spring it on the public.
Emily M. Bender (00:59:09):
That is one of the most bananas details to me that any public servant would say, I agree to keep this secret from the public that I serve. This does not make sense.
Nandita Bajaj (00:59:19):
Well, Emily, this was such an enlightening episode. Thank you so much for the work that you have done in offering all of us these strategies to look past these parlor tricks that are being offered by these tech bros and the excellent examples you've given of how we might individually and collectively resist these really harmful technologies and ideas. We are so grateful for your time today. Thank you so much.
Alan Ware (00:59:49):
Thanks, Emily.
Emily M. Bender (00:59:50):
Thank you. It's been a really interesting conversation.
Alan Ware (00:59:53):
That's all for this edition of OVERSHOOT. Visit populationbalance.org to learn more. To share feedback or recommendations write to us using the contact form on our site or by emailing us at podcast@populationbalance.org. If you enjoyed this podcast, please rate us on your favorite podcast platform and share it widely. We couldn't do this work without the support of listeners like you and hope that you will consider a one-time or recurring donation.
Nandita Bajaj (01:00:21):
Until next time, I'm Nandita Bajaj, thanking you for your interest in our work and for helping to advance our vision of shrinking toward abundance.
