Garrison Lovely wrote the cover story for Jacobin magazine's special issue on AI, which explained how leftists should think about the risks posed by the new technologies. He also recently wrote for the New York Times about AI safety and has written for Current Affairs about psychedelic drugs and McKinsey. Garrison joins today to discuss the real harms that AI could do, why Big Tech can't be trusted to self-regulate, and how we can avoid a nightmarish future.

Nathan J. Robinson

You have written previously about numerous topics for us. You’ve written about psychedelics and your experiences at McKinsey, but you've written a lot recently about artificial intelligence. Why do you think the subject of AI is of such importance that you're willing to devote so many hours of research and writing to cover it?

Garrison Lovely

I have been interested in AI since college. I just think it's really interesting. There are many intersections with other things. There are the technical questions of consciousness, intelligence, and thinking. And then there's this question: If you built systems that could influence the world in dramatic ways, what values would they have? David Chalmers recapitulates all the questions of political philosophy into one topic, and I always thought it was interesting. Then, once ChatGPT came out, it became very hot to write about. I held off on writing about it personally because I don't have a technical background. I figured other people could do it better. But once I saw the average quality of a lot of the journalism on it—a mix of either rehashed press releases for companies or dismissing the technology itself out of hand—I figured I could do a bit better than that, hopefully.

I think it's only going to grow in importance as a topic. I think even if you are dismissive or skeptical of the idea of AI systems that rival human intelligence, the belief that that could happen is already motivating lots of decision-making, and I think that we're now entering a new phase of AI competition where the last few years has been marked by inter-firm competition, like in the marketplace between like OpenAI and Google and Anthropic. And now we're entering into this Cold War-like “clash of civilizations” narrative. The United States is investing tens of billions of dollars in chip manufacturing and doing export controls to make sure China doesn't have access to certain semiconductors. This framing, I think, is potentially the most dangerous idea of the 21st century.

Robinson

Let's talk a little bit more about what we're talking about with AI. And you said there that you think that a lot of the journalism has been either corporate press releases or dismissive of the technology. So to take the technology seriously, what are the kinds of breakthroughs that you see having occurred, and what are the kinds of things that we are moving towards?

Lovely

I think with AI, there's a lot of controversy over what this word even means. Some people don't like it as a term, especially on the left, and they think it's marketing or whatever. So when I talk about AI, I'm typically talking about systems that are powered by deep learning, which is a multilayered neural network where you feed a bunch of data into an algorithm and then the system learns from that data and produces outputs that better reflect the data over time. And in some ways, all the recent developments in AI are actually related to each other.

There was this revolution in deep learning which started in 2012, where people realized that we had enough computing power and the algorithms to make use of huge amounts of data that was not previously possible. Neural networks and deep learning were seen as a dead end before that. And then in 2012, there was this competition, ImageNet, where a deep learning-powered system created by Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky, called AlexNet, won. It was kind of a coming out moment for this approach to AI, and since then, there's just been a massive increase in the amount of computing used—literally, the number of chips and how many operations they can do, the amount of data fed into these systems, and also algorithmic improvements as well. And so all of that has culminated in dramatically increased levels of performance. The time it takes for AI systems to go from way worse than human performance to matching or exceeding human performance in things like image recognition, image generation, handwriting, and reading—a huge range of things—has just collapsed over time. This has led to ChatGPT being the big famous thing, which was both a culmination of this trend and a big surprise to many of the people working within the industry.

And things are moving faster, actually, than many of the experts predicted. You can look at when people thought AI systems would achieve gold medal performance at the International Math Olympiads, or how well they would do on certain benchmarks, and they consistently underpredicted how fast things were moving. The thing that is most interesting to me, and the thing that, if it continues, has massive repercussions for the entire world, is the progression of capabilities.

Robinson

Or it accelerates?

Lovely

Yes. And the thing that a lot of people worry about, or are excited about, is the idea of, what if you make an AI system that's capable of automating chunks of AI research and development? There's already evidence that this is happening. Whether that's happening at a fully autonomous level versus bits and pieces matters, but automating scientific and technological research and development is a thing where, if you do that, you could imagine a very large increase in capabilities very quickly.

Robinson

We're talking about one thing, but also many things. The broad category of stuff that you're talking about are things powered by deep learning, but then the applications are sort of infinite, and the power that we're seeing is immense across many different domains. I started experimenting with image generators, and I became totally fascinated by how, first, they improved very quickly. The early generation of DALL-E was making just ugly crap, and then they became better and better very quickly. And then I started experimenting with music generators, which are totally fascinating. They are just immensely powerful at creating things that are indistinguishable from, I don't know, music recorded in the 1920s, but with the lyrics that you've written and then doing custom modifications. But a lot of people will have seen deepfakes and how difficult it is to identify what's real and what's fake. So people may have seen the many different applications of what is broadly lumped together as AI.

ChatGPT can obviously do many different things. You can have a debate with it and ask it to take up a certain position. You can ask it pretty much anything in the world, and it will come up with a fairly decent answer. What are the applications that you think that are most concerning or alarming? Obviously, having ChatGPT come up with interesting recipes or making goofy pictures of Donald Trump is not terribly concerning. And you could say, the better it gets at that doesn't really matter to me. Where do you think our attention should be?

Lovely

I guess it depends on the timescale. There's a lot of attention on hallucination: the problem of these models just making shit up, basically. And there was a recent news story about ChatGPT being used in hospitals and hallucinating patient notes or conversation notes, and that's a pretty bad outcome. So there are certain domains that these systems are just not ready for, and there might just be really difficult problems in the underlying technology with the way next token prediction works that you'll just never quite be able to solve or have be perfectly accurate in the way that a lookup table kind of system would work. 

Robinson

You've used a little bit of jargon there—next token and lookup table.

Lovely

The basic way that ChatGPT and these other language models work is, if you give them a string of words, they'll try to predict the next word in the sequence. And there are modifications on top of that, but that's the fundamental kind of thing. And you asked about developments. The fact that that works as well as it does is remarkable. It solved this problem where in the past, to train a big AI system, you had to feed it a lot of labeled data. So, here's a picture of a cat, and it says “cat” under it. And most data is not labeled. It's just unstructured text and images and videos and whatever on the internet. And by coming up with a way to consume the entire internet and do something useful with it through next word prediction, language models were able to massively increase the amount of data that they could use and actually learn from.

And then a lookup table: the way a classic database works is you put in search terms, or use a language to look up specific pieces of information, and that's just very reliable. It finds the information, and then it presents it to you. And Google Search kind of worked in a way like this. And then when they moved to AI-generated results, there are all these issues. I think that's just not a good use case right now for this type of thing. I don't know how that will be in the future. I'm not going to say this will never be useful for that type of use case. The stuff that I'd be worried about, and am worried about, is lethal autonomous weapons such as killer robots. There's already been reported uses of these in Ukraine. Israel has used a program called Lavender to automatically generate targets, and this has been associated with massive increases in civilian deaths.

Granted, with the context of the current war, I think there's a good chance that would have happened no matter which system they used. I think there's going to be a competitive pressure between countries to automate more and more of their weapons. And people talk about having a human in the loop, but there's going to be strong pressure to remove that human from the loop when your adversary is doing it or when you think they will do it. So that's a thing that I think will continue to be more of an issue. It has implications for stable authoritarian regimes, where one of the ways in which these regimes fall is when soldiers refuse to fire on civilians, but if your 'soldiers' are perfectly loyal killer robots, that's a pretty different situation.

And then I think a big thing is the automation of software engineering. You can start to get into this feedback loop of things moving a lot faster. It also leads to potentially massive security vulnerabilities. So if ChatGPT or Gemini or Claude are generating huge percentages of code, which is starting to happen, there could be systematic errors in that code that people are missing that introduce vulnerabilities that could apply across many systems. And then, for the more sci-fi stuff down the line, if there's actually a super smart AI system that's writing a huge part of our programs and people are not really reviewing or understanding the code sufficiently, there could be back doors installed into that code. And then with bioweapon development, these systems are starting to show signs of being able to help human experts in developing bioweapons.

Robinson

For example, one of the things that keeps humanity a little bit safe from complete catastrophe is the fact that for an individual—a malicious bad actor or psychopath—there is a very high barrier to, in our bedroom or in our garage, developing a virus that causes a pandemic on the level of COVID or even worse, like the Black Plague. Thank god, that can't happen. Thank god, it's very difficult to do that. Because as we know from the COVID pandemic, the moment it's unleashed, it's very difficult to stop. It's not easy or thought possible right now for you or me in our garage to develop nuclear weapons. It's good that there are many things that individual bad actors can't do. But as AI develops further and puts an immense amount of intelligence capability in the hands of every single person who has it—if I have the equivalent of the brains of a thousand scientists, or 100,000 scientists, working for me on my computer, that rather changes the distribution of the capacity to do terrible things.

Lovely 

I think that's right. And I think AI is, in some ways, the ultimate dual-use technology, where the things that it's good at are tied up inextricably in the things that it's bad at. And if you think the current approach to building AI systems is fundamentally flawed and will not get you to very capable systems or automated science, then you're worried about a different set of problems, maybe more like bias or surveillance or labor automation in some more narrow domains. And those are all real issues, but the medium to long-term issues are, as we discussed, whether AI continues to progress and changes the economics of various actions and makes it so that a million people could have the skills to develop a new bioweapon with GPT-6 or something. With nuclear, the materials are the main bottleneck there, so I'm less worried about it.

Robinson

Thank god. 

Lovely

We're very lucky that it's as hard as it is.

Robinson

Because if there was different distribution of materials on Earth—it's interesting how much depends on things that are arbitrary or fragile.

Lovely

Totally. We're very lucky in a lot of ways. And then, in terms of other things to be looking out for, there's a new study about labor automation with AI models, just at GPT-4 level. There's been a lot of research in the past that's been overhyped about how jobs would be automated away. I think that that's largely been something that hasn't panned out, but banking on that indefinitely is wrong. At some point, there will be large-scale automation, and I think that our society is not set up for that. Our welfare state is not set up for that. And the interesting thing is, these companies are explicitly trying to build something that would make humanity obsolete. OpenAI's goal is to do something that can replace most jobs. That's the implication of their stated, explicit goal. Whether they can do that or not, it's still a very bold thing to go out and try to do. You're trying to do what most people don't want you to actually do if you ask them.

Robinson

It's very bizarre, actually. You quote, in your Jacobin piece and elsewhere, the kind of techno-optimists whose stance on this is that it's going to usher in an age of plenty and solve basically all human problems. But it will solve them by doing something that's quite alarming, which is to say, to disrupt the entire fabric of every social and economic structure that we have.

Lovely

Yes, the world that the people building this technology want looks radically different from the world that we inhabit. And the world we live in now is radically different from the world before the Industrial Revolution. And in many ways, it's much better. So, they'll be like, look, if we can have material abundance, make it such that everybody in the world lives at a level that a millionaire lives today, diseases are eradicated, and green energy is abundant, we can live in this fully automated luxury communism utopia—that’s sort of the lower bound of the dream for a lot of the people running these AI companies. And then the upper bound is something like from the Culture series, this series of sci-fi books by Ian Banks, where it's very hard transhumanist-like utopia, a kind of libertarian socialist world where people’s brains can produce drugs that make amazing experiences and people can transition their gender just by thinking about it. It's a pretty radical thing, but humans are not really in control in that world. They're sort of like pets. I don't know if I want to be a pet.

Robinson

One of the incredible things that you point out in pieces that you've written is that a lot of people working on this think it's going to lead to catastrophe. In fact, there is an incredible quote you have from Sam Altman himself of OpenAI, who said, before he founded the company, “AI will probably most likely lead to the end of the world, but in the meantime, there will be great companies.” That's stunning. There's so much to analyze about that quote.

Lovely

It's amazing. It's said in this jokey way, and people laugh in the video, but he has said something else like that in other places. I think it really does reflect his views, or at least did at that time. An interesting thing is that he no longer speaks in that way. And there's this narrative, especially on the left, that these guys don't actually believe in the extinction risk stuff, and they're just saying that to hype up their products or to defer regulation, and I think that there's some truth to this. There are specific examples where this narrative was being used in that way. But I think it's notable that Sam, as he's gotten closer to power and profitability, has been speaking a lot less about the downside risks. His most recent writing on AI is this short blog post, and the only downside risk he talks about is large-scale unemployment. On a podcast within the last year, he said, I don't think we're going to go extinct. Maybe his views have actually changed. But it's also convenient, now that he's the guy closest to bringing this into the world, that he thinks this way or speaks this way now.

Robinson 

Well, one of the things you write about as a leftist analyst of AI is the structures of the institutions that are developing the technology and the effect that the profit motive and competition have on how the technology is developed. You talk a lot about how you know that when it's for-profit companies doing this, there are bad incentives, as leftists know what the profit motive does. In the Milton Friedman story of the world, the pursuit of self-interest creates the best possible world because no one would buy your product if it weren't beneficial to them. We know as leftists that, actually, there are some bad incentives.

One of those incentives that you write about is the externalizing of risk and cost. That is to say, the company wants to get all the benefit and push the costs of whatever it does—if it's a polluting company, if it's a fossil fuel company—onto everyone else, and they get all the benefits. You also have a quote in one of your pieces where someone said, well, if it were harmful, we wouldn't build it, which is another kind of classic defense of capitalism: if a company’s products did harm, they wouldn't build them. So, can you talk about the role of the capitalist institution in this?

Lovely

I think that you summarized it nicely. The basic idea is, it is profitable to build things that are powerful and agentic, and safety costs money and time. And there are some marketing benefits. You want to have your model not say something racist, or be obviously misbehaving in some way, and so these companies will invest in some amount of trust and safety in these models. But to do the actual level of safety research that you would want to do if you took these risks seriously would require devoting a huge amount of your computing, engineering, and scientific resources toward the problem.

A great example of this is OpenAI. Last July—a bit over a year ago—they set up this team called Superalignment. The premise was, we have to figure out how to align a super-intelligent AI system to human values. This is an unsolved problem. We don't know how to do it, and we have to do it within four years. We're going to dedicate 20 percent of our compute, as of right now, to this problem. A year later, the team was disbanded, and both of the people running the team left. One of them voted to fire Sam Altman and then undid his vote. The other one complained about safety taking a back seat to products and commercial incentives on his way out the door.

And then there was a report that they never even got the 20 percent of compute that they were promised in the first place. And this was only going to get worse as the race heated up. I spoke with somebody at one of these companies who was in a senior position, and they were like, I think the voluntary commitments that these companies are making to do safety practices work well enough until you get really close to human-level AI, and then once you get close to that, things break down and fall apart because the incentive to be first is just so strong. I will just say, though, that I think capitalist competition has been the driver of a lot of the dangerous practices recently, but I think the long-term or the meta driver is like just competition in general. And so that can be between companies but also between countries. And I think it just gets even scarier once you go to the nation-state level.

Robinson

The risk of war and the role of these new technologies in war is the thing that perhaps frightens me in the near term the most. Because, first off, we've seen World War One, World War Two. In the years leading up to World War One, nobody could have conceived that a war on that scale could have happened. And what happened was that countries tried to destroy each other in a conflagration that, given the technology of the time, was a staggering horror.

And World War Two, of course, was much worse, in part because the technology was worse. So if countries decide to devote themselves to killing as many of each other as possible, the technology you have determines the scale of how that's going to go. And obviously, in the era of nuclear weapons, the capacity for mass murder is so great.

But I do believe that nuclear weapons create a powerful disincentive towards war between nuclear powers because we all understand that it would lead to ultimate self-destruction. But I also don't think that that is guaranteed, and the level of uncertainty that AI brings into it—just when you think about the idea of, say, killer robots, it sounds silly and like science fiction, but I saw that humanoid robot that Elon Musk demonstrated recently. I was very alarmed by that. My first thought is, what happens when they give that thing a weapon, which they will, and what happens when you give it the instruction to exterminate the village? It's going to go exterminate the village. No moral qualms, no human thought. Especially when you introduce the bioweapon element. So, as you say, it's very unfortunate that the era of AI is also coinciding with the era of increased tension between the great powers of the world.

Lovely  

And to some extent, it's motivating that tension. I think there are two big pieces that you touched on there. One is that AI, and technology more broadly, can increase the destructive potential of weapons. And I've seen at least one estimate that found, based on some assumptions, that you could have AI-powered drones that are more destructive per dollar than nuclear weapons and more targeted. And if you think that you can have these things and not start World War Three, you might be more tempted to use them, and that's very risky.

That leads to the second point, which is that this technology can be profoundly destabilizing, both in the perception that your opponent is about to get something that will give them a decisive strategic and indefinite advantage on the battlefield, or geopolitically or economically, or just if it actually does what the boosters want it to do. If it does automate science and technology, and then all other parts of society, and you get 1,000 percent economic growth in a year or something, the world would just change overnight, and that will have tons of repercussions and consequences and could be extremely destabilizing to nation-states and to what it even means to be a human. I don't think that the world we live in is prepared for that level of change. And then this idea that whoever builds Artificial General Intelligence first will dominate the future indefinitely. It makes the stakes existential in a way that that is extremely dangerous.

Robinson  

It seems to me that even the skeptics of the capacities of artificial intelligence have to concede that we face enormous risks. Everyone sees the deepfakes. Everyone sees the capacity to generate lies at a massive scale. So, as you mentioned, the skeptics now talk about things like bias, surveillance, automation, and such. Really, we're talking about whether this technology will cause massive disruption or colossal risks. Massive risk or colossal risk, and we don't know the answer to that. It's unknown, and that's one of the things that's quite frightening about it.

I understand now better than I did, I think, before ChatGPT came out why someone like you wants to pay so much attention to the potential downsides of the implementation of this new technology. I see that whether you're skeptical or not of the ultimate upper bound of the capacities, wherever you fall in that, this is a very serious issue. And so I want to conclude here by talking about what you wrote in the New York Times and elsewhere about what we need in terms of a regulatory political response and what we're not getting. When there is a situation where bad capitalist incentives produce an incredible risk, that's when the state has to step in. The state is basically the only actor, other than maybe a well-organized and militant union, that can control corporate behavior. So tell us a little bit more about the missing policy response.

Lovely 

AI and software more broadly have been left largely unregulated in the United States. And this is something of a historical accident. Just like when computing came on the scene, we had just moved away from an industry based regulatory approach, but it's taken as a given and necessary by many people in Silicon Valley that this is just how it should be. And so the way AI is governed in the United States is through a series of voluntary commitments in which companies promise to do certain safety testing, cybersecurity practices, and auditing and agree to share their data and their models for pre-release testing with government agencies. But there are no punishments—there are no teeth that force them to actually do this, and so that's an issue. You're just taking it on the good word of these companies. And then people who are skeptics of regulation will say, well, we don't know how to regulate this technology, it's just so new. Generative AI, ChatGPT—it's two years old, at least, at the scale of what we're talking about. But AI, the technology, is much older, and I think there are regulatory frameworks that you can use that are flexible and allow the industry to come up with best practices along with government standard-setting organizations. And you say you have to follow these best practices, and if you don't, you're on the hook for damage that your model does. That was the basis of SB 1047 in California.

The thing I wrote about in the New York Times was the case for whistleblower protections for AI employees. Employees at these companies will have firsthand knowledge of whether the company is doing something risky. Right now, they can do safety testing and then find that the model is dangerous, and then they can do different safety testing until they get to results that show the model to be safe. And if an employee knew about that and wanted to tell the public or tell the government, they would not be legally protected. They would be risking their careers. They would be risking violating their nondisclosure agreement by doing this. And so I think that we need expanded protections specifically for employees at these companies and for anybody who has access to this information so that they can come forward and notify people. And ultimately, I think you need to have domestic regulations that ladder up into a bilateral agreement between the U.S. and China, the two leaders in this technology. I think you need to do all of that before you can have any hope of some kind of robust international agreement governing how this technology is developed and deployed.

This is utopian in some sense. It's very hard to see how this happens, especially in the current geopolitical climate. But I think the alternative is an insane race to build more powerful and autonomous technology and deploy it in more domains without prioritizing safety—the other guy is not going to do it as safely, and so we need to rush to get there. It looks just like the missile gap and nuclear arms buildup all over again.

Robinson  

Well, I’m going to disagree with you on the characterization of utopian. I think we should not characterize it that way because during the height of the Cold War, we were capable of making arms control agreements with the Soviet Union. There was a mutual understanding that we can't have an arms race spiral completely out of control. Even Ronald Reagan understood that to a certain point.

Lovely  

Yes, I think the main difference is that there was a widespread belief, at least by the end of the Cold War, that a nuclear war could not be fought and could never be won. I think Reagan said that. And the world, after an exchange between the Soviet Union and the United States, would be just unrecognizable and horrible, something that nobody wants. But with AI, there are two broad buckets of risk that people worry about with very advanced systems. One is misuse risk, which is when somebody uses it to do something I don't want. This will depend on your perspective. And so if you're a U.S. hawk, you're worried about the CCP building superintelligence and using it to take over the world and make it all communist or something. That would be a misuse risk. And then if you're China, maybe you're worried about the United States using superintelligence to fully be the hegemon of the entire world and force regime change. And that's a thing that people are calling for in the United States, so that's pretty scary.

But then there's this second bucket of risk, which is loss of control. And this is the idea that if you build a system that is truly more capable than humans, you cannot guarantee that you will control that system, and it may be that nobody wins, that we are like an anthill in the way of a hydroelectric dam project—maybe the AI doesn't care about us being there or not, and we just get swept away when the dam is built anyway. I'm borrowing that from Stephen Hawking. That's a real concern, and I don't think anybody has satisfactorily addressed it. And I don't think there's some technical solution where, if you do this thing, an arbitrarily powerful AI system will do exactly what you want. Nobody serious thinks that's happened, and yet we're trying to build these things. And so that kind of framing—that we are all in this boat together, that this is an issue of global security, not national security—I think that framing and that understanding needs to permeate the power centers of the world for us to have a chance of things going well.

Robinson  

You mentioned earlier the piece of California legislation that was ultimately vetoed by Gavin Newsom. And just to conclude here, I think that the thing that causes it to seem utopian that humanity could address this problem in a rational way is, in part, the fact that people like the governor of California, the state where a lot of this stuff is being developed, doesn't understand the technology, doesn’t take seriously the risks around it, and a pretty blasé about it. And so that's why I appreciate the work that you're doing. Obviously, step one is to try to get the people who are in a position of power, and thus responsibility, to understand that we are dealing with something that is quite real and dangerous. Coming with a path towards the safe continuation of its development is pretty urgent. 

Lovely  

Yes, I think if Gavin Newsom believed that there was a high chance, or even a decent chance, that AI systems would cause a catastrophe of the sort that the bill was aiming to prevent before he runs for president, I think he would have maybe not vetoed it. And so there is this real gap in what people believe. This is inherently speculative, and we don't know for sure what these systems are capable of and what they will be capable of in the future. The major problem when you're addressing any kind of risk is that, if you do a good job of addressing the risk, you might just see the risk never materialize, and then people will be like, this was all unnecessary. To quote somebody, you can't do a press conference in front of a pandemic that never happened. And so I think that the way I imagine some kind of robust AI regulations happening is there's some smaller scale disaster—there's some kind of wake-up call, where an AI system behaving autonomously or used by a bad actor causes a lot of harm, and then people actually act. And in that world, is the regulatory response going to be more thoughtful and well-reasoned, or is it just going to be a reactive and a band-aid solution? But I think that's the world that we're trending towards.

Robinson  

Well, it strikes me that there are few downsides to taking safety seriously. Because, as you say, if the risks never materialize, great.

Lovely

The kind of trump card is the China thing. The argument that you're seeing from Sam Altman and Dario Amodei, who leads Anthropic, is that this is about democracy versus autocracy, and we have to both invest in our domestic capabilities to produce enormous numbers of chips and more energy and data centers and not spend too much time on safety. We got to race forward as fast as possible, and the bigger the gap between us and them is, the more we can afford to take safety seriously. But once you set up this whole system to race forward as fast as possible and you don't actually have insight, you're not going to have perfect clarity into what your rivals are doing and what they're capable of. And so, we're just on the cusp of building this thing that could be the dominant super weapon forever, and we're going to slow down now and take safety really seriously—that's hard to imagine in the way we've set things up.

Transcript edited by Patrick Farnsworth.