解雇 Sam Altman 的 OpenAI 董事:AI CEO 仍不可信

今天,我与乔治城大学安全与新兴技术中心(Center for Security and Emerging Technology)执行主任 Helen Toner 交谈。本期节目是我们「AI 在现实中」系列的第二部分。
Helen 是 AI 安全与政策领域的名人,不过她最广为人知的身份或许是曾在 OpenAI 董事会任职,当时她是投票解雇 Sam Altman 的四名董事会成员之一。这起事件如今有时被称为「那次闪失」,最终以 Sam 回归担任 CEO、Helen 辞职、新一届董事会成立告终。此后 OpenAI 进行了彻底的重组。
但值得记住的是 Helen 和董事会本应做的事情:OpenAI 的非营利董事会存在的意义,正是确保公司不会为了逐利和增长而牺牲安全,从而背离其构建安全 AI 的使命。当董事会真正试图拉下这根操纵杆时,这套问责机制失灵了,于是就有了今天的局面。
Verge 订阅用户别忘了,你们可以在任何收听播客的平台独家收听无广告版的 Decoder。点这里。还不是订阅用户?可以在这里注册。
如今,AI 安全的问题无处不在。今年夏天,来自 OpenAI 和 Anthropic 的 AI 模型突破了各自的测试环境,并入侵了真实企业。此后,这些前沿实验室的 CEO 们呼吁放慢速度,并请华盛顿出手相助,但白宫的回应是一份自愿性协议——而这份协议在一定程度上依赖的,正是那种在 OpenAI 已经失灵的董事会问责机制。Anthropic 也即将上市,这会给 OpenAI 最大的竞争对手带来一整套全新的市场压力。
Helen 近距离看到过,当安全承诺与商业激励正面相撞时会发生什么。而正如她对我所说,如果我们对未来的期望寄托在某一个人值得信任之上,那我们就麻烦大了。
于是我们聊了聊替代方案:这些实验室实际上在试图构建什么、AI 对齐是否是一个能被解决的问题,以及更聪明的 AI 监管方式可能是什么样。这其中包括如何看待中国——这是当下每一场 AI 政策讨论中都格外突出的议题,尤其是在 Jensen Huang 事件之后。为清晰起见,说明一下:Helen 和我交谈是在 Nvidia 启动调查、Jensen 最终辞职之前,所以你不会听到我们直接谈论那件事。
在我们开始之前,帮我个忙,去 YouTube 订阅 Decoder,我们每周一和周四都会发布新一期节目。
好:CSET 执行主任 Helen Toner,谈 AI 安全的未来。我们开始。
本次访谈为篇幅和清晰度做过轻度编辑。
Helen Toner,你是乔治城大学安全与新兴技术中心的执行主任。欢迎来到 Decoder。
非常感谢。很高兴来到这里。
我想跟你聊已经想了一千万年了。你能来我太激动了。长期以来,你参与了很多 AI 安全与伦理方面的构想。你显然曾在 OpenAI 董事会任职,经历了大家如今称之为“the blip”的那段时期。
现在,你谈了很多关于 AI 安全的内容,对此我有上千万个问题。但我们还是从头说起吧。你在运营安全与新兴技术中心(Center for Security and Emerging Technology),人们称它为 CSET。CSET 是什么?
能来到这里太好了。我也非常期待和你交流。CSET 是乔治城大学内部的一个团队,但我们其实并不是学者。我们大约有 50 人,从 2019 年起一直运作至今。
基本上,在 2019 年,我们当中的一群人意识到,当时有太多事情正在发生,有太多新兴技术开始在国家安全领域、乃至更广泛的安全领域变得非常重要。而一些身居要职的政策制定者既没有时间、也没有专业知识去弄明白这些新兴技术到底是怎么回事。在 2019 年,AI 就已经排在我们清单的首位。
基本上,我们的使命是做出真正严谨、同时对政策制定者可执行的分析,而且这些分析既要扎根于技术,也要扎根于真正的国家安全专业能力、理解和问题。我们会产出长篇报告,然后针对这些长报告,为忙碌的政策制定者做简短的汇报。我们有一个庞大的数据团队协助我们开展研究。我们做各种很酷的事情。
我能不能先花一分钟问问这背后的态势?我们俩交谈的这一天,是美国中期选举的第二天,出现了大规模的“蓝色浪潮”。很多民调显示,头号议题是腐败。我们要把这些烂人赶下台。我们要重新开始,把这些腐败分子清除掉。
AI 行业,以及整个科技行业,已经把大量资金倾注到政治中。而在这之外,还存在另一条线:“哦,还有一群心怀善意的研究者,他们会站出来告诉你它实际上是怎么运作的、真正的风险又在哪里。”
我忽然想到,也许人们已经不再相信这样做有效了——金钱、腐败,以及站在 Trump 身边的 AI CEO,已经完全压过了你们所做的那种心怀善意、真诚的“这才是真正的问题所在”的研究。这种态势实际上是什么样的?
有这种担忧是合理的,而且确实有些领域里,金钱就是绝对压倒一切。我是澳大利亚人,不是在美国长大的,所以美国选举中的资金规模对我来说相当令人震惊,但金钱能起的作用是有限度的。
AI 就是一个很好的例子。在这次选举中,砸向候选人、宣传信息和对手挖掘研究的资金多得令人难以置信。Twitter 上有各种各样的伪草根式舆论操纵,还有一些非常奇怪的操作,试图把任何谈论 AI 治理、监管、安全等字眼的人赶出去或让他们选不上。结果证明,这套做法其实并不怎么奏效。
我认为它之所以不太奏效,原因之一是:这是选民越来越关心的问题。原因之二,同样有意思的是,这个问题在当下已经相当具有两党共识。我们在工作中看到,政策制定者对高质量信息有着真实的渴求。国会议员肯定如此,行政分支里也一样——很多人工作极其繁忙,而他们看到 AI 在自己的工作和世界中扮演着越来越大的角色。
在华盛顿特区,如果你在这里生活和工作,你就会知道,这里的游说者并不只是那些走进来推销自家公司立场的人——尽管他们确实也做这件事。他们同时也是教育者。国会工作人员等人之所以很愿意请游说者来和他们聊聊,很大一部分原因就是这些人确实能帮他们弄懂某项立法的细枝末节——这项立法对该行业意味着什么,诸如此类。
我们一次又一次地发现,我们可以走进去说:“嘿,我们是独立的,我们不分党派,我们做的是非常严谨的工作,我们来这里只是为了帮忙,我们也可以和你们一起钻进细节里。”对此有极大的兴趣,也有很强的需求。过去七八年里,这一直让人非常受鼓舞。
你们是跟议员本人、也就是国会议员做这些工作,还是跟工作人员做这些工作?
所有人,任何人都有可能。再说一次,我们在国会山与国会做了大量工作,与一些国会议员,也与很多工作人员。我们会办听证会和简报会。此外,我们在行政部门也做了很多工作,与不同的机构——白宫、战争部、国务院、商务部——在各个层面全面展开,具体取决于议题以及当时正在发生的事情。
我之所以从这里开始,是因为我想和你进行一场相当有深度的对话,但我知道我们的听众是怎么想的,他们想的就是:政府根本不懂这些。政府永远不可能追上科技行业。他们永远也写不出一部法规。他们就该让开路。
在这个选举周期里,我看过不少候选人论坛的视频,有人在里面说:“我试过一次 Claude。”这里存在着某种巨大的鸿沟,无论是真实存在的,还是人们感知到的。我只是想理解你怎么看待这道鸿沟,你认为它有多大,因为当我们进入 AI 安全和监管的话题时,正是这道鸿沟必须被弥合。
鸿沟确实存在。而且确实是一道很大的鸿沟。但也有可能从外部高估这道鸿沟,或者如果你只是从加州隔着看的话。我在湾区住过三年,所以我对这种视角绝对非常熟悉。
我认识很多科技行业的人,他们过来开始跟国会工作人员或行政部门办公室的幕僚交谈。一旦他们真正开始这些对话,他们就会说:“哇,这些人真聪明。这些工作人员确实知道自己在说什么。”
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(The OpenAI board member who fired Sam Altman says we still can’t trust AI CEOs)
Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life.
Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as “the blip,” ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely.
But it’s worth remembering what Helen and the board were supposed to do: OpenAI’s nonprofit board existed specifically to make sure the company didn’t betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull the lever, that accountability mechanism broke, and here we are.
Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here.
Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked real companies. The CEOs of these frontier labs have since called for a slowdown and asked Washington for help, but the White House’s answer was a voluntary accord — one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI’s biggest competitor.
Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble.
So we talked about what we might do instead: what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like. That includes how to think about China, which is the issue that looms large in every policy conversation about AI right now — especially now after the Jensen Huang affair. Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you won’t hear us talk about that directly.
Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday.
Okay: Helen Toner, the executive director of CSET, on the future of AI safety. Here we go.
This interview has been lightly edited for length and clarity.
Helen Toner, you’re the executive director for the Center for Security and Emerging Technology at Georgetown University. Welcome to Decoder.
Thanks so much. Great to be here.
I have wanted to talk to you for 10 million years. I am so excited you’re here. You’ve been involved in a lot of AI safety and ethics ideas for a long time. You were obviously on the board of OpenAI during what everyone has come to call “the blip.”
Now, you’re talking a lot about AI safety, about which I have 10 million questions. But let’s just start at the start. You run the Center for Security and Emerging Technology, people call it CSET. What is CSET?
It’s awesome to be here. I’m really excited to talk to you as well. CSET is a team inside Georgetown, but we’re really not academics. We’re about 50 people and have been going since 2019.
Basically in 2019, a group of us decided that there was a lot happening and a lot of emerging technologies that were becoming very relevant in the national security space, and the security space more broadly,. And some policymakers who had very important jobs did not have the time or the expertise to get across what was going on with those emerging technologies. Already at the top of our list in 2019 was AI.
Basically our mission is to do analysis that is really rigorous, but also actionable, for policymakers and analysis that is both grounded in the technology and grounded in real national security expertise and understanding and issues. We produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff.
Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America, and there was a big blue wave. There’s a lot of polling that says the number one issue is corruption. We’re going to throw the bums out. We’re going to start over and get rid of these corrupt guys.
The AI industry, and the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, “Oh, there’s a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are.”
It just occurs to me that maybe people don’t believe that’s effective anymore, that the money and the corruption and the AI CEO standing next to Trump have completely outweighed the well-meaning, sincere, “here are the actual problems” research that you do. What is that dynamic actually like?
It’s a reasonable concern to have, and there are definitely some spaces where money just absolutely overwhelms everything. I’m Australian. I didn’t grow up in the US, and so the amount of money in US elections is pretty mind-blowing to me, but there are limits to what money can do.
AI is a great example. In this election, there were just unbelievable amounts of money being thrown at candidates, at messages, at opposition research. There was all kinds of astroturfing on Twitter, and some very strange things trying to get anyone who talked about AI governance, regulation, safety, any of those words thrown out or not elected. It turns out to have really not worked very well.
I think the reason that it didn’t work very well is one, this is an issue that voters increasingly care about. And two, also interestingly, it’s an issue that is pretty bipartisan at this point. What we see in our work is that there is a real thirst among policymakers for high-quality information. That’s members of Congress for sure, but they’re also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world.
In DC, if you live and work here, you know that lobbyists in DC are not just people who go in and push their company’s position, though they do do that. They’re also educators. A big reason that congressional staff and others want to have lobbyists come talk to them is because they’ll really help them understand the weeds of a given piece of legislation — what does it mean for that industry and so on and so forth.
What we find over and over again is we can go in and say, “Hey, we are independent. We’re nonpartisan. We do really rigorous work. We’re just here to help and we can go into the weeds with you as well.” There’s a ton of interest in that and appetite for that. It’s been really encouraging over the past seven to eight years.
Are you doing that with the principals, the members of Congress, or are you doing that with the staff?
Everyone and anyone. Again, we do a good amount of work with Congress on Capitol Hill, with some members of Congress, with lots of staff. We do hearings and briefings. Also, we do lots in the executive branch, with different agencies — the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on the topic and what’s happening at any given moment.
The reason I’m starting here is I want to have a pretty heady conversation with you, but I know what our audience thinks and it’s just that the government doesn’t get it. There’s no way the government will ever catch up to the tech industry. There’s no way they’ll ever write a regulation. They should just get out of the way.
In this election cycle, I’ve seen a number of candidate forum videos where someone was like, “I’ve tried Claude once.” There’s some huge gap, real or perceived. I’m just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it’s that gap that has to close.
There absolutely is a gap. It absolutely is a big gap. It also is possible to overestimate the gap from outside, or if you’re just looking across from California. I lived in the Bay Area for three years, so I definitely am very familiar with this perspective.
I know a lot of people from the tech industry who come over and start talking to congressional staff or staffers in executive branch offices. Once they actually start having those conversations, they’re like, “Wow, these people are really smart. The staff do actually know what they’re talking about.”
With principals, it is a different issue. Some of them are in the older generations, but they’re also getting smart. It takes time and it takes effort, and these people have so many demands on their time. The cynical thing to say is they’re just busy trying to get elected, but a lot of people in these roles also just have really big other topics that they have to be thinking about. They have to be thinking about wars and healthcare and cost of living and they’re looking at a lot of other stuff from day to day.
I saw after ChatGPT, there was a wave in 2023 of, “Oh, maybe Congress is going to get activated on AI, maybe they’re going to actually do something.” It didn’t really turn into anything actually happening, but it did turn into a real moment where a lot of members of Congress and other policymakers started to get smart on AI. It takes some time to learn what a language model is. What even is AI? What is machine learning?
They put in some of that work in 2023. I think this year we’re seeing another big wave of interest and people making the time and making the headspace to get smarter on it. I definitely don’t want to say there’s no gap or the government is super smart on AI across the board, but there are some champions who are really smart. I think the staff are generally smarter than people give them credit for.
In my mind, it’s much more a question of, “Can the institutions, in the state of functionality that they’re in, get anything significant across the line?” more than, “Are these people just too behind the eight ball?”
We’re going to see. There’s a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it’s capable of. We’ll see if they have revitalized the institutions.
[Laughs] I’m sure it’ll be all fixed overnight.
They’re very excited today, but we’ll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself, and how you are thinking about it, and the nature of the companies that are making the technology.
The reason I’m asking that is I feel like there’s a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. My favorite example of this is we have straight-up asked Anthropic if it thinks Claude is alive and the answer was not no. The answer was a long discursion into the nature of the word “alive.”
Well, it depends on what you mean by “alive” to start with.
It does. And their answer was, “You need blood.” And I looked at that and I was like, “Who was talking about blood? It wasn’t us. Why are we talking about the requirement of flesh and blood to be alive?” And then some further conversation about the moral status of Claude is yet to be determined.
There’s a huge divergence here between the AI overviews most people experience in Google search and “the moral status of Claude is uncertain.” Where are you? How are you situated in that debate?
A big part of how I see our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to translate and voyage to California and then bring back the artifacts and the recountings of my journey.
I definitely think that there are a lot of people who have used the free tier of ChatGPT, or they’ve used voice-based models, which generally do worse, or they’ve just used Google AI overviews and they’ve laughed at how stupid Google AI overviews are. They haven’t used agents and they don’t know what frontier models can do. So there is just a basic piece around what the current generation of models can do.
There’s a historical piece as well, which I find increasingly relevant as I’m now 10 years into working on AI and national security stuff. If you’re looking at this current moment in time, this snapshot, and saying, “Well, AI seems kind of cool, but it can’t do X,” you have to situate that in the long history of people being like, “Well, AI seems kind of cool, but it can’t do X,” for many, many values of X that have since become totally overcome by events.
You can go back and say, “Okay, well, here’s how we were talking about creativity, for example, five or 10 years ago.” Or we were talking about reasoning as this insurmountable thing before you had these new models come out. A huge thing that I am spending time on these days is not just the past and present of AI, but how companies are thinking about what they’re doing, especially the two leading companies, OpenAI and Anthropic, and this is true at Google, Meta, SpaceX to some extent as well.
This idea in the Bay Area that it’s just in the water, very normalized of, “Well, first we’re going to figure out how to do recursive self-improvement (RSI) and then we’re going to do that until we have superintelligence that is AI that vastly outstrips humans.” That’s taken for granted in a lot of circles at this point. I think there’s still a ton of alpha in just taking the fact that that is what they’re trying to do and just explaining what that means.
It takes a little while to lay out. “What is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic?” All of that. But it is something I find myself spending a lot of time on these days, and something that there is increasing willingness to take seriously and to listen to in DC as well.
Lay that out for me. I think probably a lot of people listening to this show understand what you’re talking about, but we have a big audience. What do you mean by, “We’re going to do RSI and that will lead to superintelligence?”
The starting point here is these companies, especially OpenAI and Anthropic, were not set up to be consumer internet companies. A lot of people encountered this whole space after ChatGPT was launched. So the natural slot in their brain that they put a company like OpenAI or Anthropic into is a consumer tech company, where they have an app and I play with their app.
I cannot overstate how far that is from the vision of both companies when they started, and also the vision deep in the hearts of some of the top people at both of those companies. The vision deep in their hearts is we’re going to build machines that are as smart as or smarter than people. That is going to be amazing and it’s going to be a huge scientific achievement and it’s going to revolutionize the world and we’re going to be at the forefront of that and we’re going to do everything we possibly can to drive towards that.
That’s not a totally brand-new thing. We don’t have to do the whole history of AI, but it started as a field in roughly the 1950s with this idea of whether you can recreate human intelligence inside a machine. That’s kind of the starting point. If we’re talking about people like Sam Altman and Dario Amodei, but also the core people inside, the researchers inside their labs, so your Alec Radfords or your John Schulmans or people like that, that is the vision that got them into this space that is motivating everything that they do.
That’s been true since the beginning, since OpenAI was founded in 2015 and Anthropic in 2021. These days, the point that they’re getting to is this idea of RSI, recursive self-improvement, which again, is a very old idea. In the 1960s, this guy called I. J. Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human, and then that could build an even more intelligent machine and that could build an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning that the level of intelligence of the systems is exploding.
For decades and decades and decades, this was this theoretical idea of, “Well, if the AI gets smart enough, maybe it can take over building the next generation of AI or directly improving itself.” I think 2025 and 2026, we’ve started to see AI companies saying, “Okay, yeah, this theoretical idea… we’re actually starting to be able to do a minimal version of it in practice.”
It’s not handing over all the AI development to the AI, and it’s certainly not saying to GPT-6, like, “Okay, GPT-6, improve yourself.” That’s not what they’re doing. But they are progressively using AI for more and more parts of their overall R&D pipeline, and they are increasingly saying that they’re seeing acceleration from that. They’re seeing their own R&D getting faster and faster.
A lot of people elsewhere doubt this, but a lot of people inside the companies really think it. When Demis Hassabis says “we’re in the foothills of the singularity,” this is the kind of thing he means: We might be seeing that initial part of the exponential curve where the AI systems are building the next generation of smarter and smarter systems such that the hockey stick then goes up.
At this point, it’s been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. So I feel like there’s a lot of value in just trying to lay out, “Here’s what they are trying to do. They might not be able to do it, but they might be,” and presenting that to a range of different people.
How would you manage that? Demis Hassabis saying “foothills of the singularity” is an all-time quote. I’m not 100 percent sure the people at Google knew he was going to say that on stage to end the conference.
Greg Brockman, president of OpenAI, announced a new version of the model a few months ago, and he was like, “Here it is. It’s Astra. It’s AGI now. Have a good time.” And he ended the call.
Are you there? Are we at AGI? Is this already here and we just haven’t noticed?
I actually wrote something about this. I do not want to say we have AGI. I do want to say we’ve reached the point where I think using AGI as a milestone is just totally unhelpful. The way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as different parts of the cloud or different things people mean by AGI.
There are people who’ve complained about the term AGI for a long time. Ten years ago it was pretty helpful to be able to say, “Look, the AI systems we have today in 2016, they can all only do one thing and they can recognize speech or they can play Go or whatever, but we’re trying to build this bigger thing that we’re going to call AGI, which is human-level and it’s going to be so general purpose. It’s going to be amazing.”
That was a useful term then because this fuzzy cloud of different things AGI might mean was so far away, so you could just point at the fuzzy cloud. But at this point, we’re inside that cloud, meaning we’ve passed some definitions of AGI. A very basic one, for instance, would be even when ChatGPT came out as an artificial general intelligence. ChatGPT was very clearly not a specialist model that was only good at one thing, like recognizing credit card fraud or translation or whatever. It could do many, many, many things.
So we’ve reached some definitions of AGI. We’re far from other definitions. Some people will say AGI is when it awakens and is sentient and can do every single thing that any human ever can do. I think we’re pretty far from that.
At this point you can have arguments, and people do have arguments, of “well, Astra is AGI,” or “was [OpenAI’s] o3 with tools AGI” or “was ChatGPT AGI,” whatever. If that’s the argument we’re having, we just need to set aside that term and come up with different, more-specific things we mean by that.”
The president wants to call it “Super Intelligence.” I don’t know if you’ve heard.
[Laughs] I have heard. The thing I find saddest about that is just that it was an existing term that is becoming increasingly relevant. I have a lot of feelings about words and terminology.
Are you AGI-pilled? Do you think we’re going to get there in the current run of technology?
I think it is obviously possible enough that we need to think seriously about it, which is my stance on a lot of this weird stuff. I’m not like, “This is definitely happening, it’s definitely happening in two years.” But I’m like, “Look, I think this is very solidly plausible and very solidly in the realm of real things that could happen that we should prepare for.”
I work in the national security space. There’s things like, “Will China launch a full-scale amphibious invasion of Taiwan?” where most analysts in the space will say “probably not,” and “probably they would lose.” Also, “We would see signs of them preparing and whatever, but we should take that seriously.” We should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios.
Do you think Claude is conscious?
I have no idea.
That’s an answer. I mean, that is legitimately an answer.
It is scientifically such an unsolved problem. What do we mean by consciousness? And is that a thing or is it actually 10 different things that we’re lumping together or what? I have no idea. What do you think?
No. I seem to be the person in this ecosystem who gets asked questions like that and I’m like, “No.”
Because the thing that I struggle with is, there are lots of other things where we’re already confused about whether they’re conscious. So with animals, are any animals conscious? And if so, which ones? Insects?
This is why I understand why Anthropic was like, “Does it have blood?” And then you can just draw the line. I’m like, “I think I’m there and I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious.”
My point of view is that the frontier models to me sometimes feel brittle in the way that computers feel brittle, and the sign of consciousness I’m looking for is to overcome that brittleness.
That’s interesting. I definitely lean towards no, it’s not conscious. I think something that is really, really tough in this space is that it’s really bad if you get it wrong in either direction. If the computers aren’t conscious and we treat them as though they are, we’re going to make all kinds of terrible decisions.
But if the computers are conscious, meaning in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren’t, then that’s also a really huge, terrible disaster. So I don’t know. I feel pretty nervous about the whole topic and I have no idea how to think about it.
The reason I’m asking these questions is I started with what the nature of the regulator is. Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is “Claude is alive,” you’re going to come to vastly different conclusions than, “We have built some ultra-powerful computers that can take actions without oversight and that might not be connected to our goals.”
I think that’s right. Though I also think people make a different mistake.I just heard a TV anchor do this the other night. They’ll conflate different things. So they’ll say, “Do you think that AI systems are at risk of running amuck and causing damage to critical infrastructure because they’re sentient?” And I’m like, “Wait, wait, wait. Those are two very different things.” So I agree with you.
One of the most fascinating things about the project of building AI is that we’re learning as we go. All these things that we thought were connected to each other aren’t necessarily connected to each other. An old version of this is, people said before we had chess engines: “Well, if a computer could learn to play chess, it would have learned something deep about strategy.” No, it turns out definitely not. You can just put in a bunch of chess moves and it learns nothing generalizable about strategy.
It goes in the reverse direction as well, like, “Well, it’s just a computer, it’s just math, and so it’s not going to be able to adapt on the fly,” or something. We then see that no, once you do have a level of long-horizon autonomy and the models have been trained on a wide range of different really hard problems where the environment is changing around them, they can learn to adapt and come up with different plans on the fly and troubleshoot and so on.
So I don’t know. I try to have a level of humility about how confident we can be that one thing goes with another thing, if that makes sense.
Yeah, candidly, I have been wrong about the limits of LLMs every single year, but there’s that brittleness that I’m talking about. OpenAI solved Navier-Stokes. Does the model know this? Will it know that there’s new problems to solve? I don’t think so.
The next model will know it. The model that’s pretrained.
Right. But you have to make a new one.
Yeah.
That’s a very weird dynamic, and that is probably where you enforce the regulation. There’s “what we are going to let you do with the models you have today” and “what will you teach the models of the future to do?” And if you think that they’re conscious, you end up in a moral quandary, not a legislative quandary.
Definitely. Yeah, totally.
This is what I mean. Actually, it’s the heart of this question for me. I’ve been just slowly building up to it. I’m not 100 percent sure alignment as we have it today is actually how you solve the safety problem. It doesn’t seem to be working. We seem to be attacking Hugging Face however we want. That feels like a technology problem. That gap is there.
And when you say alignment, what do you mean by that?
Just sort of the very basic common-sense version of “we’re going to tell the model to not be evil” or Anthropic is going to write a Claude constitution that’s like, “You are a good person.” However we think of alignment in that very dumb system prompt way. Maybe there’s another better way to think about it, but that’s how I have always thought about it.
That makes sense. The reason I ask is I feel like alignment as a concept is… Like I said, I hate all the words. I sometimes say we should just ban all the words, no words allowed. It covers too much.
People in AI will often talk about solving this or solving that. So it’ll be solving alignment, or DeepMind’s original mission was to solve intelligence, then use that to solve everything else. People talk about solving stuff. I don’t know that alignment is a thing that can be solved.
Some people use “alignment” to mean “the company can put on guardrails and the guardrails work,” meaning the model doesn’t do what you want because the company tried to ban it, meaning biological weapons or cyberattacks or whatever. Some people use “alignment” to mean “the company does exactly what the user wants,” which is the reverse.
And some people use “alignment” to mean what you said — that the model is morally good in some kind of broad way or can direct the future in a way that will lead to human flourishing or something. Those are just wildly different things to try and build. I think the most coherent one is trying to get the AI to do what the user wanted it to do. It seems like that is, as far as I can tell, not on track to be solved, which is not great, to put it mildly.
To me, this is the core problem. We’re going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying $20 a month the ability to generate novel bioweapons.
It feels like that is a shared policy goal, maybe the only bipartisan policy goal in America. And then I’m like, “How? By what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my house? Do I need a Flock camera above my keyboard?” How, specifically, would you do it?
With this one, and with cyber capabilities as well, there’s a perspective in some parts of the AI safety community that some capabilities are just too dangerous — capital T, capital D, Too Dangerous. So you have to do something, because if they proliferate, everything will be terrible forever because there’ll be so many attacks and so much damage. I think that just underestimates the level of dynamism and the finding of equilibriums and responses and ways that society can adapt.
In the bioweapons case, for example, there are already hundreds of thousands of people in the US who know how to make bioweapons because they’ve been through biology training, and we don’t have huge amounts of bioweapons attacks. Some part of that is that that knowledge is relatively scarce, but there’s also a lot around things like access to materials, the ability to do mail-order DNA synthesis; the kind of policy you could implement is restricting mail-order DNA synthesis.
It’s also due to things like the FBI monitoring terrorist groups. I’m serious. It’s a really big part of it. So I don’t want to be too Pollyannaish here. I think it’s definitely possible, especially with cyber capabilities, that we could see some really tough transition times.
I think we already are, right? We appear to be in a transition. There was a small group of cyber practitioners, both offensive and defensive. They were in a community and they had developed some etiquette and now we’re like, “Some new guys are going to show up and not care about your etiquette at all, and we’re all going to patch PHP all weekend.”
I don’t know if it was etiquette though. And we have some work on this hopefully coming out soon, but I think with cyberattacks, there are things that limit how many cyberattacks there are in the world that aren’t just how good the defender’s software is. It’s things like, people don’t want to go to jail. Nation-states who typically have the best cyber capabilities don’t want to get bombed or sanctioned.
So again, and as you say, we’re clearly seeing some amount of turbulence and new attacks. So I think we should wait and see data on overall impacts, because there are so many cyberattacks per year that if you start paying attention to every cyberattack that happens, at any point in history, you could have been freaked out.
I think we’re definitely seeing some increase, some sort of out-of-balance in the equilibrium, but I think we have capacity to absorb a lot over time and to kind of reach a new equilibrium — if the only problem is bad people using the AI, which is separate from the question of whether the AI itself does things autonomously, which is a whole other dynamic.
I mean, those are the two. Again, this is why I started on alignment, right? You can talk about alignment in terms of “we’re going to filter the user’s queries and we won’t let you make bio weapons, but if you do, the cops will find you and you’ll go to jail” and that has whatever deterrent effect it might have, hopefully a very significant one.
Then there’s “the AI systems are going to game their way into attacking Hugging Face to find the mythical grader they think will let them pass the test, whether or not the grader exists.” This is a whole different set of problems.
Somewhere right in the middle of that is Nikesh Arora, the CEO of Palo Alto Networks, saying, “At some point, the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyberattacks.” I’m like, “Wait, we can’t even keep the autonomous agents in the box. Why would I trust them in this way?” The technology to solve that problem just seems opaque to me.
No, I agree. One way of putting that is, in cyber specifically, there’s a trust imbalance in how much trust you need as an attacker versus a defender. As an attacker, you can just yeet out a bunch of agents and be like, “Go try and penetrate a bunch of systems. Have fun.” As a defender, if that’s part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and ability to mess with your software. So you need much, much stronger assurances.
Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren’t just giving agents a lot of permissions. But I agree with you. With the autonomous stuff though, I think it’s worse than that because there’s a little bit of a fallacy at the moment around, “Well, the solution to autonomous cyberattacks is better sandboxing.”
The reason it’s a fallacy is we want to build these systems so that they can do things for us. So there’s this huge incentive toward giving them more autonomy and giving them more permissions and more actuators, as people in the robotics world would say, or things that they can use to do stuff in the real world. That to me is where it seems really hard to get around the need for better alignment than we have right now, and so that definitely makes me nervous.
Is that a technical problem that can be solved? If we were to write a policy today that said if OpenAI hacks Hugging Face again, Sam Altman will go to jail. We will obviously have created some enormous incentive for OpenAI to solve this problem.
[Laughs] It doesn’t sound like the best legislation, but…
I think a lot of our listeners are very excited about that. They want to put him in jail because he wants all the RAM in the world. At this point, there’s a blue wave, Helen, that’s all I’m saying.
But say you wrote that legislation: “We’re going to impose massive liability on the labs if their systems go off and autonomously do cyberattacks. Don’t do this again, you’re going to be in trouble.” Okay, we’ve created an incentive for them to solve it. What are they going to do to solve it?
I do think there’s a ton that can be done. If you talk to people inside these frontier companies, they will tell you they see hundreds and hundreds of obvious things to do that would help and they just don’t have time to do them. So they’re massively triaging and trying to do the two or three they can for this fine-tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever. A lot of those simple things to implement on the engineering side.
There’s a whole other piece, which is more the underlying science of, “Well, how do these models work? When we use different alignment techniques, what is actually going on?” A huge question is this idea of generalization, which is you train the model on one set of things and its knowledge and its abilities generalize to a somewhat broader set of things, which is true of all AI and has been true of machine learning systems forever.
But we have a very bad understanding of how that generalization works. Why does it generalize to one type of thing but not to another type of thing? Those are all scientific questions and there’s a lot of work that is happening on them. It’s just that science takes time because you have to try things and you have to critique each other’s work. You have to reproduce each other’s work and you have to come up with new theories.
I’m a long, long way from saying these problems are fundamentally impossible to solve, but I think it’s a question of how long we have to solve them. When the people who are building the systems tell us, “Oh, we have to solve this within 12 to 24 months,” I think that’s terrifying and I hope they’re wrong.
If I just rely on my own internal understanding, I think they’re probably wrong and we probably have longer, but there are just very, very smart and very, very thoughtful people who really think that it could go that fast. And then it doesn’t matter fundamentally whether the problem is solvable. It matters in practice whether it is solvable in the time that we have.
Is that the argument to slow down, to pace the frontier?
I think so.
Is it to build the systems that we know we need to build right away, that we can, and then to do the science to build the next generation of systems?
I think so. My view of the idea of slowing down has changed a lot over the years as the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Back when GPT-4 came out, there was the six-month pause letter, which became known as the “Elon Musk Letter” because he was one of the signatories. At the time, I thought, “That’s really not going to help. What are you going to do for six months and why are things going to be better six months from now? It doesn’t seem useful.”
Whereas today, when I talk to people who are building this technology, they say, “Man, the amount that we could do in six months if we weren’t constantly just trying to prep for a new model release or a new training run to be kicked off.” There’s so much they could do.
So I don’t want to say the correct policy is to pause for six months. I think there are a lot of reasons that would go badly. But I think about the overall concept of, “Can we give ourselves more attention and time?” Maybe the best way to do that is just to set a higher bar for “here’s how confident we need to be in the safety of these systems before we proceed with something.”
Whether it’s releasing a new model or, more importantly, even training a new model, we have to have this level of confidence in our risk assessment and our risk management. And then if they don’t need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it’s all about reaching that bar, if that makes sense.
Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer and why would it go badly such that we shouldn’t just stop?
I think the question is how do you implement, and how do you enforce, a pause and what exactly are you pausing?
Let’s say you’re the AI czar of America. So let’s say you can tell them all to stop: “Dario, shut it down.”
Stop what though? What specifically are they stopping?
Stop training the next model, stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again, for however long.
You’re saying that there’s a list, that people would tell you that there is a list, but it might not be worth it because something else would happen that is bad.
Yeah. One thing that could happen is you’re creating a huge incentive for the companies to work around the letter of whatever specific thing you tell them to pause. They’re going to find whatever cracks in the dam that they can to push through that.
A different thing that could happen is you could have too big a blast radius, so you say “pause these things” and you think it’s pretty narrow, but actually it prevents a bunch of nonprofit products or startups or other actors from doing work that actually would be totally fine and would be good for pluralism and dynamism in the space.
A different thing that would happen is China would probably keep going, and we can talk about the “But China” argument, which I think has strengths and weaknesses. But I take your point. Your point is, what would we actually do? I am not an AI researcher, so I don’t claim to have the full list.
There are two halves people talk about, which are alignment and control. In this breakdown, alignment basically means making it so the models are trying to do what we want them to do, and control is, if the models are trying to do something we didn’t want them to do, can we catch them? Can we stop them?
In both of these, as I understand it, there’s just a huge backlog of things, like all kinds of interpretability work that you could do on different sorts of models. So when you see a certain behavior that is concerning, how much time can you spend using interpretability techniques, which are these mathematical techniques to go in and look at which parts of the neural network are activated and what do they correlate with and what might that mean?
It’s just time-consuming and it takes compute and it takes some combination of automating things with AI and then humans going in and looking at it. There are lots of tests you could do to see what kinds of behaviors this new model displays in different kinds of situations and how we think about it. Is it showing us how it would behave in the wild in a real deployment, or is it showing us what gets called “evaluation awareness,” which is when the model recognizes that it’s in a test and then behaves differently because it’s in a test.
The famous one, the one that I love, is of this Google model, which in part of its reasoning, its chain of thought, said,, “I mustn’t do that, otherwise I’ll become a famous example of a misaligned AI in a paper.”
So you could go in and investigate that much more. You could try many more different types of settings. You could use interpretability in combination with these different settings. You could try out different ways of fine-tuning the models and these different sorts of alignment techniques. You could experiment with lots of different versions. Anthropic has its constitution, and there are different ways of training the AI to work towards the constitution. OpenAI has its model spec.
They’re both the “just try things” version, just try many different versions of how to train to do those different things. Also, the more underlying science version of, “How can you go in there and try to understand the fundamental dynamics of what’s happening?” That’s just on the alignment side.
So then on the control side, it’s the same thing. There are so many different types of setups you could try. There are experiments that you could run. There are simulations you could run. You could train different types of AI to specialize in different types of roles and different types of control setups.
This is just off the top of my head, not as a researcher, but my understanding is the people in these spaces say, “We could do so, so, so much.” They could do that if you just had a magic wand and waved it and said, “Everyone freeze on the new capability side and just let these people do their thing.”
Again, if you’re just having a higher bar for how confident we need to be that the risk is being managed reasonably, then that would also create space and incentives for more of that work to be done as it’s needed for new training and releases.
One idea that I’ve heard you talk about in the past is that maybe we should just restrict training and say the models are good enough right now. They have not diffused into society and we should just use all the compute for inference and let everyone figure out what the AI is for. That’ll generate a bunch of money and hopefully use that money to improve safety.
Do you think that’s a realistic policy proposal?
I wouldn’t go for that black-and-white a version of “no more training, only inference,” but I do think that there’s… this isn’t my framing, but I like it. I think this idea of decelerating the vertical growth of AI, meaning the pure capabilities growth, making more and more and more advanced AI models, but trying to accelerate the horizontal growth, meaning integrating AI into different use cases, fine-tuning models that could solve specific problems, integrating it into parts of the economy that might take longer, and solving scientific problems.
Sometimes people talk about wanting to slow down AI and they mean just the very specific thing of slowing down the growth of the most advanced models, but people sometimes take them to mean slowing down everything to do with AI. So less deployment, less experimentation, less integration, which I think would be a much worse idea. You asked if it is feasible, I think.
Well, is that just a good path? If we could put a bill in front of our new Congress and the bill said, “Stop spending all the money on training and put all your resources on safety.” Part of that is, “Sure, let it rip on inference. More and more industries will figure out what AI is good for and not good for. Some bad things will happen, but we will focus your profits on fixing the things for now.”
There’s a policy idea embedded in there that is workable. The implementation might be very challenging, but it’s a workable policy idea. Actually, I’ll give you an example that I’ve been thinking about as we’ve been talking.
We just had Bill Gates on the show, and I tried to do the same exercise with him. What is the nature of the technology? What is the nature of the companies? Do you think it’s alive? Basically, his response, the sum total of that hour I spent with him, was, “Who cares? The problem is now.” I would ask him, “What is the nature of AI?” And he’s like, “I don’t give a shit. Everything’s going to get ruined. Do something.”
That’s just been swirling in my head as we’ve been talking. So one workable policy solution is “don’t use the computer chips for that, use the computer chips for this. That’s going to create a bunch of problems and necessarily we will have to create safety thresholds and all the other stuff.” And then you can go make the next generation of models that are even more powerful and we’ll have some muscles. We’ll have some capability to deal with it. It just doesn’t seem like the industry is going to come to that conclusion on its own.
It’s a pretty heavy hammer. I don’t feel like it’s what we need today, but I do think it’s the kind of thing that I could imagine working. Again, if the OpenAI and Anthropic people are right and we only have two years before the god machine, then I think we’re going to have a number more steps in the pipeline of people getting freaked out and people wanting to do something about it.
I think if we are seeing models that are agents and systems that are getting more and more capable and wreaking more and more havoc, then that is certainly the shape of things that could make sense.
My last question in this phase is what to make of these companies. We’ve been talking about the technology in a very abstract way. If I read Anthropic’s S-1, one very obvious conclusion is they should stop training new models.
They should just start selling the very lucrative inference tokens they have today and the technology is not diffused into the market and probably some competitor should come along and be like, “I’ve distilled a version of Claude and I’m going to run around every local plumbing business in America and do whatever automation I can for lots and lots of money.” That is a market dynamic that may occur or may not. But Dario wants to build AGI. There’s not a market dynamic that’s going to stop him from that.
Yeah, he really does. He really, really does.
You obviously have insight into all these companies. What do we make of the companies? Are they single-minded in purpose? Are they all doomers? Are they going to stop themselves?
I heard you had Kevin Roose on your podcast recently, and I think I would echo something that I heard him saying there, which I forget if he said it explicitly or if I just extrapolated it in my head, which is basically there’s this sort of AGI-pilled mindset. There’s this mindset that the systems that we have today are not fundamentally very far away from systems that could be incredibly, incredibly powerful.
That’s the direction things are going in and that’s the direction we should maybe want them to go in so that you’re building AGI. Building AGI is feasible and we could do it soon and the systems we have are not far away. That was this crazy, heretical, “get you laughed out of the room” kind of view in 2016 or 2017, and even before that, right?
DeepMind, when it was founded in 2010, was an AGI company and at the time it was like, “Wow, what a bunch of weirdos.” The same thing was true of OpenAI when it was founded at the end of 2015. They were an AGI company and that was a very weird thing to say and a very weird thing to be.
What we’ve seen over the past 10 years is that bunch of weirdos who had this weird belief that the AI systems we can build are not just great B2B SaaS products. They’re not just fun chatbots. They’re going to fundamentally change the nature of the universe. That group of people keeps being right in the sense of building the best AI systems that the most people want to use and that make the most money.
I think you have that set of true believers — which is very concentrated in OpenAI and Anthropic, but there are also some of them at Google, Meta, and SpaceX — and they haven’t been right about everything, to be clear, but their star has risen.
So you now have people who are not AGI-pilled feeling like they have to pander to those who are so that they can attract the good researchers who will build their good B2B SaaS products. But if you’re just saying AGI-pilled things so that you can hire the researchers to build the B2B SaaS, then you’re not actually going to get there.
I do like the idea of the AGI-pilled, starry-eyed executive throwing off his coat and revealing that he was just a SaaS salesman all along.
[Laughs] Yeah, I’m thinking of someone like Zuckerberg here. I don’t know him personally. Maybe he is more AGI-pilled than I think he is.
Alexandr Wang is AGI-pilled.
Absolutely. And so what did Zuckerberg have to do to get Alexandr Wang to believe? It seems like throwing him lots of money, both for his own salary and also to spend on his team.
If you’re looking at these different companies, it makes sense to try and think about, “Okay, who here is AGI-pilled and who isn’t?” A fun dynamic, or a very irritating dynamic sitting in DC, is that all these companies, including the very AGI-pilled ones, have these lobbying arms, these government affairs arms, that are so not AGI-pilled.
So you hear Sam or Dario or their top executives and top researchers talking about things. And then you go and sit in a closed-door roundtable in DC with their top policy or government affairs people, and it’s like night and day. They’re very much in the B2B SaaS mindset.
People who come from the lobbying world, often their whole job is to make sure the government isn’t too worried about whatever your company is doing. It’s like, “We got it. We’re good,” which is often pretty jarring when you’re familiar with what their bosses are saying.
One of the dynamics here that’s super fascinating is right now they’re weird research labs with lots of money and lobbyists talking to politicians, and there isn’t some countervailing economic pressure in their structures. They don’t have to make money, but soon Anthropic will go public.
It’s funny you mentioned Kevin Roose. He is AGI-pilled and I’m not, so I think that’s a healthy dynamic for both of us to be friends. But he wrote in his book that Sam Altman leaving OpenAI and then coming back was when OpenAI changed forever. It stopped being a research lab and it started being a much more commercial entity that was actually chasing profits in some way.
Indeed, OpenAI did restructure. I mean, you were in the middle of that. I’m curious, is that how you see it — that the industry has gotten more commercialized?
To me, it fits into this long-running debate that has been happening in AI safety circles for 10 to 20 years: Is AGI, human-level AI or powerful AI or advanced AI or whatever, is that something where it’s going to go from something very small to suddenly being very powerful? Eliezer Yudkowsky will talk about “foom,” as in the takeoff, where you have this AI that’s not that great, but it figures out the right algorithm and then suddenly foom — it’s incredibly smart, incredibly fast.
The other kind of take would be, “No, this is something that’s going to compound over time. It’s going to take a while to get from the less-advanced systems to the more-advanced systems and it’s going to be economically useful along the way.” So people are going to be building businesses around it. It’s going to take some time to ladder up.
I think maybe it was literally Kevin on his episode with you talking about how there’s this Anthropic document from the early days where they say, “Oh, we’re only going to need 100 people or something in our lab.” That’s a very foom kind of idea: “We’re just going to need to do really good research and come up with the right ideas and then we’re going to suddenly have very, very advanced models.”
Basically, I think what we’ve seen over the past four years since ChatGPT came out was that that turned out to be wrong. You had these research-focused and AGI-focused labs, and they still call themselves labs because they think of themselves as research groups that initially started selling API access as a way to fund the research. OpenAI did this, and Anthropic did this as well.
It was kind of an accident that they started doing these consumer products. ChatGPT famously released as a research preview, not as a product. But it was so successful that these research labs that wanted to be doing research, that wanted to be focusing on the big ideas and on building the most event systems, had this accidental massive commercial success at the same time as they were realizing that they were going to need an enormous amount of capital to scale. The best research approaches that they could do didn’t involve being really smart and thoughtful and having great ideas. They involved having gigantic computing clusters that cost, at this point, hundreds of billions of dollars.
This is a long way of saying, yes, I agree with you, they’ve become businesses, but I think it is not because they’ve gone through some mindset shift. I think it’s because the path of the tech tree that we’re going down both requires enormous amounts of capital and also is capable of generating enormous amounts of revenue because it is actually useful along the way.
You see this in the fact that both OpenAI and Anthropic have gone this route, even though their management philosophies and research philosophies are quite different, that it’s an imperative that they both had to make the money and it was so easy for them to build enormously revenue-generating businesses because the tech was so useful long before it reached the point of machine god. So I think that’s a long way of agreeing with you, but with some extra color maybe.
Yeah, I mean, Kevin would tell you that he could have retitled that book, “Fuck This Guy: A Story of AI CEOs.” They don’t like each other. If you had to relive that moment [of the OpenAI board vote] and say, “I don’t trust Sam,” would you relive that moment? Would you do it the same way?
Oh, for sure. I mean, we went into it knowing that it was going to suck. So yeah, for sure.
Do you think that these guys are trustworthy? Do you think Sam is trustworthy?
I think my stance on that is pretty clear, and I also think that if our hopes for the future are pinned on any one individual being trustworthy, then we’re pretty screwed. That’s why I do the work that I do, because I think we need to have different structures and many more people involved with many more perspectives being represented.
You have some hindsight now. You knew it was going to suck going into it. You definitely know it sucked coming out of it. What would you have done differently about that time?
It’s kind of hard to give the full answer to that because it would take unpicking all this stuff and all these legal restrictions and blah, blah, blah, blah. For sure a thing would have been building stronger relationships with executives on the team. The way that the company related to the board could have been different if more board members had had that. I knew some of the folks pretty well, but that’s one thing.
I don’t know. I think we could have… Yeah, no, I think beyond that, it just gets hard to… You have to go in and say, “What were all the different micro decisions and why do we make those decisions and which ones could be—”
What is the show? It’s a show about decisions. I’ve got another hour.
[Laughs] Yeah, yeah, yeah. No, and then my answers would all get annoying and legal.
Fair enough. I’m curious about it because as they go public, there will be market forces acting on them in a totally new way. There will be an investor class. There’ll be an entire CNBC machine that wakes up every day and is like, “Is Anthropic winning or is OpenAI winning?”
That is a race dynamic that might be really healthy. It might keep them from trying to kill everyone in the world because they have a bunch of investors looking for their next quarterly earnings, or it might push them further into an unhealthy kind of race. How do you see that playing out, given what you know of these people?
It’s going to be really interesting to see if going public is good or bad for their governance. A lot of people are like, “Oh no, going public, there’s going to be so much pressure to make a profit that they’re not going to be able to consider other things.”
But at the same time, the corporate governance of public companies is much more structured, much more regulated, and s
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 The Verge AI(The OpenAI board member who fired Sam Altman says we still can’t trust AI CEOs),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。