守护知识之门

AI 垃圾内容威胁科学信任 - IEEE Spectrum
IEEE Spectrum 11 月刊现已上线!
守护知识之门
面向技术内行人
IEEE Spectrum 是 IEEE 的旗舰出版物,而 IEEE 是全球最大的致力于工程与应用科学的专业组织。我们的文章、视频和信息图向读者介绍技术、工程与科学领域的最新进展。
创建账户,即可享受更多免费内容和权益
保存文章以便稍后阅读,需要一个 IEEE Spectrum 账户
The Institute 的内容仅会员可访问
下载完整 PDF 期刊仅限 IEEE 会员
下载此电子书仅限 IEEE 会员
访问 Spectrum 数字版仅限 IEEE 会员
关注主题是 IEEE 会员专属功能
加入全球最大的致力于工程与应用科学的专业组织,即可访问 Spectrum 的全部文章、档案、PDF 下载及其他权益。进一步了解 IEEE →
加入全球最大的致力于工程与应用科学的专业组织,即可获取此电子书以及 IEEE Spectrum 的全部文章、档案、PDF 下载及其他权益。进一步了解 IEEE →
访问数千篇文章——完全免费
创建账户即可获取专属内容和功能:保存文章、下载合集、发表评论——全部免费!如需完整访问权限和更多权益,请订阅 Spectrum。
守护知识之门
AI 只需按一下按钮就能伪造出一篇看似合理的论文。检测工具和同行评审能否跟上?
Harry Goldstein 是 IEEE Spectrum 的主编。
最近刷社交媒体让我开始思考信任问题。我的信息流中大约有一半充斥着 AI 垃圾内容,精明的用户会指出这些是机器生成的,这常常让那些不知情就转发的容易上当者难堪,有时则让那些故意分享的发帖者得意,因为他们觉得这很有趣,或者有助于制造混乱。
还记得 GPT-3 的美好旧时光吗?那时我们可以向聊天机器人问任何问题,它几乎会立刻、且非常自信地吐出一个看似合理但错误的答案。也正是在那时,“幻觉”一词被用来描述 AI 给出的回答,并进入了常用词汇。六年过去,几款不同聊天机器人也已迭代了多个版本,我仍然无法信任自己得到的答案,尽管 Claude 或 Gemini 常常能为我完成有用的工作。这就是当前 AI 的关键所在:信任,但要核实。而且你必须核实。
把这条准则大规模应用于维护科学记录的诚信,你就能体会到同行评审学术出版机构所面临的挑战。在《我用 AI 伪造了一篇科学论文》一文中,自由撰稿人、应用数学博士 Benjamin Skuse 讲述了人类“守门人”试图击退不断上涨的 AI 垃圾内容浪潮的故事——他尝试让自己用 AI 生成的论文闯过 IEEE 的防线。
“他基本上就是点了‘运行实验’和‘写论文’。然后他压根没对论文做任何修改。”——Eliza Strickland
正如特稿与 AI 版块编辑 Eliza Strickland 告诉我的,Skuse 惊讶地发现,生成一篇大致像样、也大致算得上科学的论文竟然如此毫不费力。她说,在他的一项实验中,“他基本上就是点了‘运行实验’和‘写论文’。”“然后他啥也没改,直接把它扔进了出版引擎的血盆大口。”
Skuse 发现,IEEE 的人工审核流程加上研究诚信检查程序,目前还能跟得上节奏——这本质上是一场与使用免费工具的人之间的猫鼠游戏。但更先进的 LLM 已经不再犯免费版本那种低级错误了。这让出版商陷入了一场军备竞赛:一边是越来越像样的垃圾内容,另一边是旨在检测这些内容的系统。
试图守住防线的并非只有学术出版商。人类生成知识的全球仓库——Wikipedia,同样如此。在本月的《五个问题》栏目中,Strickland 采访了 Ellia Lebleu,她是 WikiProject AI Cleanup 的创始成员之一,该志愿项目致力于发现并删除平台上的 AI 生成内容。Lebleu 的团队面临着与商业出版商相同的问题,但人手更少、工具也更少。
“Wikipedia 被视为人类生成信息的仓库,他们想守住这一点,”Strickland 说,“谁知道他们还能守多久。”
如果我们无法共同假定科学记录值得信赖,或者关于世界的基本事实是准确的,那么我们对何为真实的理解就会瓦解为一片不确定。也许这让量子科学家们感到欣喜,也让那些在不存在共同现实的信息环境中如鱼得水的骗子们感到欣喜。但这种不确定性最终会侵蚀人们对那些机构的信任——这些机构的存在,正是为了整理人类知识的总集,并为了人类的福祉推动科学与技术的进步。
又或许,在一个由 AI 从事科学研究、撰写论文的世界里,人类是否信任这些产出将不再重要。正如 Strickland 所说:“想到某个时候将由 AI 为其他 AI 写作,难道不觉得这很疯狂吗?我们只会被排除在循环之外。”
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Guarding the Gates of Knowledge)
AI Slop Threatens Trust in Science - IEEE Spectrum
The November issue of IEEE Spectrum is here!
Guarding the Gates of Knowledge
FOR THE TECHNOLOGY INSIDER
IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, videos, and infographics inform our readers about developments in technology, engineering, and science.
Enjoy more free content and benefits by creating an account
Saving articles to read later requires an IEEE Spectrum account
The Institute content is only available for members
Downloading full PDF issues is exclusive for IEEE Members
Downloading this e-book is exclusive for IEEE Members
Access to Spectrum 's Digital Edition is exclusive for IEEE Members
Following topics is a feature exclusive for IEEE Members
Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →
Join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of IEEE Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →
Access Thousands of Articles — Completely Free
Create an account and get exclusive content and features: Save articles, download collections, and post comments — all free! For full access and benefits, subscribe to Spectrum .
Guarding the Gates of Knowledge
AI can fake a plausible paper at the push of a button. Can detection tools and peer review keep up?
Harry Goldstein is Editor in Chief of IEEE Spectrum.

A recent scroll through social media got me thinking about trust. About half my feed was filled with AI slop that astute users would call out as being machine-generated, often to the embarrassment of the rube who unknowingly spread it and sometimes to the delight of the poster who shared it deliberately because they found it amusing or useful for sowing confusion.
Remember the good old days of GPT-3 when we could ask the chatbot anything and it would almost instantly and very confidently spit back a plausible but false answer? That’s when the word “hallucination” as applied to AI answers entered the lexicon. Six years and several versions of a few different chatbots later, I still can’t trust the answers I get, even though Claude or Gemini often do useful work for me. That’s the thing about current AI: Trust but verify. And verify you must.
Apply that maxim at scale to maintain the integrity of the scientific record and you get an idea of the challenge facing publishers of peer-reviewed scholarship. In “I Used AI to Fake a Scientific Paper,” freelance writer and doctor of applied mathematics Benjamin Skuse tells the story of the human gatekeepers trying to beat back the rising tide of AI slop by attempting to get his own AI-generated papers past IEEE’s defenses.
“He basically hit ‘run experiment’ and ‘write paper.’ And then he didn’t do a damn thing to it.” —Eliza Strickland
As features and AI beat editor Eliza Strickland told me, Skuse was surprised at just how effortless it was to generate a paper that was more or less plausible and more or less science. For one of his experiments, “he basically hit ‘run experiment’ and ‘write paper,’” she said. “And then he didn’t do a damn thing to it. He just launched it into the maw of the publishing engine.”
Skuse discovered that the combination of IEEE’s human-review processes and research-integrity checker programs are currently keeping pace in what amounts to a cat-and-mouse game against people using free tools. But the more advanced LLMs are no longer making the same dumb mistakes the free versions do. That leaves publishers locked in an arms race between increasingly plausible slop and the systems aimed at detecting it.
Scholarly publishers aren’t the only ones trying to hold the line against slop. So is that global depository of human-generated knowledge, Wikipedia. For this month’s Five Questions, Strickland interviewed Ellia Lebleu, a founding member of WikiProject AI Cleanup, a volunteer effort to detect and delete AI-generated material from the platform. Lebleu’s team is confronting the same problem as commercial publishers, but with fewer people and fewer tools.
“Wikipedia is known as a repository of human-generated information, and they want to hold on to that,” says Strickland. “Who knows how long they’ll be able to do that.”
If we can’t collectively assume that the scientific record is trustworthy or that basic facts about the world are accurate, then our understanding of what is real unspools into uncertainty. Perhaps that delights quantum scientists and the grifters who thrive in an information environment where there is no shared reality. But that uncertainty ultimately undermines trust in the institutions that exist to curate the corpus of human knowledge and to promote the advancement of science and technology for the benefit of humanity.
Or maybe in a world where AI is doing the science and writing the papers, it won’t matter whether or not humans trust the outputs. As Strickland put it, “Doesn’t it just feel crazy that we’ll have AI writing for other AIs at some point? We’ll just get left out of the loop.”
Related Articles Around the Web
原文为英文,正文由自动抓取并保留原文。
原文发布于 IEEE Spectrum AI(Guarding the Gates of Knowledge),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。