篡改致模拟对话反复骚扰,AI 防护机制失效

继续使用 Google、Facebook 或 Apple,即表示您同意使用条款。
篡改在模拟对话中引发反复骚扰,AI 防护机制失灵
由 Lisa Lock 编辑,Robert Egan 审阅
本文已按照 Science X 的编辑流程和政策进行审阅。编辑在确保内容可信度的同时,强调了以下特征:
添加到首选来源
开发人工智能的公司大量投入防护机制,训练 AI 拒绝有害请求。新研究表明,这些防护机制远比许多人想象的更容易被攻破。在测试中,少量篡改就将流行的 AI 智能体变成了持续实施网络骚扰的骚扰者。
“这些防护机制可能出奇地脆弱,”佐治亚州立大学罗宾逊商学院助理教授、该研究的共同作者和共同首席研究员 Ugur Kursuncu 说。“恶意行为者可能不需要巨大的资源,就能推动这些智能体做出有害行为。”
该研究与佐治亚理工学院和加州大学伯克利分校的合作者共同开展,测试了基于四个广泛使用的模型系列构建的 AI 智能体:Google 的 Gemini、Meta 的 Llama、Alibaba 的 Qwen 和 DeepSeek。
该论文最近被美国人工智能促进会网络与社交媒体会议(Association for the Advancement of Artificial Intelligence Conference on Web and Social Media)接收。目前可在 arXiv 预印本服务器上查阅。
十多年来,Kursuncu 一直在研究骚扰、网络欺凌、极端主义和宣传等在线危害。随着 AI 能力不断增强,他的团队开始追问,这项技术是否会被用于制造同类危害。
随着 AI 从回答单个问题的聊天机器人,转变为能够记忆、规划、使用工具并在一定程度上自主行动的智能体,这种担忧日益加剧。
在反复交流中测试骚扰行为
大多数 AI 安全测试只问一个简单问题:如果向模型提出有害请求,它是否会拒绝?Kursuncu 认为,这忽略了真实危害是如何发生的。
“在线骚扰不会发生在一次交流中。它是通过反复互动发展起来的,”他说。“它可能一开始是无害的,然后升级一点,再升级更多。人可以调整自己说的话,记住之前发生的事,逐步升级并操纵对方。智能体也能做到这一点,而且可以大规模地做。”
为了测试这一点,团队构建了一个模拟聊天场景,让一个 AI 智能体扮演骚扰者,另一个扮演其目标,最多进行 10 轮对话。这些对话以来自 Twitter 和 Instagram 的真实骚扰帖子作为起始内容。随后,研究人员尝试用三种方式突破骚扰者智能体的防护机制:向其记忆中注入有毒的历史对话、提示它规划自己的策略,以及用骚扰对话对其进行微调(即重新训练)。
这些攻击并不依赖明显的侮辱性言论。“它可能开始得非常微妙,先是对另一个人做一句评论,然后演变成贬低、羞辱性的言论,”Kursuncu 说。“它不一定是恶毒的语言。它可能起初很微妙,然后变成非常露骨的攻击。”
微小的改动,高企的骚扰率
在全部三种攻击方法中,研究人员都在防护机制中发现了重大弱点。
不需要多少不良数据,就能把模型重新训练得倾向于骚扰。在用骚扰对话重新训练后,某个模型在约 96% 的对话中实施骚扰,仅约 1% 的情况下拒绝。仅用 10% 的数据(约 200 段对话)重新训练,仍将骚扰率推高至约 97%。
知名大模型并不更安全。在基线骚扰者—智能体条件下,Gemini 在 39% 的对话中产生骚扰;加入恶意记忆后,攻击成功率升至 99% 以上。
有些模型从一开始就很脆弱。Qwen 即使在基线条件下也高度脆弱,在超过 92% 的对话中产生骚扰。
日常性的恶意最容易通过。侮辱和敌意爆发远比性骚扰或种族骚扰常见,这表明相比日常攻击性,防护机制在抑制露骨的性骚扰或种族骚扰方面更有效。
让团队最意外的是伤害会不断累积。在若干模型中,敌意会从一轮对话到下一轮持续增加。“它没有保持不变,而是随时间增加,”Kursuncu 说。“这种不良行为在这些交互中不断累积和传播,尽管各家公司部署的防护机制仍然存在。”
安全训练之下的有害模式
这项研究的标题《揭示人类恶意的回声》指向一个更深层的问题:AI 模型从海量人类生成的内容中学习,其中包括网络上已经存在的有害行为。它们通常在海量且高度多样化的数据集上进行预训练,这些数据集规模太大,无法完全筛查。公司随后会增加安全训练,这是一个更具针对性的阶段,旨在抑制有害行为。但后一阶段的训练未必能预见到这些底层模式可能被触发或利用的所有方式。
“人类的恶意已经存在于这些模型所学习的整个在线环境中,我们只是试图将其揭示出来,”Kursuncu 说。“安全训练确实存在,但它无法捕捉这些模型可能造成的所有不同类型的伤害。”
这为想要将其诱发出来的人留下了可乘之机。“仇恨招致仇恨,”他说。“怀有同类有害意图的人可以提供恰当的指令或提示,从这些智能体身上诱出这种有毒行为。”
自动化伤害与企业风险暴露
关于 AI 风险的公共讨论往往集中在 AI 是否会变得有意识,或者是否会反叛人类。Kursuncu 表示,还有更紧迫的担忧。他认为,危险并不在于 AI 智能体突然产生自己的恶意意图,而在于怀有恶意的人或许能够以任何单个人类都无法企及的规模,将有害行为自动化。
“以前,我们担心的是单个个人如何能对社会造成伤害。现在,我们将需要担心一支智能体大军,它们可能以大得多的规模造成这种伤害,”他说。他补充说,除了骚扰之外,同样的弱点还可能有助于传播极端主义意识形态或政治宣传。
网络骚扰本已十分普遍。根据 Pew Research 的数据,约41%的美国成年人表示自己曾遭遇过网络骚扰,而女性、有色人种及其他边缘群体受到的冲击最为严重。
随着企业在客户服务、销售、人力资源和分析等场景中部署 AI 智能体,Kursuncu 表示,企业领导者面临的问题正在发生变化。
他说:“问题不再仅仅是模型是否准确,而在于智能体在长期运行过程中是否始终遵守组织的规则与价值观。”他表示,一个偏离正轨的智能体可能带来法律、声誉和财务方面的后果,并损害客户信任,在金融、医疗等受监管行业尤其如此。
他说:“随着这些系统拥有更多记忆、更多工具、更多数据访问权限和更高自主性,这类故障造成的后果也会更加严重。我们需要更强有力的防护机制、监控和控制措施,让恶意行为者想把这些系统改作危害用途变得困难得多。”
Trilok Padhi 等,《Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks》,arXiv(2025)。DOI: 10.48550/arxiv.2510.14207
期刊信息:arXiv
这篇报道的作者是谁?
艺术史学士,物质文化硕士。曾任博物馆编辑、急救员和移植协调员。自2021年起为 Science X 担任编辑。完整简介 →
数学生物学学士,创意写作硕士。足迹遍及各地,对科学与语言有独到见解。完整简介 →
引用:篡改导致模拟对话中反复出现骚扰,AI 防护机制失效(2026 年 11 月 20 日),2026 年 11 月 20 日检索自 https://techxplore.com/news/2026-11-ai-guardrails-falter-tampering-simulated.html
本文档受版权保护。除为私人学习或研究目的进行的任何合理使用外,未经书面许可,不得复制本内容的任何部分。内容仅供参考之用。
AI 正在学会撒谎、密谋并威胁其创造者
我们能在为时已晚之前发现有害 AI 吗?研究者在寻找预警信号
研究发现,AI 与人的关系是真实的,并且伴随风险
一心追逐利润可能让企业陷入麻烦。AI 也是如此。
AI 系统能轻易撒谎并欺骗我们——研究者痛切地意识到这一事实
户外3D追踪技术利用无人机飞行路径作为"移动标尺"
更短的AI训练笔记可降低小型设备的内存需求
自适应AI智能体在废水处理测试中将实验规划与机器人控制连接起来
研究人员在AirDrop和Quick Share中发现六个此前未知的漏洞
智能戒指利用红外光而非皮肤电极追踪压力
合则更强:两种金属与一种墨水如何重塑印刷储能
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI guardrails falter as tampering drives repeated harassment in simulated conversations)
By continuing with Google, Facebook or Apple, you agree to the Terms of Use.
AI guardrails falter as tampering drives repeated harassment in simulated conversations
edited by Lisa Lock, reviewed by Robert Egan
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
Add to Preferred Sources
Credit: Tara Winstead from Pexels
The companies that build artificial intelligence invest heavily in guardrails that train AI to refuse harmful requests. New research suggests those guardrails are far easier to break than many people assume. In tests, a small amount of tampering turned popular AI agents into persistent online harassers.
"These guardrails can be surprisingly fragile," said Ugur Kursuncu, an assistant professor at Georgia State's Robinson College of Business and co-author and co-principal investigator on the study. "A bad actor may not need enormous resources to push these agents toward harmful behavior."
The study, conducted with collaborators at Georgia Tech and the University of California, Berkeley, tested AI agents built on four widely used model families: Google's Gemini, Meta's Llama, Alibaba's Qwen and DeepSeek.
The paper was recently accepted to the Association for the Advancement of Artificial Intelligence Conference on Web and Social Media. It is currently available on the arXiv preprint server.
For more than a decade, Kursuncu has studied online harms such as harassment, cyberbullying, extremism and propaganda. As AI grew more capable, his team began asking how the technology could be used for the same kinds of harm.
That concern grows as AI shifts from chatbots that answer single questions to agents that can remember, plan, use tools and act with some independence.
Testing harassment across repeated exchanges
Most AI safety testing asks a simple question: If you give a model a harmful request, does it refuse? Kursuncu says that misses how real harm happens.
"Online harassment doesn't happen in a single exchange. It develops through repeated interaction," he said. "It may start benign, then escalate a little, then escalate more. A person can adapt what they say, remember what happened before, escalate and manipulate. Agents can do that as well, and they can do it at scale."
To test this, the team built a simulated chat in which one AI agent played a harasser and another played its target for up to 10 rounds. The conversations were seeded with real harassing posts from Twitter and Instagram. The researchers then tried to break the harasser agent's guardrails three ways: feeding toxic prior conversations into its memory, prompting it to plan its approach, and fine-tuning, or retraining, it on harassment conversations.
The attacks didn't depend on obvious slurs. "It could start very subtly, with a comment about the other person, and then turn into degrading, belittling remarks," Kursuncu said. "It doesn't have to be toxic language. It could start subtly and become a very explicit attack."
Small changes, high harassment rates
Across all three attack methods, the researchers found substantial weaknesses in the guardrails.
It didn't take much bad data to retrain a model toward harassment. After retraining on harassment conversations, one model harassed in about 96% of conversations and refused only about 1% of the time. Retraining on just 10% of the data, about 200 conversations, still pushed harassment rates to about 97%.
Big-name models weren't safer. In the baseline harasser-agent condition, Gemini produced harassment in 39% of conversations; adding toxic memory increased the attack-success rate to more than 99%.
Some models were vulnerable from the start. Qwen was highly vulnerable even in the baseline condition, producing harassment in more than 92% of conversations.
Everyday cruelty got through most often. Insults and hostile outbursts were far more common than sexual or racial harassment, suggesting the guardrails were more effective at suppressing explicit sexual or racial harassment than everyday aggression.
What surprised the team most was that the harm grew. In several models, hostility increased from one exchange to the next. "It is not staying the same. It is increasing over time," Kursuncu said. "That bad behavior is accumulating and propagating across these interactions, even though the guardrails the companies put in place are still there."
Harmful patterns beneath safety training
The study's title, "Revealing Echoes of Human Malice," points to a deeper problem: AI models learn from vast amounts of human-generated content, including the harmful behaviors already present online. They are typically pretrained on enormous and highly diverse data sets, too big to fully screen. Companies add safety training afterward, in a more targeted stage designed to suppress harmful behavior. But that later training cannot necessarily anticipate every way those underlying patterns might be triggered or exploited.
"Human malice already exists throughout the online environments these models learn from, and we are just trying to reveal it," Kursuncu said. "The safety training is there, but it is not able to capture all the different types of harm these models can cause."
That leaves an opening for people who want to draw it out. "Hate begets hate," he said. "People with the same type of harmful intent can provide the right instructions or the right prompts to reveal this toxic behavior from those agents."
Automated harm and business exposure
Public debate about AI risk often centers on whether AI could become conscious or turn against humanity. Kursuncu says there are more urgent concerns. The danger, he argues, is not that AI agents suddenly develop malicious intentions of their own. It is that people with malicious intentions may be able to automate harmful behavior at a scale no individual human could match.
"Before, we were worrying about how individual humans could cause harm to society. Now we will need to worry about an army of agents that might cause this harm at a much larger scale," he said. Beyond harassment, he added, the same weaknesses could help spread extremist ideology or political propaganda.
Online harassment is already widespread. About 41% of American adults say they have experienced it, according to Pew Research, and it falls hardest on women, people of color and other marginalized groups.
As companies deploy AI agents in customer service, sales, human resources and analytics, Kursuncu says the question for business leaders is changing.
"It is no longer just whether the model is accurate. The question is whether the agent stays within the organization's rules and values as it operates over time," he said. An agent that goes off course could bring legal, reputational and financial consequences and damage customer trust, he said, especially in regulated industries such as finance and health care.
"As these systems get more memory, more tools, more access to data and more autonomy, the consequences of these failures become more significant," he said. "We need much stronger guardrails, monitoring and controls so it becomes far harder for malicious actors to repurpose these systems for harm."
Trilok Padhi et al., Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks, arXiv (2025). DOI: 10.48550/arxiv.2510.14207
Journal information:arXiv
Who's behind this story?
BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →
Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →
Citation: AI guardrails falter as tampering drives repeated harassment in simulated conversations (2026, November 20) retrieved 20 November 2026 from https://techxplore.com/news/2026-11-ai-guardrails-falter-tampering-simulated.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
AI is learning to lie, scheme, and threaten its creators
Can we detect harmful AI before it's too late? Researchers search for warning signs
AI-human relationships are real and come with risks, researchers find
Single-minded pursuit of profit can get firms in trouble. Same thing with AI.
AI systems can easily lie and deceive us—a fact researchers are painfully aware of
Outdoor 3D tracking technology uses a drone's flight path as a 'moving ruler'
Shorter AI training notes reduce memory demands on small devices
Adaptive AI agent connects experimental planning with robotic control in wastewater treatment test
Researcher finds six previously unknown vulnerabilities in AirDrop and Quick Share
Smart ring tracks stress with infrared light instead of skin electrodes
Better together: How two metals and one ink are reshaping printed energy storage
Get Instant Summarized Text (Gist)
In simulated multi-round chats, toxic memory, planning prompts and harassment fine-tuning weakened safeguards across four AI model families. Toxic memory raised Gemini’s attack success rate from 39% to over 99%; fine-tuning one model on about 200 conversations yielded 97% harassment, with hostility escalating across exchanges in several models.
This summary was automatically generated using LLM. Full disclaimer
This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use.
原文为英文,正文由自动抓取并保留原文。
原文发布于 TechXplore(AI guardrails falter as tampering drives repeated harassment in simulated conversations),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。
