AI 智能体如何沦为攻击者的工具,又如何防范

保护 LLM 智能体免受提示注入与数据泄露威胁
AI 正在成为网络安全领域最热门的话题。各地的 CISO 都在向民调机构和采访者表示,他们对 AI 可能以各种方式让所在组织更脆弱、更不安全的担忧。AI 可能成为攻击的目标,也可能成为加速并助力攻击者的工具。它还可能只是带来引入意外错误的机会,从而导致无法预见的风险。
然而,LLM 和 LLM 智能体系统同时正越来越多地融入各种各样的业务任务中。我们这些身处机器学习/AI 工程领域的人,正被要求为各行各业的众多不同用例构建这类系统。
例如,它们可用于:
让内部数据或文档更易于查找和检索
让用户通过聊天机器人提交支持工单
合同与法律文件审阅支持
服务中断的事件分诊与响应
自主合规监控
人力资源任务,涵盖入职、福利和薪酬等领域
会计,包括报表、费用管理和对账
客户服务与客户关系管理(CRM)
销售自动化及客户记录访问
商业智能与分析
IT 管理与基础设施管理
网络安全运营,包括威胁检测和自动化修复
……这样的例子还有很多很多,一个接一个,没完没了。
可以想见,这其中的许多任务意味着 AI 正在访问你所拥有的最敏感、最有价值的业务数据。在追求业务效率的过程中,我们对 AI 应用和工具寄予了极大的信任。问题在于:我们应当如何构建治理机制、做出开发选择,才能让 AI 的使用尽可能安全?
在接下来的两个月里,我将分享一些笔记,谈谈组织在 AI 开发、部署和实施的安全方面需要警惕的场景。在第一部分,我们将讨论原本无害的智能体如何可能被不法分子改作从事不良活动的工具。
AI 极易被利用或被劫持,用于恶意的网络安全活动。就像一条非常友善的看门狗,AI 实际上会在无意中、却十分积极地为你组织带来危险的人开门,因此你需要提前做好防护。
数据访问与知识库整理
内部 AI 工具能帮助你的团队更快找到完成工作所需的数据,但要知道,攻击者同样能从中获益。举例来说,如果你在内部构建 AI 工具,帮助员工快速查找文件、访问数据,那么一个想要收集并窃取数据的内部威胁行为者同样可以得到该 AI 工具的助力,而他们只需一组被泄露的凭据就能做到。这些数据可能暴露内部架构,起到侦察作用,也可能泄露对黑客具有金钱价值的敏感数据。
管理 AI 智能体可以访问的数据,是最大限度降低这类风险的关键。使用 RAG 或其他类似技术进行数据检索时,通常需要先对数据进行汇总和整合,然后才会返回结果。当 Bob 问“Jenkins 这个客户的历史情况是怎样的?”,这可能涉及多份包含不同类型信息的文档——合同、通话录音、发票、内部 Slack 消息或电子邮件等等。
不过,这些数据要素的敏感程度各不相同。也许那些内部沟通信息本来就不是所有人都能看的,如果这个请求是 Susan 提出的,你就不希望这些信息出现在回答中,但因为 Bob 是副总裁,他可以看到这些内容。你现在可能已经明白了:每份文档对应的用户权限都不一样,因此你需要在文档层面把这些权限明确指定出来。
别搞错了,为模型可用的每一条数据确定正确的访问权限可能极具挑战性。如果你的 LLM 为了回答一个问题而从知识库中访问五个条目,你需要确保你的预过滤系统会遵循这五个条目各自不同的访问权限。你不能仅仅通过指令来依赖 LLM 区分权限,因为它极有可能失败。要确保访问得到妥善保护,唯一的方法是在上游设置防护机制。如果某个文件被认为比其他文件更敏感,并且超出了用户的访问权限,那么当该用户提问时,LLM 绝不能从知识库中收到该文件,否则它在整合信息以返回给用户时可能会使用它。这是设计信息检索 LLM 系统时的一个架构挑战,有时会被忽视,但实际上非常重要。把所有文件一股脑倒入知识库、让 LLM 自己去理清,这很诱人,但你最终会让自己陷入严重的访问泄露。
正如我已经提到的,在能够妥善保护信息之前,你必须先弄清楚哪些数据应该对谁可见。某些数据在与其他要素结合后可能会变得更为敏感。例如,如果智能体能够访问各项目的人员排班表以及这些项目的预算,用户就可能要求 LLM 反推出员工的预估薪资。这凸显了仔细考量 LLM 将能访问的每一项数据要素和每一份文档的重要性,而不是在没有经过筛选整理的情况下,就把整个 wiki 或整个共享盘一股脑倒进 LLM 的知识库。妥善管理文档是解决这类问题的真正关键。智能体访问的任何文档都应连同相关元数据一起存储,元数据要清楚标明该文档的敏感级别,以及不同用户应被授予何种访问权限。
如上所述,外部攻击者同样可以通过 AI 获取数据,用于侦察。对内部用户来说可能并不特别敏感的文档和数据,却可能成为攻击者熟悉该系统及其结构的极其有效的手段,从而使攻击更快、潜在破坏性更大。
现在,假设你已经完成了数据的编目与治理,接下来我们需要考虑用户权限。设想有这样一位用户:出于有意设计,他无权访问最敏感的数据。我们此前已经讨论过,他们可能通过 AI 智能体获取到自身权限范围之外的信息,但风险同样存在于“操作”层面。例如,如果 Bob 联系上具备执行系统或网络操作能力的内部 IT 智能体,要求更改某个特定文件的权限,那么他(或者一个冒充 Bob 的攻击者)就可能把自身的访问权限扩展到超出你预期的范围。与之类似,更改用户权限是 IT 职责中的常规工作,但你也清楚,这种能力应当只限于极少数管理员使用。
让智能体来帮忙处理这类任务、为系统管理员分担工作,可能很有诱惑力,但从第一天起就必须配套强有力的防护。正如数据访问一样,智能体的操作也需要内建权限检查,以免智能体做出用户本人无权做的事情。智能体可能拥有执行脚本、调用 API 或管理部署流水线的权限,而这些能力可能为其提供间接访问用户原本无法触及的系统的途径。
为缓解这一问题,理想情况下,你不应仅仅信任智能体会遵照指令行事,而应设置更严格的防护机制。当仅具只读权限的用户使用智能体时,涉及写入数据的工具和技能应被禁用。不要仅仅因为指令如此规定,就相信智能体不会调用这些能力,而应把这种机会(或许可以说是诱惑)直接移除。除此之外,还有些操作你会认定其危险程度过高,不能交由任何智能体自行处理,必须要求人工审核。这条界线划在哪里取决于你的业务和风险承受能力,但你必须做出这一决定,并据此落实相应的防护机制。
在设计 LLM 智能体时,还有一些非常出色的架构技术有助于降低风险,这也进一步说明,在开发过程中把安全作为一等考量因素是何等重要。
提示注入/智能体操控
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(How AI Agents Become a Tool for Attackers, and How to Stop It)
Securing LLM Agents Against Prompt Injection and Data Leaks
AI is turning into the hottest topic in cybersecurity. CISOs everywhere tell pollsters and interviewers how concerned they are about various ways that AI may be making their organizations more vulnerable and less safe. AI can be the target of attacks, or it can be a tool to accelerate and enable attackers. It can also just present an opportunity for unintended errors to be introduced, leading to unforeseen risk.
However, LLMs and LLM agent systems are simultaneously becoming more and more integrated into myriad business tasks. Those of us in the machine learning/AI engineering field are being tasked with building these systems for many different use cases across industries.
For example, these can be useful for:
making internal data or documents easier to find and search
letting users submit support tickets through a chatbot
Contract and legal document review support
Incident triage and response for outages
Autonomous compliance monitoring
HR tasks in areas such as onboarding, benefits, and payroll
Accounting, including reporting, expense management, and reconciliation
Customer service and customer relationship management (CRM)
Sales automation and access to customer records
Business intelligence and analytics
IT administration and infrastructure management
Cybersecurity operations, including threat detection and automated remediation
… and the list goes on, and on, and on.
As you might imagine, lots of these tasks mean AI is accessing the most sensitive and valuable business data you have. In our pursuit of business efficiencies, we’re putting a lot of trust in AI applications and tools. The question is: how we should be structuring governance and making development choices, in order to make our AI usage as safe as possible?
In the next two months, I’m going to be sharing some notes about the scenarios organizations need to be watching out for when it comes to security in AI development, deployment, and implementation. For this first part, we’ll be discussing how normally benign agents can be repurposed by bad actors as tools for unsavory activity.
AI is keenly susceptible to being used or hijacked for malicious cybersecurity activities. Kind of like a very friendly guard dog, AI can actually unwittingly but eagerly open the door for people who pose a danger to your organization, so you need to have protections in place ahead of time.
Data Access and Knowledge Base Curation
Internal AI tools can help your teams find the data they need to do their work much faster, but recognize that this goes for attackers too. For example, if you are building AI tools internally to help your staff find files and access data swiftly, an internal threat actor looking for ways to collect and exfiltrate data can equally be aided in their efforts by the AI tool, and all they need is a set of compromised credentials to do it. That data could reveal internal architecture, serving a reconnaissance purpose, or it could uncover sensitive data that has monetary value to a hacker.
Managing the data that your AI agents have access to is key to minimizing this kind of risk. Data retrieval with RAG or other similar techniques usually requires data to be summarized and consolidated before it is returned. When Bob asks “What is the history of the Jenkins account?” that might involve multiple documents with varying kinds of information — contracts, call recordings, invoices, internal Slack messages or emails, and so on.
These data elements have different degrees of sensitivity, however. Maybe the internal communications weren’t meant for everyone to see, and if Susan made this request, you would not want that information to be part of the response, but because Bob is a VP, he can see them. You might be getting the picture now that the user permissions are different for every document, so you need to have these specified at the document level.
Make no mistake, determining the correct access permissions for every piece of data available to the model can be deeply challenging. If your LLM accesses five items from your knowledge base in order to produce an answer to a question, you need to ensure that your pre-filtering system will observe the distinct access permissions of each of the five. You can’t just rely on the LLM to differentiate the permissions, through instructions, because it’s too likely to fail. The only way to be sure access is properly protected is by setting the guardrails upstream. If one file is considered more sensitive than the rest, and is beyond the access permissions of the user, the LLM must not receive that from the knowledge base when that user is asking, or it might use it when consolidating information to return to the user. This is an architectural challenge for designing information retrieval LLM systems that is sometimes overlooked, but is actually very important. It’s tempting to just dump all your files into the knowledge base and let the LLM sort it out, but you are setting yourself up for a serious access breach eventually.
As I’ve already alluded to, you need to know what data should be visible to whom before you can properly defend your information. Some data might become more sensitive in combination with other elements. For example, if the agent has access to staffing schedules for projects and the budgets for those projects, a user might be able to ask the LLM to reverse engineer estimated salaries for the staff members. This highlights the importance of considering carefully each data element and document the LLM will have access to, instead of just dumping an entire wiki or a whole shared drive into the LLM’s knowledge base without curating it. Managing documents well is a real key to solving these kinds of issues. Any documents that your agent accesses should be stored alongside the relevant metadata, which makes clear what the document’s sensitivity is and what access different users should be given.
As mentioned above, external attackers can access data via AI to use for reconnaissance, as well. Documents and data that might not be particularly sensitive for internal users could be extremely effective ways for the attacker to get familiar with the system and its structure, making the attack faster and potentially more damaging.
Now, assuming you’ve got your data catalogued and curated, we need to consider the user permissions. Envision a user who doesn’t have access to the most sensitive data, on purpose. We’ve already discussed how they could get information that is outside their access from an AI agent, but there’s also a risk around actions. For example, if Bob reaches out to the internal IT agent that has the capability to take system or network actions, and asks for permissions to be changed on a particular file, he (or an attacker pretending to be Bob) could extend their access beyond what you intended. Similarly, changing user permissions is a routine part of IT responsibilities, but you know that this capability should be limited to a very small set of administrators.
It may be tempting to let your agent help with such tasks, to take work off the plate of your system administrators, but significant protection has to be involved from day one. Just like with data access, agentic actions need to have permissions checks built in, so the agent doesn’t do something the user wouldn’t be empowered to do themselves. An agent might have permission to execute scripts, interact with APIs, or manage deployment pipelines, and those capabilities could provide indirect access to systems that the user couldn’t otherwise reach.
To mitigate this, ideally you don’t want to just trust the agent to behave according to instructions, but have firmer guardrails. When a user with read only permissions uses the agent, tools and skills that involve writing data should be disabled. Don’t just trust the agent to not call those capabilities because the instructions say so, but take the opportunity (perhaps temptation) away. Beyond that, there are some actions that you will decide are just too dangerous to be left to any agent, and human review must be required. Deciding where that line is depends on your business and your risk tolerance, but you need to make that decision and implement the guardrails accordingly.
There are also some very good architectural techniques for the design of your LLM agents that can be helpful in reducing risk, and this reinforces the importance of having security as a first class consideration in the development process.
Prompt Injection/Agent Manipulation
Even the most well-prompted and tuned LLM has some capacity to be manipulated. In fact, attempting to manipulate model behavior by providing malicious instructions in prompting is a well established technique for attacking deployed LLMs and agents. Prompt injection, where instructions are passed in an obfuscated way that is hard for humans to detect or interpret, but is very easy for LLMs to understand, can lead to assorted kinds of LLM behavior that you would not expect.
Even when the user doesn’t mean to do anything wrong, or an attacker can’t directly access your agent, an unwitting user can pass doctored data or a fraudulent document to an AI agent without careful vetting, and that object could contain malicious instructions the LLM would act upon.
If models are not carefully structured and heavily tested, models are at high risk of releasing sensitive data and/or taking inappropriate actions. This again is why I encourage you to make permissions upstream settings, instead of just relying on your agent to make the right decisions. Take manipulative actions off the table wherever possible, so that even if your agent is confused or fooled, it can’t do major harm.
You can reduce this kind of risk by making your agent’s workflow for ingesting documents and data very cautious. New files and data should be viewed as insecure until proven otherwise, and scanning/checking uploaded materials is important. You should also perform testing on your own, essentially red-teaming your agent. Try to find the boundaries where the agent performance degrades, especially lengthy exchanges where the prompt instructions can be overwhelmed by the volumes of conversation material.
Introducing Vulnerability
Finally, a brief note: AI is an integral part of software development work today, and your dev teams are certainly employing it in their daily workflows. However, sloppy AI development practices can lead to vulnerabilities or security flaws being introduced into your codebases without anyone’s knowledge, and without any malicious intention. This can take the form of AI generated code containing unexpected security flaws, configuration or infrastructure setup mistakes, or the speed of development resulting in insufficient testing, among other things. While this isn’t exactly a case of AI being used as a tool for malicious intentions, it’s adjacent and worth keeping in mind. Ensure that code being produced with AI tools is tested and validated thoroughly to make sure security vulnerabilities are not being introduced by accident.
Next month, I will continue on by discussing ways that AI can become a target of bad actors in cybersecurity, and how we can protect systems and prepare ourselves to respond in the case of disaster.
Read more of my work at www.stephaniekirmer.com.
An example of prompt injection manipulating tool selection: https://arxiv.org/abs/2504.19793
Can we explain the predictions our model makes? Let's see why our model predicted an image to be part of a certain class
November 23, 20214 min read
Using autoresearch to optimize marketing campaigns under budget constraints
April 28, 202612 min read
What worked, what broke, and why I did it
December 15, 20255 min read
August 23, 202111 min read
Why MCP servers keep losing to CLIs once the agent gets a terminal
The math behind Linear Regression and the Python way of implementation
October 18, 20206 min read
ARTIFICIAL INTELLIGENCE First, we had GPT-3. Now we have LaMDA. In mid-2020 OpenAI presented the...
A selection of resources that explain complex concepts regardless of readers' technical background
September 8, 20225 min read
I saved my day with a mere $52 bill. But you don't have to repeat my mistake.
September 7, 20266 min read
原文为英文,正文由自动抓取并保留原文。
原文发布于 Towards Data Science(How AI Agents Become a Tool for Attackers, and How to Stop It),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。
