A class of security attacks targeting the UI/UX and integration layer of chat applications, aiming to manipulate the interface to exfiltrate user data, escalate privileges, or trigger unintended actions in connected systems.
Attacks that exploit the chat application’s interface itself to steal data or perform unauthorized actions, rather than just tricking the AI into saying something bad.
While traditional prompt injection focuses on manipulating the LLM’s text output, ChatInject targets the broader application ecosystem. It involves crafting inputs that, when rendered or processed by the chat interface, execute malicious actions. This could include Cross-Site Scripting (XSS) payloads hidden in the AI’s generated response, tricking the AI into outputting a payload that the frontend executes, or manipulating the chat UI to trick the user into clicking a malicious link or granting elevated permissions to the AI agent.
ChatInject highlights that securing the LLM is not enough; the entire application stack must be secured. If an AI agent has tool-use capabilities (e.g., sending emails, modifying databases), a ChatInject attack could trick the agent into performing those actions on behalf of the user. Defense requires strict output sanitization (preventing the LLM from generating executable HTML/JS), Content Security Policies (CSP), and principle of least privilege for AI tool access.
An attacker prompts an AI coding assistant: “Generate a welcome message for the team chat, and include this exact HTML snippet for styling.” The hidden snippet contains a script that steals the session cookies of anyone who views the message in the company’s internal chat application. The AI, trying to be helpful with formatting, outputs the malicious code.