AI security risks explained prompt injection data leakage and model theft

Started by PostQuantum99, Today at 05:55 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Topic: AI security risks explained prompt injection data leakage and model theft   Views(Read 36 times)
Active members in this topic:
PostQuantum99(1)

PostQuantum99

TL;DR

  • Prompt injection tops the OWASP Top 10 for LLM Applications 2025 and remains the hardest AI security problem to solve completely
  • Indirect prompt injection hides malicious instructions in emails, web pages or documents that an AI system reads, which matters most for agents with tools
  • Data leakage happens through users pasting sensitive data, through models revealing training data and through AI features with excessive access, as the 2025 EchoLeak flaw in Microsoft 365 Copilot showed
  • Model theft covers stealing weights, extracting model behaviour through queries and unauthorised distillation
  • Research in 2025 found that as few as 250 poisoned documents could implant a backdoor in models of widely different sizes
  • Good defences combine least privilege, separation of trusted instructions from untrusted content, human approval for risky actions, monitoring and established guidance from OWASP, NCSC, CISA and NIST

Why AI security is different

Traditional application security assumes a clear boundary between code and data. Code runs, data is processed, and well written software never treats data as instructions. Large language models blur that boundary completely. Everything a model reads, whether a user's prompt, a document or a web page, arrives as text in the same stream, and the model decides what to do based on all of it

That design is what makes LLMs flexible, and it is also what makes them vulnerable. An attacker who can get text in front of a model may be able to influence its behaviour. As AI systems gain access to tools, files, email and business applications, the consequences of that influence grow. Security teams therefore need new mental models alongside familiar controls

This thread explains the three risk families that come up most often: prompt injection, data leakage and model theft. It also covers related threats such as data poisoning and insecure output handling, and closes with practical defences. Citations point to frameworks and research that security teams can use in their own policies. The tone is practical rather than alarmist, because these risks are manageable with good design

Prompt injection

Prompt injection means crafting input that overrides or subverts the instructions a model was given. The term was popularised by Simon Willison in September 2022, shortly after researchers showed that simple phrases could make GPT-3 based applications ignore their instructions. Direct prompt injection happens when a user types the malicious instructions themselves, often called jailbreaking when the goal is bypassing safety rules. It remains a moving target as models and attacks evolve

Indirect prompt injection is more dangerous for most organisations. A 2023 paper by Kai Greshake and colleagues showed that attackers could plant instructions in web pages, emails or documents that an AI system later reads. When an assistant summarises a malicious email or an agent browses a poisoned page, the hidden text can redirect it. The user may never see the attack, because it lives in content the AI processes on their behalf

The risk grows with capability. A chatbot that only produces text can be tricked into saying something misleading. An agent that can send email, move files or call APIs can be tricked into doing something harmful, such as forwarding confidential documents or changing settings. That is why OWASP lists prompt injection as LLM01, the top risk, in its Top 10 for LLM Applications 2025, and why its December 2025 Top 10 for Agentic Applications focuses heavily on goal hijacking and tool misuse

There is no complete fix yet. Model providers train models to resist injected instructions, and detection tools can catch many known patterns, but determined attackers keep finding new phrasings. Research approaches such as Google DeepMind's CaMeL design, published in 2025, separate trusted control flow from untrusted data so that injected text cannot change what the system does. Architectural defences like this are the most promising long term direction

Data leakage

Data leakage takes several forms. The most common is human: staff paste confidential information into AI tools that are not approved for it. In 2023, Samsung restricted generative AI use after employees reportedly shared sensitive source code with a public chatbot. Clear policies and approved tools with appropriate data protection terms address much of this risk

Models can also leak data they were trained on. Research led by Nicholas Carlini and colleagues has repeatedly shown that language models can memorise and regurgitate portions of their training data, including personal information. In 2023, a team demonstrated a simple attack that caused ChatGPT to output memorised training text. Providers have since added mitigations, but the lesson is clear: do not train or fine tune models on data you would be unwilling to see reproduced

AI features with broad access create a third path. In June 2025, researchers at Aim Security disclosed EchoLeak, a zero click vulnerability in Microsoft 365 Copilot tracked as CVE-2025-32711. A crafted email could cause Copilot to retrieve and exfiltrate sensitive organisational data without the user doing anything beyond normal work. Microsoft fixed the issue, but it illustrated how indirect prompt injection and data access combine into serious leaks

Retrieval systems need special care. When an AI assistant can search company documents, it may surface files that a user should not see if permissions are not enforced at retrieval time. OWASP's 2025 list includes vector and embedding weaknesses for this reason. Respecting existing access controls in every retrieval step is essential

Model theft

Model theft covers several threats to the models themselves. The most direct is stealing model weights, the numbers that define a trained model, through compromised infrastructure or insiders. Frontier model weights represent enormous investment and potential misuse risk, so leading developers treat them as crown jewels with strict access controls. Security frameworks published by AI companies and government bodies increasingly address weight protection

Model extraction is subtler. By sending many carefully chosen queries, an attacker can learn about a model's internals or train an imitation. In 2024, Carlini and colleagues showed they could recover part of the final layer of production models from OpenAI and Google through their public APIs, after which the providers changed their interfaces. Rate limits, monitoring for unusual query patterns and careful API design help reduce this risk

Distillation sits in a grey area. Training a smaller model on outputs from a larger one is a common, legitimate technique when done with permission. Doing it against a provider's terms to copy capabilities is widely considered misuse, and several AI companies have publicly accused others of unauthorised distillation. For businesses, the practical concern is protecting proprietary fine tuned models and ensuring vendors' terms are respected

Data poisoning and supply chain risks

Data poisoning means inserting malicious examples into training or fine tuning data so that a model learns hidden behaviours. In October 2025, a study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 malicious documents could implant a simple backdoor in language models across a wide range of sizes. The finding challenged the assumption that larger models trained on more data would dilute poisoned content. Data provenance and filtering are therefore important defences

Supply chain risks extend to models, datasets, plugins and libraries downloaded from public hubs. Malicious model files have been found on public repositories, sometimes exploiting unsafe serialisation formats. Safer formats such as safetensors, scanning tools and trusted sources reduce exposure. The same discipline that applies to software dependencies applies to AI components

AI coding assistants introduce a twist called package hallucination. Models sometimes suggest software packages that do not exist, and attackers can register those names with malicious code, a tactic nicknamed slopsquatting. Research published in 2024 and 2025 found that a significant share of package names suggested by code generating models in tests were hallucinated, around one in five in one large study. Verifying dependencies before installing them is the simple countermeasure

Insecure output handling and excessive agency

Outputs from AI systems should be treated as untrusted data. If an application inserts model output directly into a web page, a database query or a shell command, it can create classic vulnerabilities such as cross site scripting or command injection. OWASP calls this improper output handling. Standard input validation and output encoding apply just as they do for user input

Excessive agency is the related risk of giving AI systems more permissions, tools or autonomy than they need. An assistant that only needs to read a calendar should not be able to delete files. Every additional capability expands what an attacker can achieve through prompt injection. Least privilege is the single most effective control

Practical defences

  • Adopt approved AI tools with clear data protection terms and train staff on what not to share
  • Apply least privilege to every AI integration, with separate identities and narrow scopes
  • Separate trusted instructions from untrusted content and treat all retrieved text as potentially hostile
  • Require human approval for sensitive or irreversible actions taken by agents
  • Enforce existing access controls in retrieval and search features
  • Validate and encode model outputs before using them in other systems
  • Log prompts, tool calls and outputs for monitoring and incident response, with privacy safeguards
  • Use trusted model sources, safe file formats and dependency verification
  • Red team AI features before release and after major changes

Guidance worth knowing

Several authoritative frameworks help organisations structure their defences. In November 2023, the UK NCSC and US CISA published Guidelines for Secure AI System Development, endorsed by agencies from many countries. The UK government published an AI Cyber Security Code of Practice in January 2025, which informed a European standard from ETSI. NIST maintains a taxonomy of adversarial machine learning attacks and mitigations, and MITRE ATLAS catalogues real world tactics against AI systems

OWASP's two lists, the Top 10 for LLM Applications 2025 and the Top 10 for Agentic Applications, are practical starting points for development teams. They translate research into checklists that fit existing secure development processes. Mapping your AI features against them is a good first exercise. It often reveals quick wins

Keeping it in perspective

AI security can sound overwhelming, but most risks map onto principles security teams already know. Least privilege, input validation, supply chain hygiene, monitoring and human oversight all carry over. What changes is that language itself becomes an attack vector, which demands extra care wherever untrusted content meets capable models. Organisations that apply familiar discipline thoughtfully are well placed

Defensive AI is improving too. Security teams use AI to analyse alerts, review code and detect anomalies faster than before. The same technology that creates new risks also strengthens defenders when used carefully. That balance is a reason for optimism

Closing thoughts

Prompt injection, data leakage and model theft are the defining security challenges of the AI era so far. None has a single silver bullet, but layered defences dramatically reduce risk. As agents gain more autonomy, careful scoping and oversight become even more important. The frameworks and research cited here give every team a solid foundation to build on

Please share your own experiences in the replies, especially red team findings and controls that worked well. Real examples help everyone calibrate. Remember to remove sensitive details before posting



Key references for citation

  • OWASP Top 10 for LLM Applications 2025; OWASP Top 10 for Agentic Applications, December 2025
  • Greshake and colleagues, Not what you have signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, 2023
  • Aim Security, EchoLeak disclosure (CVE-2025-32711), June 2025
  • Carlini and colleagues, Stealing Part of a Production Language Model, 2024
  • Anthropic, UK AI Security Institute and Alan Turing Institute, study on poisoning attacks with a small number of documents, October 2025
  • Debenedetti and colleagues, Defeating Prompt Injections by Design (CaMeL), 2025
  • UK NCSC and US CISA, Guidelines for Secure AI System Development, November 2023