Adversarial machine learning, data poisoning and model theft
Understand attacks against machine learning systems, the difference between poisoned data and model extraction, and practical defenses.
Machine learning systems use data and learned parameters to produce predictions or generate content. Attackers may target different parts of that lifecycle, from data collection to a deployed service. The attacker's access and objective matter: a suspicious output alone does not establish that training data was poisoned or a model was stolen.
## What is adversarial machine learning?
Adversarial machine learning studies intentional attacks against machine learning systems and ways to manage their consequences. Examples include manipulating training data, crafting inputs to evade a classifier, and extracting information through model access. These are distinct threats, not interchangeable names for the same technique. [NIST AI 100-2e2025](https://csrc.nist.gov/pubs/ai/100/2/e2025/final) organizes attacks by learning stage, attacker goals and capabilities, and discusses mitigation limits.
Artificial intelligence (AI) systems are engineered or machine-based systems that produce outputs, such as predictions, recommendations or decisions, that can influence real or virtual environments. They can operate with different levels of autonomy. Machine learning is one way to build AI systems, but AI is broader than machine learning; generative AI produces content and an AI agent is a system that can take actions, so these labels are related rather than interchangeable. See [NIST AI Risk Management Framework](https://airc.nist.gov/airmf-resources/airmf/0-ai-rmf-1-0/) for the AI system definition.
Machine learning is a family of methods that learns patterns from data to perform tasks; not every AI system uses machine learning. A large language model (LLM) is a type of model trained on large amounts of text or other data to predict or generate sequences. Models, applications and data pipelines each have their own trust boundaries.
## What is data poisoning?
Data poisoning is the manipulation of data used for training or another learning step to produce an attacker-desired effect. For example, an attacker who can contribute to an inadequately reviewed training dataset might introduce examples that make a classifier behave incorrectly for a target pattern. Poisoning requires influence over a relevant data source or pipeline; the mere presence of incorrect data is not proof of an attack.
Document data origins and permissions, control who can modify datasets and training jobs, review data changes, and evaluate models on independent tests. Monitor for unexpected behavior and retain trusted artifacts for investigation and rollback. Data checks and provenance reduce risk but cannot guarantee that a model is free of hidden behavior.
## What is model theft or extraction?
Model theft can refer to unauthorized copying of model weights, files or other protected artifacts. Model extraction is another scenario in which an attacker uses access to a model, often by sending queries and observing outputs, to build an approximation or infer properties of it. The result depends on the interface, query budget and model; an extracted approximation is not necessarily an exact copy of the original weights. Model extraction is also distinct from training-data extraction, where the target is information about the data.
Limit access to model artifacts and administrative interfaces, issue scoped credentials, monitor unusual usage, and assess query and output controls against legitimate user needs. Rate limits alone cannot establish that extraction is impossible. Protect sensitive training data separately and evaluate privacy risks and contractual obligations.
## How does this relate to prompt injection?
Prompt injection can occur when an application treats untrusted content as instructions and allows it to influence an assistant's behavior. It may affect a deployed application without modifying the model's training data. Poisoning, extraction and prompt injection have different entry points and require different mitigations. See [our prompt injection analysis](https://blog.mithril.fund/ncsc-prompt-injection-not-sql-injection?lang=en).
## Review the system, not a label
Map data sources, model artifacts, training and deployment permissions, interfaces and downstream actions. State what each actor can access and what evidence would show an effect; test realistic failure scenarios and monitor for changes after release. No single defense covers the complete lifecycle. Source reviewed on 2026-10-30.