What is AI data poisoning?
Data poisoning is the manipulation of data an AI system learns from or relies on, so that it behaves the way an attacker wants. It can target training and fine-tuning data, the documents in a retrieval system, or an agent's memory. The effects range from wrong answers on particular topics to hidden backdoors triggered by specific inputs.
Where poisoning happens
- Training and fine-tuning data. Corrupted or malicious examples included in the data a model is trained on. Listed as LLM04 in the OWASP Top 10 for LLM Applications.
- Retrieval sources. Documents added to a knowledge base, wiki, ticket system or website that a retrieval-augmented system searches.
- Agent memory. Content stored by an agent and reused in later sessions, listed as ASI06 in the agentic Top 10.
- Third-party models and datasets. Pretrained models and public datasets from sources you do not control.
What it achieves
- Wrong or biased answers on chosen topics.
- Hidden instructions that fire when certain words appear, a backdoor.
- Leakage, when poisoned content tells the model to reveal data.
- Gradual erosion of trust in the system.
Defences
- Control and log who can write to every data source a model uses.
- Track the provenance of models and datasets.
- Screen content entering retrieval stores.
- Monitor outputs for drift on sensitive topics.
- Include poisoning scenarios in testing. The HYDN AI Attack Engine supports poisoning tests, and HYDN’s Defensive AI Solution reports poisoning attempts as they happen.
Questions
Frequently asked questions
Do we need to worry if we don't train our own models?
Yes, if you use retrieval or agent memory. Anyone who can add or edit documents in a knowledge base your assistant searches can influence its answers.
How is poisoning detected?
Through provenance and change tracking on data sources, monitoring for unusual content entering them, and testing that checks answers on sensitive topics stay correct.
Published Updated