When open source bites back: Data and model poisoning
5 minute read time
Artificial intelligence (AI) continues to redefine what is possible in software, from predictive models to generative content. But as AI systems grow in power, so too do the threats targeting their foundations, including a particularly insidious category: data and model poisoning.
In 2025, the Open Worldwide Application Security Project (OWASP) Top 10 for Large Language Model (LLM) Applications identified "Data and Model Poisoning" as a key vulnerability in software development. This type of attack undermines trust at the most foundational level, by injecting malicious data into training or fine-tuning pipelines, or altering model weights to bias outputs.
This blog post is the second in a four-part series exploring how Sonatype helps organizations address key OWASP LLM Top 10 categories — this time by defending against data poisoning and model poisoning attacks.
In a recent webinar on AI risk and governance, Tyler Warden, Senior Vice President of Product at Sonatype, explained why we prioritize a focused approach:
"At Sonatype, we're not trying to tackle all ten categories — and that's intentional. We wake up every day laser-focused on helping organizations govern the AI they choose to bring into their products. That's why we're highlighting these four categories."
What is data poisoning in the context of AI?
Data poisoning occurs when malicious actors intentionally manipulate the training data used to develop machine learning (ML) models.
By introducing corrupt or adversarial data into the dataset, they can skew model behavior, bias outputs, or even implant backdoors into deployed AI systems. Similarly, model poisoning involves direct tampering with model weights or architecture, often in pretrained models downloaded from third-party repositories.
According to OWASP, common examples include:
-
Injecting harmful training samples into publicly sourced datasets.
-
Publishing tainted pretrained models with subtle backdoors.
-
Biasing model outputs via adversarial fine-tuning data.
-
Targeted poisoning that impacts only specific inputs or conditions.
These attacks are particularly hard to detect, and their impacts range from degraded performance to deliberate misinformation or unauthorized access.
Why LLMs are uniquely vulnerable to model poisoning
LLMs rely heavily on large-scale, often publicly sourced data. They frequently incorporate pretrained models from open source repositories and apply fine-tuning using custom datasets.
This open and decentralized development process introduces unique risks:
-
Lack of provenance and validation: Developers may have limited visibility into the origin and integrity of training data or pretrained weights.
-
Dependency on third-party sources: Public model hubs and datasets increase exposure to tampered assets.
-
Silent manipulation: Poisoned inputs can have targeted effects that evade traditional validation or evaluation processes.
The result? A poisoned LLM can behave incorrectly in specific scenarios, or serve as an attack vector itself.
How Sonatype helps detect and prevent AI model poisoning
The Sonatype platform is designed to secure the modern software supply chain, and that includes the AI/ML components used in LLM development.
From dataset curation to dependency validation, Sonatype offers multiple layers of defense against data poisoning attacks and model poisoning threats.
AI-aware malware detection and research
Sonatype's malware detection uses a combination of automated behavioral analysis and human-in-the-loop validation to detect:
-
Backdoored machine learning packages
-
Counterfeit components impersonating trusted ML libraries
-
Packages exhibiting signs of obfuscation, evasion, or payload delivery
As detailed in our Open Source Malware Index Q1 2025, we uncovered over 18,000 malicious open source packages, many targeting AI ecosystems like PyTorch, TensorFlow, and Hugging Face. This level of threat research is critical in identifying poisoning attempts that originate from malicious package uploads.
Behavioral and reputation-based scoring
Sonatype evaluates packages using a proprietary reputation system. Factors like versioning irregularities, novel obfuscation methods, dependency graph anomalies, and unusual publication behavior help flag suspicious AI components, even when traditional signature-based tools fail.
This approach is vital when screening pretrained models or training data preprocessing tools, which are increasingly targeted for model poisoning.
Full visibility with software bills of materials (SBOMs) for AI projects
With Sonatype SBOM Manager, development teams gain full insight into every component — including ML libraries and data processing utilities — used in the AI pipeline.
This allows teams to:
-
Track the provenance of models and dependencies
-
Validate integrity across environments
-
Respond quickly to poisoning disclosures or zero-day threats
SBOMs also play a critical role in audit readiness, enabling organizations to show evidence of due diligence in model sourcing and training practices.
Policy enforcement to keep tainted data out
Sonatype Lifecycle empowers organizations to define and enforce policies that block:
-
Suspicious or unverified open source packages
-
Specific maintainers or domains linked to prior poisoning attempts
-
Components that do not meet internal vetting standards
This policy-driven governance prevents data poisoning threats from ever entering your pipeline, whether at build time or during runtime updates.
Data and model poisoning is a growing risk in AI development
The threat of poisoning is no longer theoretical. Real-world examples have shown how poisoned datasets and tampered models can silently distort outputs, sabotage downstream applications, or create backdoors that persist undetected. These attacks are difficult to identify and even harder to remediate after deployment.
In the AI era, security must begin not just with your code, but with your data.
Sonatype is your partner in trustworthy AI
At Sonatype, we help organizations secure every layer of their AI development pipeline. Whether you are fine-tuning models for enterprise use or deploying LLM-powered tools at scale, our platform offers the visibility, automation, and threat intelligence needed to stop data poisoning before it starts.
By combining software composition analysis (SCA), behavioral malware detection, and SBOM governance, we help you protect the integrity of your AI models — from training to production.
Explore our AI security capabilities and learn more about how to accelerate innovation with AI/ML.
Aaron is a technical writer on Sonatype's Marketing team. He works at a crossroads of technical writing, developer advocacy, software development, and open source. He aims to get developers and non-technical collaborators to work well together via experimentation, feedback, and iteration so they ...
Explore All Posts by Aaron Linskens
Try Nexus Repository Free Today
Sonatype Nexus Repository is the world’s most trusted artifact repository manager. Experience the difference and download Community Edition for free.
