- Deploying open-source AI means you own the governance. The safety testing, content filtering, incident reporting, and documentation that a commercial vendor handles in the background now sit on your project plan.
- EU AI Act Article 25 reclassifies you as the provider — with all associated compliance obligations — if you substantially modify an open-source model, including fine-tuning or repurposing it for a new use case. Open-source doesn’t mean liability-free; it means the liability moved to you.
- Fine-tuning is a compliance event, not just a technical task. NIST AI 600-1 MANAGE 3.1 requires a full risk re-assessment after every fine-tuning or RAG implementation, because the model’s behavior and risk profile may have changed.
- If an open-source model lacks a model card, you’re responsible for producing the documentation yourself. NIST AI 600-1 MEASURE 2.9 requires provenance, limitations, and safety testing results to be documented regardless of where the model came from.
In January 2025, Chinese startup DeepSeek released an open-source model matching industry leaders at a fraction of the cost. Within days, it became the most downloaded app in the US. Then came the problems: security researchers found it failed to block 100% of jailbreak attempts, a database breach exposed over a million records, and governments started banning it.
DeepSeek isn’t an edge case — it’s an early preview of a world where powerful AI is freely available and governance responsibility lands entirely on the deployer.
Why Open-Source AI Changes Everything
With proprietary AI through an API, governance is shared: the provider handles model safety, you handle how you use it. Open-source changes that arrangement at exactly the point where it matters most.
EU AI Act Article 25 makes this explicit: any party that substantially modifies a high-risk AI system — including fine-tuning or repurposing it for a new use case — becomes the provider of that system and takes on all the associated compliance obligations. Open-source doesn’t mean liability-free. It means the liability moved to you.
What changes operationally is that the governance work your commercial vendor was doing in the background — safety testing, content filtering, incident reporting, documentation — now sits on your project plan.
Key Risks
Safety and Security Gaps
Open-source models often have weaker safety guardrails than commercial alternatives. A Cisco study found DeepSeek failed to block harmful prompts in 100% of jailbreak attempts, compared to ChatGPT blocking 86%. Commercial providers invest heavily in safety research; open-source developers may prioritize capability over safety.
NIST AI 600-1 MANAGE 3.1 instructs teams to test AI system value chain risks — including data poisoning, malware, software and hardware vulnerabilities, labour practices, data privacy and localisation compliance, and geopolitical alignment — when relying on third-party or open-source models. These aren’t hypothetical risks; they’re a documented checklist of things that have already gone wrong in open-source deployments.
Data Privacy and Sovereignty
Where does your data go when inference runs? It depends entirely on your deployment architecture.
| Deployment Model | Data Governance Implication |
|---|---|
| Self-hosted | Under your control — data stays within your infrastructure and jurisdiction. |
| Third-party hosted | Depends on the provider’s jurisdiction and data agreements — must be assessed and documented. |
| Vendor app (e.g. DeepSeek) | Subject to the vendor’s jurisdiction and applicable laws — DeepSeek stores data in China, subject to Chinese law, which can require data sharing with authorities. |
The EU AI Act’s Recital 91 is clear that deployers must ensure persons implementing human oversight have the necessary competence, including adequate AI literacy, training, and authority. Understanding where your data flows is a baseline literacy requirement, not an advanced topic.
Supply Chain Opacity
Commercial providers typically publish model cards documenting training data, limitations, and safety testing results. Open-source models vary widely in documentation quality — and this gap has direct governance consequences.
The EU AI Act’s Recital 89 encourages open-source developers to implement widely adopted documentation practices such as model cards and datasheets. The word “encourage” is telling: it is not a mandate for the developer. If the model you’re deploying lacks a model card, the information gap is yours to fill.
NIST AI 600-1 MEASURE 2.9 (MS-2.9-002) specifies exactly what model documentation must contain: training data provenance, data quality, model architecture, optimization objectives, fine-tuning approaches, evaluation data, ethical considerations, and legal and regulatory requirements. If the model cannot answer most of these, that is a risk your register needs to reflect.
Modification Risks
Open-source models can be fine-tuned, quantized, and modified — that’s the point. But modifications create compounding governance challenges. Fine-tuning can introduce biases or erode safety guardrails. Quantization can shift behavior in ways not obvious from benchmark scores.
NIST AI 600-1 MANAGE 3.1 (MG-3.1-003) requires teams to re-assess model risks after fine-tuning or retrieval-augmented generation implementation. This is triggered every time the model changes, not just at initial deployment.
Under EU AI Act Article 25, a deployer who makes a substantial modification to an open-source model becomes the new provider, inheriting full compliance obligations. Fine-tuning a model for employment screening, credit assessment, or healthcare triage is not just a technical decision — it is potentially a compliance event.
Governance Essentials
Risk Assessment Before Selection
Before selecting an open-source model, your risk assessment should address:
- What is the model’s documented training data provenance? Is it sufficient to assess bias, copyright risk, or data poisoning exposure?
- What safety testing was performed, and by whom? Is there a model card or equivalent documentation?
- What is the model’s country of origin, and what legal obligations does that create for data handling?
- If the model has been fine-tuned by a third party, what modifications were made and when?
NIST AI 600-1 GOVERN 6.1 (GV-6.1-009) provides a clear frame: due diligence for open-source tools should assess GAI vendors, open-source tools, and service providers against incident and vulnerability databases. Checking a model’s known incident history is easy to skip. Don’t.
Required Guardrails
Because open-source models typically arrive with weaker built-in safety controls, your project must implement compensating controls.
| Guardrail | Requirement Grounding |
|---|---|
| Input/output filtering | Add your own content filters. The model’s built-in safety may be inadequate for your deployment context. |
| Production monitoring | Log and review outputs. NIST AI RMF MEASURE 2.4 requires ongoing monitoring of AI system functionality and behavior in production. |
| Named human oversight | EU AI Act Article 26 requires deployers of high-risk AI systems to assign human oversight to natural persons who have the necessary competence, training, and authority. |
| Incident response | NIST AI 600-1 GOVERN 6.2 requires contingency processes to handle failures or incidents in third-party AI systems deemed high-risk. |
| Version control | Document exactly what model version you’re running and every modification applied. MANAGE 3.1 (MG-3.1-002) lists version control as part of value chain risk testing. |
Regulatory Reality
Open-source does not mean unregulated. This is the most persistent misconception in open-source AI adoption.
The EU AI Act’s treatment is carefully qualified. Free and open-license GPAI models receive reduced transparency obligations under Article 53 — but this exemption does not apply to models with systemic risk (those trained with more than 1025 FLOPs). And for deployers building high-risk applications, Chapter III requirements apply based on the use case, not the license.
Recital 89 clarifies that open-source developers are not mandated to comply with value chain responsibilities when they release components for free. But when you deploy that component, those responsibilities land with you. The Act didn’t eliminate the obligations — it clarified who holds them.
PM Responsibilities by Phase
During Planning
- Assess whether open-source is appropriate — capability advantages must be weighed against the governance work you are absorbing from the developer
- Evaluate model provenance, documentation quality, and safety track record as formal procurement criteria
- Determine deployment architecture early — self-hosted vs. third-party hosted carry materially different data governance implications
- Budget explicitly for safety testing and guardrail implementation
During Development
- Conduct safety testing using structured methods — NIST AI 600-1 recommends diverse red-teaming, not just internal testing
- Implement input/output filtering and monitoring before any production exposure
- Document the exact model version, every modification, and all testing results
- If fine-tuning, trigger a full risk re-assessment per NIST MANAGE 3.1 (MG-3.1-003)
Post-Deployment
- Monitor for safety issues and emerging vulnerabilities on a defined cadence
- Track community reports through the model’s repository and associated disclosure channels
- Re-assess risk after each update that materially changes model behavior
- Document incidents — NIST AI 600-1 GOVERN 6.2 (GV-6.2-002) requires documenting incidents involving third-party GAI data and systems, including open-data and open-source software
Questions for Your Project
Model Selection
- Why open-source over proprietary for this use case, and does that rationale hold against the governance work you are absorbing?
- What is this model’s safety and security track record? Has it appeared in AI incident databases?
- Is documentation sufficient — model card, training data provenance, known limitations, safety evaluation results?
Risk and Deployment
- What is the worst case if this model misbehaves in production?
- Does the model’s country of origin or hosting jurisdiction create regulatory, contractual, or reputational risk?
- If we fine-tune or modify this model, at what point does EU AI Act Article 25 reclassify us as the provider?
- What specific guardrails will we add, and who owns them?
Right-Sizing for Your Situation
Start with provenance and documentation before evaluating capabilities. Does the model have a model card that answers NIST AI 600-1’s MEASURE 2.9 checklist: training data provenance, data quality, model architecture, evaluation data, known limitations? If it doesn’t, that documentation gap is your first project deliverable. Treat your deployment architecture decision as a data governance decision — self-hosted keeps data in your jurisdiction, third-party hosted requires a separate assessment. Make that choice explicitly and document it.
Formalize the risk re-assessment cycle. NIST AI 600-1 MANAGE 3.1 (MG-3.1-003) requires re-assessment after every fine-tuning or RAG implementation — build this into your release process as a mandatory gate, not an optional step. Establish a structured third-party risk monitoring cadence: track model repositories for updates, monitor AI incident databases for newly disclosed vulnerabilities, and set a trigger threshold for when a reported issue warrants a pause in production.
At scale, the governance challenge is consistency across projects. NIST AI 600-1 GOVERN 6.1 (GV-6.1-009) provides the framework: integrate due diligence processes for open-source acquisition into procurement and vendor management — not handled project by project. Build a shared model registry that tracks version, modification history, and risk assessment status for every open-source model in production. The audit trail requirement doesn’t change because the model is open-source; it just means you own it.
The AI Governance Advisor at app.aipmo.co can help you assess governance gaps for specific open-source models, work through the EU AI Act Article 25 provider reclassification analysis, and structure your model documentation requirements.
Framework References
EU AI Act (Regulation (EU) 2024/1689) — Article 25 (deployers who substantially modify open-source models become providers), Article 26 (deployer obligations regardless of model origin), Recital 89 (open-source developer obligations), Recital 91 (human oversight competence requirements). Chapter III compliance requirements apply based on use case, not model license.
NIST AI 600-1 GenAI Profile (2024) — MANAGE 3.1 / MG-3.1-003 (re-assessment after fine-tuning or RAG implementation), GOVERN 6.1 / GV-6.1-009 (due diligence against incident and vulnerability databases), GOVERN 6.2 (contingency processes for open-source failures), MEASURE 2.9 / MS-2.9-002 (model documentation requirements when vendor documentation is absent).
NIST AI RMF 1.0 (NIST AI 100-1, 2023) — GOVERN 6.1 (supply chain transparency and third-party risk policies), MAP 4.2 (internal risk controls for third-party AI components), MEASURE 2.4 (ongoing monitoring of AI system behavior in production).
PMI CPMAI Guide (2025) — Phase III and V. Open-source model evaluation and validation as project deliverables; governance gap assessment when commercial documentation is unavailable.
This article is part of AIPMO’s Emerging Topics series. See also: Third-Party AI and Vendor Management | AI Testing and Validation (TEVV) | Monitoring AI Systems in Production | The PM’s Guide to NIST AI RMF