NEW - Open-Source AI: The Governance Challenges You Didn't See Coming

In January 2025, Chinese startup DeepSeek released an open-source model matching industry leaders at a fraction of the cost. Within days, it became the most downloaded app in the U.S. Then came the problems: security researchers found it failed to block 100% of jailbreak attempts, a database breach exposed over a million records, and governments started banning it.

DeepSeek isn’t an anomaly — it’s a preview of a world where powerful AI is freely available but governance responsibility falls entirely on you.

Why Open-Source AI Changes Everything

When you use proprietary AI through an API, governance responsibility is split. The provider handles model safety; you manage how you use it. Open-source breaks this model at the point where it matters most: accountability.

The EU AI Act makes this explicit in its value chain provisions. Under Article 25, any distributor, importer, deployer, or other third party that “makes a substantial modification to a high-risk AI system” — including fine-tuning or repurposing a model for a new use case — is reclassified as the provider of that system, assuming all associated compliance obligations. Open-source doesn’t mean liability-free. It means the liability moved to you.

What changes operationally is that the governance work your commercial vendor was doing in the background — safety testing, content filtering, incident reporting, documentation — now sits on your project plan.

Key Risks

Safety and Security Gaps

Open-source models often have weaker safety guardrails than commercial alternatives. A Cisco study found DeepSeek failed to block harmful prompts in 100% of jailbreak attempts, compared to ChatGPT blocking 86%. Commercial providers invest heavily in safety research; open-source developers may prioritize capability over safety.

NIST AI 600-1’s MANAGE 3.1 guidance instructs teams to “test GAI system value chain risks (e.g., data poisoning, malware, other software and hardware vulnerabilities; labor practices; data privacy and localization compliance; geopolitical alignment)” when relying on third-party or open-source models. These aren’t hypothetical risks — they’re a documented checklist of things that have already gone wrong in open-source deployments.

Data Privacy and Sovereignty

Where does your data go when inference runs? It depends entirely on your deployment architecture:

•      Self-hosted: Under your control

•      Third-party hosted: Depends on the provider’s jurisdiction and data agreements

•      Vendor app (like DeepSeek): Subject to the vendor’s jurisdiction and applicable laws

The DeepSeek app stores data in China, subject to Chinese law — which can require data sharing with authorities. The EU AI Act’s Recital 91 is clear that deployers must ensure persons implementing human oversight “have the necessary competence, in particular an adequate level of AI literacy, training and authority.” Understanding where your data flows is a baseline literacy requirement, not an advanced topic.

Supply Chain Opacity

Commercial providers typically publish model cards documenting training data, limitations, and safety testing results. Open-source models vary widely in documentation quality — and this gap has direct governance consequences.

The EU AI Act’s Recital 89 encourages open-source developers to “implement widely adopted documentation practices, such as model cards and data sheets.” The word “encourage” is telling: it’s not a mandate for the developer. If the model you’re deploying lacks a model card, the information gap is yours to fill.

NIST AI 600-1’s MEASURE 2.9 (MS-2.9-002) specifies exactly what model documentation must contain: training data provenance, data quality, model architecture, optimization objectives, fine-tuning approaches, evaluation data, ethical considerations, and legal and regulatory requirements. If the model can’t answer most of these, that’s a risk your register needs to reflect.

Modification Risks

Open-source models can be fine-tuned, quantized, and modified — that’s the point. But modifications create compounding governance challenges. Fine-tuning can introduce biases or erode safety guardrails. Quantization can shift behavior in ways not obvious from benchmark scores.

NIST AI 600-1’s MANAGE 3.1 (MG-3.1-003) requires teams to “re-assess model risks after fine-tuning or retrieval-augmented generation implementation.” This is triggered every time the model changes, not just at initial deployment.

Under the EU AI Act, Article 25 establishes that a deployer who makes a “substantial modification” to an open-source model becomes the new provider, inheriting full compliance obligations. Fine-tuning a model for employment screening, credit assessment, or healthcare triage isn’t just a technical decision — it’s potentially a compliance event.

Governance Essentials

Risk Assessment Before Selection

Before selecting an open-source model, your risk assessment should address:

•      What is the model’s documented training data provenance? Is it sufficient to assess bias, copyright risk, or data poisoning exposure?

•      What safety testing was performed, and by whom? Is there a model card or equivalent documentation?

•      What is the model’s country of origin, and what legal obligations does that create for data handling?

•      If the model has been fine-tuned by a third party, what modifications were made and when?

NIST AI 600-1’s GOVERN 6.1 (GV-6.1-009) provides a clear frame: due diligence for open-source tools should “assess GAI vendors, open-source or proprietary GAI tools, or GAI service providers against incident or vulnerability databases.” Checking a model’s known incident history is easy to skip. Don’t.

Required Guardrails

Because open-source models typically arrive with weaker built-in safety controls, your project must implement compensating controls:

•      Input/output filtering: Add your own content filters. The model’s built-in safety may be inadequate for your deployment context.

•      Monitoring: Log and review outputs. NIST AI RMF MEASURE 2.4 requires ongoing monitoring of AI system functionality and behavior when in production.

•      Human oversight: EU AI Act Article 26 requires deployers of high-risk AI systems to “assign human oversight to natural persons who have the necessary competence, training and authority.”

•      Incident response: NIST AI 600-1 GOVERN 6.2 requires contingency processes to handle failures or incidents in third-party AI systems deemed high-risk.

•      Version control: Document exactly what model version you’re running and every modification applied. MANAGE 3.1 (MG-3.1-002) lists version control as part of value chain risk testing.

Regulatory Reality

Open-source doesn’t mean unregulated. This is the most persistent misconception in open-source AI adoption.

The EU AI Act’s treatment is carefully qualified. Free and open-licence GPAI models receive reduced transparency obligations under Article 53 — but this exemption does not apply to models with systemic risk (those trained with more than 10²⁵ FLOPs). And for deployers building high-risk applications, Chapter III requirements apply based on the use case, not the licence.

Recital 89 clarifies that open-source developers are not mandated to comply with value chain responsibilities when they release components for free. But when you deploy that component, those responsibilities land with you. The Act didn’t eliminate the obligations — it clarified who holds them.

PM Responsibilities

During Planning

•      Assess whether open-source is appropriate — capability advantages must be weighed against the governance work you’re absorbing from the developer

•      Evaluate model provenance, documentation quality, and safety track record as formal procurement criteria

•      Determine deployment architecture early — self-hosted vs. third-party hosted carry materially different data governance implications

•      Budget explicitly for safety testing and guardrail implementation

During Development

•      Conduct safety testing using structured methods — NIST AI 600-1 recommends diverse red-teaming, not just internal testing

•      Implement input/output filtering and monitoring before any production exposure

•      Document the exact model version, every modification, and all testing results

•      If fine-tuning, trigger a full risk re-assessment per NIST MANAGE 3.1 (MG-3.1-003)

Post-Deployment

•      Monitor for safety issues and emerging vulnerabilities on a defined cadence

•      Track community reports through the model’s repository and associated disclosure channels

•      Re-assess risk after each update that materially changes model behavior

•      Document incidents — NIST AI 600-1 GOVERN 6.2 (GV-6.2-002) requires documenting “incidents involving third-party GAI data and systems, including open-data and open-source software”

Questions for Your Project

Model Selection

•      Why open-source over proprietary for this use case, and does that rationale hold against the governance work you’re absorbing?

•      What is this model’s safety and security track record? Has it appeared in AI incident databases?

•      Is documentation sufficient — model card, training data provenance, known limitations, safety evaluation results?

Risk and Deployment

•      What’s the worst case if this model misbehaves in production?

•      Does the model’s country of origin or hosting jurisdiction create regulatory, contractual, or reputational risk?

•      If we fine-tune or modify this model, at what point does EU AI Act Article 25 reclassify us as the provider?

•      What specific guardrails will we add, and who owns them?

Right-Sizing for Your Situation

Greenfield — New to Open-Source AI

Start with provenance and documentation before evaluating capabilities. Determine whether the model has a model card that answers NIST AI 600-1’s MEASURE 2.9 checklist. If it doesn’t, that documentation gap is your first project deliverable. Treat your deployment architecture decision as a data governance decision, not just an infrastructure one. The AIPMO AI Governance Advisor at app.aipmo.co can help you build a model selection checklist grounded in NIST MAP and EU AI Act supply chain requirements.

Emerging — Open-Source in Production

Formalize the risk re-assessment cycle. NIST AI 600-1 MANAGE 3.1 (MG-3.1-003) requires re-assessment after every fine-tuning or RAG implementation — build this into your release process as a mandatory gate. Establish a structured third-party risk monitoring cadence across model repositories and AI incident databases. The AIPMO AI Governance Advisor can help you design a post-deployment monitoring plan mapped to NIST MEASURE 2.4 and MANAGE 3.1.

Established — Governing Open-Source Across Multiple Systems

At scale, the governance challenge is consistency. NIST AI 600-1’s GOVERN 6.1 (GV-6.1-009) provides the framework: integrate due diligence processes for open-source acquisition into procurement and vendor management — not handled project by project. The AIPMO AI Governance Advisor supports multi-organization context for Consultant-tier users and can help you develop portfolio-level open-source governance standards.

Framework References

•      EU AI Act (Official Journal 2024) — Art. 25 (value chain responsibility); Art. 26 (deployer obligations); Art. 53 (GPAI open-source exemption); Recitals 89, 91, 102–104.

•      NIST AI 600-1 — AI RMF: Generative AI Profile, 2024. GOVERN 6.1 (supply chain risk); GOVERN 6.2 (contingency for third-party failures); MANAGE 3.1 (fine-tuning re-assessment); MEASURE 2.9 (model documentation).

•      NIST AI 100-1 — AI RMF 1.0, 2023. MEASURE 2.4 (production monitoring).

AIPMO sits at the intersection of project management and AI governance. All content is grounded in published frameworks. For deployment-context guidance, visit app.aipmo.co.