- Michigan MiDAS was built by a contractor. Detroit’s wrongful FRT arrest used a commercial product. In both cases the deploying agency owned the liability. Procuring AI from a vendor doesn’t transfer accountability — it changes who you need contractual access to in order to meet it.
- Vendor validation is not institutional validation. Canada’s Directive requires agencies to validate AI in their own environment, on their own population. Performance claims in a proposal must be testable and tested before contract award — not accepted on vendor warranty.
- If a vendor won’t provide audit rights over training data, model documentation, and validation results, that is a disqualifying factor — not a negotiating position. “Trade secret” does not override a government agency’s legal obligation to govern systems that affect citizens’ rights.
- Vendors update AI models after contract award. A facial recognition system updated to a new architecture may have different bias characteristics than the version tested at procurement. Contracts must require advance notification and agency re-testing rights before material changes go live.
- OMB M-25-22 requires contracts to protect against vendor lock-in through data portability, model portability, and knowledge transfer. If a vendor fails or becomes non-compliant, the agency needs an exit path that doesn’t leave government services inoperable.
The decision to buy AI from a vendor is a governance decision. It determines what you can audit, what you can challenge, what you can change, and who bears legal liability when something goes wrong. Most government AI procurement treats these as contract administration questions — to be worked out during vendor management after the contract is signed. They are not. They are procurement design questions that must be addressed before the solicitation is issued, because once a contract is signed with a vendor who has excluded audit rights, restricted bias testing, and protected model documentation as proprietary, those terms are extraordinarily difficult to renegotiate.
The Core Procurement Problem
Most commercial AI is developed for private sector clients who bear limited accountability to the people their AI affects. A vendor’s terms and conditions, designed for enterprise software sales, do not contemplate the due process obligations, freedom of information exposure, judicial review rights, and constitutional constraints that apply when AI is deployed in government. The standard enterprise AI contract is not fit for government use.
| Commercial AI Contract Default | Why It’s Inadequate for Government |
|---|---|
| Vendor controls training data documentation | Government needs to verify data doesn’t contain proxies for protected characteristics in its specific population. |
| Validation conducted by vendor on vendor data | Government’s due process obligation requires validation in government’s environment on government’s population. |
| Model documentation as trade secret | Government transparency requirements (ATRS, AIA, OMB use-case inventories) require public disclosure of how AI systems work. |
| Unilateral right to update model | Model updates can change bias characteristics; government needs notification and testing window before updates go live. |
| No audit rights beyond performance SLA | Judicial review, FoI requests, and legislative scrutiny require deeper access than commercial SLA metrics provide. |
| Liability capped at contract value | Government liability to harmed citizens may far exceed contract value; liability allocation must reflect this asymmetry. |
| Data used to train vendor’s models | Government citizen data may not be used to train models that will be sold to other clients; data sovereignty provisions required. |
Five Jurisdictions: What Good Looks Like
| Jurisdiction | Key Procurement Requirements |
|---|---|
| Canada | AI Source List of pre-approved suppliers. AIA obligation cannot be outsourced — agencies must complete their own AIA for vendor-supplied systems. Any vendor AI used for administrative decisions about clients falls within scope. Validation in government’s own environment required. |
| United States (OMB M-25-22) | Disclosure of AI use in vendor proposals, transparency provisions, privacy protections preventing vendor use of government data to train other clients’ models, data and model portability, agency testing rights before contract award, and closeout provisions. Rights- and safety-impacting AI requires additional terms on bias testing and validation documentation. |
| United Kingdom | Central Digital and Data Office guidelines: responsibility record for accountability across AI lifecycle, governance approach before procurement, regular model testing for bias, service level definition, knowledge transfer so agencies can operate or audit without vendor dependency. Procurement Act 2023 and ATRS obligation flow through to vendor documentation requirements. |
| European Union | EU AI Act compliance flow-through. Providers of high-risk AI have specific obligations: technical documentation, conformity assessment, CE marking, post-market monitoring. Contracts must allocate obligations clearly. Commission draft guidance on serious incident reporting (November 2025 consultation) clarifies deployer and provider responsibilities. |
| Australia | Post-Robodebt: agencies cannot outsource accountability for decisions that affect citizens’ rights. Governance obligations must flow through contracts. Vendor change management must be subject to government review for systems affecting rights and benefits. DTA Policy (September 2024) and Commonwealth Ombudsman Guide (March 2025) establish that accountability persists regardless of whether AI was built in-house or by a contractor. |
The Contract Checklist
Documentation Requirements
| Contract Requirement | Governance Purpose |
|---|---|
| Training data description: sources, date range, demographic coverage, known gaps | Enables assessment of whether training data reflects the government’s population; identifies potential bias sources. |
| Model development documentation: architecture, feature selection rationale, known limitations, intended use boundaries | Enables independent validation; required for ATRS entry (UK) and AIA publication (Canada). |
| Validation results: performance by demographic subgroup, false positive/negative rates by group | Required for bias assessment; enables government to verify vendor claims before deployment. |
| Acceptable use policy: what the system is designed for and what it should not be used for | Establishes deployment boundaries; required by US OMB M-25-22. |
| Change log and version history | Enables government to track model changes and assess whether re-validation is needed. |
| Incident log: prior known failures, errors, or concerning outputs from other deployments | Due diligence requirement; prior incidents in other government deployments are material to procurement decisions. |
Testing and Validation Rights
- Pre-award testing: agency has the right to test proposed AI solutions in a representative environment before contract award. Performance claims must be tested, not accepted on vendor warranty.
- Environmental validation: agency has the right to conduct independent validation in its own technical environment on its own population data.
- Demographic subgroup testing: for any system affecting decisions about individuals, agency has the right to test performance across relevant demographic subgroups in its population.
- Re-testing on material change: any material change to the model triggers agency re-testing rights before the updated model is deployed in production.
Audit Rights
- Audit access: agency has the right to audit vendor’s model documentation, training data descriptions, validation methodology, and performance records on reasonable notice.
- Judicial and regulatory cooperation: vendor must cooperate with government in responding to judicial review, freedom of information requests, parliamentary inquiries, and regulatory examinations.
- Third-party audit: for high-risk government AI, agency has the right to commission independent third-party audit with vendor cooperation.
Change Management
- Material change notification: vendor must provide at least 30 days advance notice of any material change to the model before deployment in government production.
- Change review and approval: agency has the right to review proposed changes, conduct re-testing, and require mitigation before changes are deployed.
- Change rejection right: agency has the right to reject a proposed model update that does not meet agreed governance standards and continue operating the previously approved version for a defined transition period.
Data Governance
- Data sovereignty: government citizen data processed by the vendor may not be used to train or improve models sold to other clients. This must be technically enforceable, not merely stated as a policy.
- Data retention and deletion: defines what data the vendor retains, for how long, and the deletion process at contract end. Aligns with government data protection obligations.
- Sub-processor disclosure: vendor must disclose any sub-processors who will process government data, and ensure these sub-processors meet equivalent standards.
- Data residency: specifies where data will be processed and stored. Government data sovereignty requirements restrict where citizen data can be held.
Exit and Transition
- Data portability: at contract end, government receives its data in a usable format, regardless of the commercial relationship.
- Knowledge transfer: vendor must provide sufficient documentation and training for government to transition without operational disruption.
- Transition period: minimum transition period during which vendor must continue to operate the system and support transition to a new solution.
PM Responsibilities in Government AI Procurement
| Phase | Key Actions |
|---|---|
| Pre-Solicitation | Complete the impact assessment — risk tier determines procurement requirements. Map transparency obligations: what must be publicly disclosed about this system? Identify applicable legal constraints: data residency, data protection law, constitutional requirements. Draft technical requirements for validation, audit, and change management before the market engagement. |
| Market Engagement & Evaluation | Issue an RFI that explicitly asks vendors about validation capabilities, audit cooperation, documentation practices, and prior government incident history. Weight governance criteria in evaluation alongside technical performance claims. Exercise pre-award testing rights for high-impact systems. |
| Contract Execution & Management | Track change notifications as a contract management KPI. Conduct periodic governance reviews separate from technical performance reviews. Test model updates before they go live in production for high-impact systems. Plan for exit from the first day of contract. |
Right-Sizing for Your Situation
For organizations procuring government AI for the first time. Covers impact-tiered procurement requirements, minimum documentation standards by impact level, basic audit right structures, and data governance essentials.
For organizations building repeatable procurement programs. Complete contract checklist by impact tier, vendor assessment methodology, pre-award testing framework, multi-jurisdiction alignment (Canada AIA, UK ATRS, OMB M-25-22, EU AI Act compliance flow-through), and change management contract design.
For mature procurement organizations. Enterprise-wide AI contract standards, vendor governance scorecard design, AI audit program design, exit management program, and procurement readiness for EU AI Act deployer obligations.
Framework References
Canada Directive on Automated Decision-Making (Treasury Board, 2019, amended 2023) — AIA obligation cannot be outsourced; validation in government’s environment required; Canada AI Source List provides pre-vetted suppliers.
US OMB Memorandum M-25-22 (April 3, 2025) — Federal AI procurement requirements: disclosure, transparency provisions, privacy protections for government data, portability, testing rights, and closeout provisions.
UK Guidelines for Public Procurement of AI (Central Digital and Data Office) — Responsibility record, governance approach, bias testing, service level definition, knowledge transfer requirements.
EU AI Act (Reg. (EU) 2024/1689) — Articles 9–15 (high-risk AI obligations flowing through supply chain); Article 25 (responsibilities of deployers); Annex IV (technical documentation requirements).
Australia DTA Policy for Responsible Use of AI in Government (September 2024) — Vendor accountability for government-procured AI; transparency statement requirements; Robodebt accountability chain findings as procurement design baseline.
NIST AI RMF 1.0 — GOVERN 6.1 (policies for third-party AI governance), MANAGE 3.1 (third-party AI risk management), MANAGE 3.2 (vendor monitoring).
This article is part of AIPMO’s Government series. See also: AI Governance in Government | Due Process and Automated Government Decisions | Law Enforcement and Criminal Justice AI