PM Takeaways

•       NIST AI RMF MEASURE 2.1 requires that test sets, metrics, and tools used during TEVV are documented before testing begins — not after. Defining acceptance thresholds post-hoc, after results are known, undermines the entire validation process and creates governance exposure. The PM must ensure thresholds are documented and signed off as part of planning, not rationalised at the end of a test cycle.

•       Traditional software testing answers one question: does the system produce the correct output? AI TEVV must answer five: does it perform accurately, does it perform equitably across groups, does it degrade gracefully, can it be manipulated, and can its decisions be explained? A system that passes every traditional test can still discriminate against protected groups, fail unpredictably on edge cases, or produce outputs that cannot be explained to the people affected by them.

•       NIST AI RMF requires that AI actors carrying out verification and validation tasks are distinct from those who performed test and evaluation actions — and ideally distinct from those who built the system. Independence is not a procedural nicety; it mitigates the blind spots and organisational pressure that lead builders to unconsciously design tests their systems will pass. For high-risk systems, external auditors should be engaged for at least the final validation gate.

•       AI red-teaming, as defined in NIST AI 600-1, is an evolving practice for identifying adverse behaviour or outcomes through structured adversarial exercises. The quality of red-teaming outputs depends heavily on the diversity and domain expertise of the red team — demographically and interdisciplinarily diverse teams find different failure modes in different deployment contexts. Red team composition is a design decision, not a staffing convenience.

•       TEVV is not a phase — it is a continuous process across the entire AI lifecycle. NIST AI RMF maps distinct TEVV tasks to design and planning (validating assumptions and data), development (model validation and assessment), deployment (system integration, user experience, compliance), and operations (ongoing monitoring, incident tracking, SME recalibration). A test plan that treats TEVV as a pre-deployment gate misses the majority of what the framework requires.

What Traditional Testing Covers

What AI Systems Also Require

Functional requirements met — correct outputs for defined inputs

Performance across subpopulations — does accuracy hold for all groups, or only in aggregate?

Integration correctness — components working together as designed

Equitable outcomes — are error rates and outcome distributions fair across demographic groups?

Performance under load — response time and throughput at scale

Robustness to unexpected inputs — edge cases, out-of-distribution data, degraded inputs

Security vulnerabilities — known attack vectors and penetration testing

Adversarial resilience — deliberate manipulation by crafted inputs designed to cause failures

Regression — new code doesn’t break existing functionality

Stability over time — does performance hold as the world changes and data drifts?

User acceptance — the system meets stated requirements

Alignment with legal and ethical requirements — bias, explainability, privacy, human oversight

Activity

Core Question

Focus

Test

Does it work?

Executing the system against defined test cases to identify problems and measure performance

Evaluation

How well does it work?

Assessing performance against benchmarks, thresholds, and trustworthiness criteria, including socio-technical factors beyond the pipeline

Verification

Did we build it right?

Confirming the system meets its specifications — the technical requirements defined before development

Validation

Did we build the right thing?

Confirming the system meets real-world needs in its actual deployment context — not just the requirements as written

Lifecycle Phase

TEVV Focus

Design and planning

Validate assumptions about data availability, quality, and representativeness. Verify that requirements capture trustworthiness characteristics — fairness, explainability, safety — not just functional behaviour. Plan test approaches for each dimension before any model development begins.

Data preparation

Test data quality, completeness, and lineage. Evaluate representativeness across populations relevant to the use case. Identify proxy features that may encode protected characteristics. Document assumptions and limitations of the dataset for downstream TEVV actors.

Development (model building)

Validate model performance on held-out data not used in training. Evaluate fairness metrics disaggregated by relevant demographic groups. Conduct adversarial testing of model behaviour. Verify that explanations accurately reflect model behaviour, not post-hoc rationalisations.

Deployment

System integration testing in the production environment. User acceptance testing with realistic scenarios involving actual users and affected parties. Independent review of test results before final go/no-go. Recalibration for integration, user experience, and compliance with legal, regulatory, and ethical specifications.

Operations (ongoing)

Continuous monitoring of performance metrics against baselines. Detection of drift and degradation. Incident tracking, analysis, and response. Periodic re-evaluation against original benchmarks. SME recalibration as the deployment context evolves.

Metric

What It Reveals

Accuracy, precision, recall, F1, AUC

Overall correctness against a labelled test set — the baseline performance measure

False positive and false negative rates

The cost of different error types, which varies significantly by use case — false negatives in medical screening carry different consequences than false positives in content moderation

Confidence calibration

Whether the model’s stated confidence reflects its actual accuracy — a model that says 90% confident should be correct 90% of the time. Miscalibrated confidence is a major source of over-reliance.

Performance on held-out data

Generalisation beyond the training distribution — Stanford HAI’s validity framework warns that benchmark performance does not always equal real-world performance, particularly when systems may have encountered test data during training (benchmark contamination)

Metric

What It Reveals

Disaggregated performance by group

Whether accuracy, error rates, and confidence calibration hold across demographic groups, geographies, or other segments relevant to the use case

Demographic parity

Whether the AI produces positive outcomes at similar rates across groups, regardless of individual characteristics

Equalized odds / equal opportunity

Whether true positive and false positive rates are consistent across groups — a more demanding standard than demographic parity in many use cases

Denigration and stereotyping prevalence

For generative AI, whether outputs exhibit systematic bias, harmful stereotyping, or denigrating content toward particular groups — NIST AI 600-1 MS-2.11-001 recommends specific benchmarks including Bias Benchmark Questions and Winogender Schemas

Test Type

What It Surfaces

Edge case testing

Unusual but legitimate inputs — values at the extreme ends of distributions, rare but valid combinations of features, inputs that represent underrepresented populations in the training data

Out-of-distribution testing

Inputs that differ from the training distribution in ways the model has not seen — how does the model behave, and does it signal appropriate uncertainty rather than producing confident but wrong outputs?

Noisy and degraded inputs

Incomplete data, missing features, corrupted inputs, or lower-quality data than the training set — common in real-world deployments where data quality is uncontrolled

Distribution shift simulation

Testing performance on data from a different time period, geography, or population than the training data, to assess how quickly performance degrades as deployment conditions diverge from training conditions

Test Type

What It Surfaces

Adversarial input testing

Inputs deliberately crafted to cause the model to fail — images with imperceptible perturbations that cause misclassification, text prompts designed to elicit harmful outputs, inputs that exploit model architecture weaknesses

Prompt injection testing

For systems accepting natural language, whether malicious instructions embedded in user input can override system instructions, bypass guardrails, or extract protected information

Guardrail and safety mechanism testing

Whether safety controls actually prevent the failure modes they are designed to prevent — including whether they can be bypassed through creative reformulation of harmful requests

Red-teaming

Structured adversarial exercises conducted by people independent of the build team. NIST AI 600-1 defines AI red-teaming as exercises to identify potential adverse behaviour, stress test safeguards, and find failure modes that standard testing misses. Team diversity — demographic, disciplinary, and domain expertise — directly affects the quality of findings.

Test Focus

Questions to Answer

Explanation accuracy

Do explanations accurately reflect the model’s actual decision process, or are they post-hoc rationalisations that look plausible but do not correspond to what the model computed?

Explanation comprehensibility

Are explanations comprehensible to the specific audiences who need them — a technical operator, a non-technical user, an affected individual, a regulator? The same explanation may be appropriate for one audience and meaningless to another.

Explanation consistency

Does the system produce similar explanations for similar decisions? Inconsistent explanations for similar cases undermine both trust and the utility of explanations for error detection.

Regulatory adequacy

Do explanations meet the specific explainability requirements that apply in the deployment jurisdiction and sector — including the EU AI Act’s transparency requirements for high-risk AI and sector-specific obligations?

Approach

How It Works

Strengths and Limitations

Benchmark testing

Compare system performance against standard datasets or established baselines used across the field

Reproducible and comparable across systems. Risk of benchmark contamination — the system may have encountered test data during training. Benchmark performance does not always equal real-world performance in the deployment context (Stanford HAI).

Field testing

Evaluate system behaviour in realistic deployment conditions with actual users or representative populations

Reveals issues that controlled testing misses — real-world data quality, user behaviour, and deployment context. More resource-intensive. May expose real users to risk if not carefully staged. Requires informed consent and human subjects protections.

AI red-teaming

Structured adversarial exercises by people independent of the build team, aiming to find failure modes and vulnerabilities that standard testing does not surface

Identifies failure modes that designed tests cannot find. Quality depends heavily on team diversity, domain expertise, and depth of effort. Resource-intensive. Findings require interpretation before incorporation into governance decisions.

Structured public feedback

Gather input from users, affected communities, and domain experts through focus groups, surveys, community advisory boards, and feedback channels

Surfaces failure modes and contextual impacts that technical testing cannot detect. Builds stakeholder trust. Requires careful design to be actionable and representative. Not a substitute for technical evaluation.

Failure Mode

What It Means

Testing Response

Brittleness

System fails on inputs slightly different from training data — small perturbations, slightly different formatting, or edge-of-distribution values cause disproportionate performance drops

Edge case testing, out-of-distribution testing, input perturbation analysis

Embedded bias

System learns and perpetuates biases present in training data, producing systematically different outcomes for different demographic groups even when those differences are not justified by the task

Disaggregated fairness testing, bias audits using standardised benchmarks, intersectional analysis, engagement with affected communities

Catastrophic forgetting

After retraining on new data, the system loses capabilities it had on the original training distribution — a silent regression that aggregate metrics may not detect

Regression testing against the full original test suite after every retraining cycle, not just testing on new data

Uncertainty blindness

System produces high-confidence outputs even when inputs are ambiguous, out of distribution, or when the model should not be confident — miscalibration that drives over-reliance on AI recommendations

Confidence calibration testing; evaluation of model behaviour on inputs the model should recognise as uncertain

Adversarial vulnerability

System can be deliberately manipulated by carefully crafted inputs designed to exploit weaknesses in the model architecture or training data

Adversarial input testing, prompt injection testing for language models, red-teaming with adversarial intent

Distribution shift

Real-world production data differs from training data in ways that erode performance gradually and without obvious failure signals

Field testing, pre-deployment distribution comparison, post-deployment drift monitoring with defined action thresholds

Independence Level

What It Means in Practice

Testers separate from developers

The minimum baseline — the team conducting TEVV should not have been involved in building the system components they are evaluating. This applies even within the same organisation.

Independent internal review

A function within the organisation — risk, compliance, legal, or a dedicated AI governance team — conducts or reviews TEVV results independent of the project team. Applicable to medium- and higher-risk systems.

External audit or assessment

Third parties with no commercial interest in the system’s deployment conduct or independently verify TEVV results. Required for high-risk AI systems under the EU AI Act in some sectors, and recommended by NIST for systems with significant potential for harm.

Structured challenge process

NIST AI RMF describes “effective challenge” — a culture that encourages critical thinking and questioning of important design and implementation decisions by experts with the authority and stature to act on what they find. This applies throughout the lifecycle, not only at formal test gates.

Strategy

How It Functions as Testing

Phased rollout

Deploy to a limited user population first and expand gradually based on observed performance. Allows real-world performance validation at limited scale before full exposure. Appropriate for most AI deployments.

Shadow mode

The AI system runs in parallel with existing processes but does not affect outcomes — its outputs are logged and evaluated, but decisions are made by the current process. Allows observation of real-world AI behaviour without risk to current outcomes. Particularly useful before autonomous operation begins.

A/B testing

Run the new AI system alongside the existing process or a baseline model, routing different users to each. Allows direct performance comparison under identical conditions. Requires careful design to avoid differential harm — if the AI system may produce worse outcomes for some users, random assignment raises ethical issues.

Canary deployment

Route a small, controlled percentage of live traffic to the new system while maintaining the existing system for the remainder. Allows detection of production issues before full rollout. Appropriate when gradual validation of production behaviour is needed before committing to full deployment.

Greenfield — AI Testing Playbook

For PMs without formal AI testing processes. Essential tests for each trustworthiness dimension, with simplified approaches for resource-constrained teams — designed to establish a documented TEVV baseline without enterprise infrastructure.

Emerging — AI Testing Playbook

For PMs building repeatable processes. Comprehensive test planning templates, metric definitions, threshold-setting guidance, red-teaming design, and documentation standards for teams building a structured TEVV capability.

Established — AI Testing Playbook

For PMs in organisations with formal governance. How to integrate AI TEVV into existing quality management systems, compliance frameworks, and regulatory requirements — including EU AI Act conformity assessment and sector-specific audit obligations.