Software Testing Strategy for Agentic Change
Define a complete testing system for changes produced or maintained by agents.
A rapid review of the chapter’s existing Quick Read, principles, definitions, lessons, and review material.
The chapter in one pass.
- Purpose: Define a complete testing system for changes produced or maintained by agents.
- Best for: Quality engineers, senior developers, architects, and agent engineers.
- Prerequisites: Quality and Evidence Architecture.
- Reading time: 16 minutes.
- You will learn: How to select complementary test methods, control flaky tests and test data, measure fault sensitivity, and preserve independent evidence.
- Keep three ideas: passing existing tests is weak evidence; test selection follows risk and change impact; and agent-authored tests must be challenged independently.
Reconstruct and defend this chapter’s architecture.
Reconstruct the architecture, name each boundary, and defend the tradeoffs.
Open the source exercise
Design the test portfolio for a payment API schema change with a browser client, batch consumer, performance SLO, and accessibility requirement. Add one flaky test and one incomplete dependency map.
3. Enduring Principle
Build a risk-based test portfolio
| Method | Primary claim | Common blind spot |
|---|---|---|
| Unit | Local logic behaves under controlled inputs | Integration and configuration |
| Integration | Components and dependencies cooperate | Full user journey |
| End-to-end | Critical workflow functions | Fault localization and speed |
| Contract | Producer and consumer expectations remain compatible | Internal behavior |
| Property-based | Invariants hold across generated inputs | Incorrect properties |
| Mutation | Tests detect introduced faults | Equivalent or costly mutations |
| Fuzz | Parsers and boundaries withstand unexpected input | Business correctness |
| Performance/load | Latency, throughput, and resource limits | Functional intent |
| Accessibility | Interaction remains usable across access needs | Product value |
| Visual regression | Rendered appearance does not drift unexpectedly | Semantic correctness |
| Security testing | Known abuse classes and policies hold | Unknown threats |
The Quality Contract chooses methods from change risk, affected behavior, architecture boundaries, data, reversibility, and production impact. No single pyramid or percentage is universally sufficient.
Perform test-impact analysis
Map changed code, APIs, schemas, configuration, dependencies, and behavior to relevant tests. Selection records why a test was included or omitted. High-risk changes run broader suites; low-risk changes may use targeted suites plus periodic full validation. A model may recommend impact, but deterministic dependency and ownership data should constrain it.
Govern agent-generated tests
Require tests to fail against the relevant pre-change behavior or a deliberate fault when feasible. Review assertions for observable outcomes rather than implementation details. Separate the agent that proposes behavior from the validator that evaluates coverage and fault sensitivity.
Treat test infrastructure as production infrastructure
Version fixtures and test data, isolate tenants, remove secrets, define cleanup, track flaky behavior, and retain environment identity. Quarantining a flaky test is visible debt with owner and expiry, not a silent pass.
Bind results to exact subjects
A test receipt identifies source commit, artifact, environment, command, selected tests, exclusions, retries, duration, raw result, and verifier. Rerunning until green without preserving failures destroys evidence.
8. Notes and lessons learned
Test quantity is an activity metric. The important question is whether independent methods can detect the plausible faults introduced by this exact change.
9. Interview and discussion questions
- How do you validate an agent-generated test?
- When is targeted test selection safe?
- What should happen to a flaky required gate?
- How do unit, contract, and end-to-end evidence complement each other?
- Which test failures should affect agent trust?
Review this chapter.
Challenge a claim, boundary, missing failure mode, unclear term, or unsupported evidence statement.
- Claim
- Boundary
- Failure
- Evidence