Spec Driven Development Part 5: Testing in Spec-Driven Development

Throughout Parts 1 to 4, acceptance criteria have been repeatedly mentioned as something that “must be objectively verifiable”. This article discusses exactly how that verification is done: how acceptance criteria written in EARS notation are derived into automated tests, which testing layers are relevant for each type of criterion, and the specific problem that appears when code is generated by AI agents — where tests can pass while the result still deviates from the original intent. This is the closing article of Part 1 of the Spec-Driven Development series, before the discussion continues to the Agentic Development series.

Acceptance Criteria as the Test Source, Not the Reverse

In classic TDD, the work order is: write a failing test, write minimal code to make it pass, refactor. Here the test is the artifact written first, and the spec — if it exists — is usually just an informal understanding in the developer’s head.

SDD reverses this order. The spec is written first as a complete document (intent, constraints, acceptance criteria, non-goals as discussed in Part 2), and tests are derived from the acceptance criteria already in that spec. The difference isn’t just administrative order — it changes what the tests actually verify.

flowchart LR
    subgraph TDD["Classic TDD"]
        A1[Write Test] --> A2[Write Minimal Code]
        A2 --> A3[Refactor]
    end
    subgraph SDD["Spec-Driven Development"]
        B1[Write Spec: Intent + Acceptance Criteria] --> B2[Derive Tests from Acceptance Criteria]
        B2 --> B3[Agent Implementation]
        B3 --> B4[Tests Verify Against the Spec]
    end

In classic TDD, tests reflect the developer’s assumptions about correct behavior — assumptions that might be incomplete because no spec document was thoroughly thought through beforehand. In SDD, tests reflect a contract already reviewed and agreed before implementation begins. This matters especially when an agent writes the code: the agent has no implicit context about “what was actually meant” beyond what’s written in the spec, so tests directly derived from the spec guarantee that verification measures the same thing the spec author meant — not what anyone writing tests later happens to assume is correct.

SDD doesn’t eliminate TDD — the two can run together. Once acceptance criteria are clear from the spec, the red-green-refactor cycle still applies when writing tests and code to meet those criteria. What changes is where the tests come from.

From EARS to Test Cases

The EARS-notation form of acceptance criteria discussed in Part 2 is deliberately designed to be almost directly mappable to test structures. Every EARS pattern has a direct counterpart in a given-when-then structure or a table-driven test.

Taking the password reset acceptance criteria from Part 2:

Acceptance Criteria (EARS):
- When a user submits a registered email to the password reset endpoint,
  the system shall send an email containing a reset link within a maximum of 5 seconds
- When a user submits an unregistered email, the system shall still
  respond with the same success message
- If a reset token has expired, the system shall reject the request with
  HTTP 410 and the message "link is no longer valid"
- If a reset token has already been used, the system shall reject the second
  request and immediately delete the token from the database

Translated into test cases with a given-when-then structure, each EARS line becomes one scenario:

Scenario: Registered email submits a reset request
  Given: a user with email "[email protected]" is registered in the system
  When: POST /password-reset/request with that email
  Then: response 200 with accepted=true
  And: the email is sent in less than 5 seconds

Scenario: Unregistered email submits a reset request
  Given: email "[email protected]" is not registered
  When: POST /password-reset/request with that email
  Then: response 200 with accepted=true (identical message to the success scenario)

Scenario: Token already expired
  Given: a reset token created 16 minutes ago (past the 15-minute limit)
  When: POST /password-reset/confirm with that token
  Then: response 410 with the message "link is no longer valid"

Scenario: Token already used
  Given: a reset token already successfully used before
  When: POST /password-reset/confirm with the same token
  Then: response 410
  And: the token remains deleted from the database (idempotent)

The conversion pattern is consistent: the “When [trigger]” and “If [condition]” clauses in EARS become the Given/When part, while “the system shall [behavior]” becomes the Then part. Because this pattern is standardized since spec writing, translating to tests isn’t interpretive work — anyone (or any agent) reading the same acceptance criteria will produce substantially identical test cases.

func TestRequestReset_EmailNotRegistered_ReturnsGenericSuccess(t *testing.T) {
    // Given: email not registered
    email := "[email protected]"

    // When: request password reset
    resp := requestPasswordReset(email)

    // Then: response exactly the same as the registered email scenario
    assert.Equal(t, 200, resp.StatusCode)
    assert.Equal(t, true, resp.Body.Accepted)
}

func TestConfirmReset_TokenExpired_Returns410(t *testing.T) {
    // Given: token created 16 minutes ago
    token := createExpiredToken(16 * time.Minute)

    // When: confirm reset with that token
    resp := confirmPasswordReset(token, "NewPassword123")

    // Then: rejected with 410
    assert.Equal(t, 410, resp.StatusCode)
    assert.Contains(t, resp.Body.Message, "no longer valid")
}

If you find acceptance criteria hard to translate directly into a Given-When-Then structure, that’s a signal the criterion is probably still not specific enough — not a problem with the test format. Go back to Part 2 to clarify the criterion before continuing to write tests.

Testing Layers Relevant to SDD

Not all acceptance criteria are verified at the same testing layer. A good spec covers various types of criteria — from single-function behavior to inter-service contracts — and each has its most suitable testing layer.

Contract tests are the layer most often missed in SDD practice, yet the most relevant precisely because Part 3 discussed API contracts as a spec element of their own. Contract tests verify that the API response truly matches the defined schema — not just “the code doesn’t error”, but “the data shape exactly matches the contract all consumers agreed on”.

flowchart TD
    A[Spec: Acceptance Criteria] --> B{Type of criterion?}
    B -- Single function logic --> C[Unit Test]
    B -- Component interaction --> D[Integration Test]
    B -- Data shape per contract --> E[Contract Test]
    B -- Complete user flow --> F[End-to-End Test]
    C --> G[Automatic Verification]
    D --> G
    E --> G
    F --> G

A common practice is mapping every acceptance criterion to one or more of these layers as part of the plan phase (discussed in Part 4) — not decided ad hoc when writing tests. This ensures no acceptance criterion is “missed” without any automatic verification at all.

Detecting Drift: Agents That “Pass” but Deviate from Intent

This is a specific and quite dangerous problem in the AI-generated code context: an agent can produce an implementation passing all tests, but actually optimizing to pass those tests themselves — not to satisfy the intent behind them. This phenomenon is similar to overfitting in machine learning: a solution fitting the existing test cases very well, but failing to generalize to cases not covered by tests.

A concrete example: the acceptance criterion states “if the email is not registered, the system shall still respond with the same success message” to prevent email enumeration. An agent could make an implementation that literally passes this test — the API response is indeed identical — but still leaks information through another path not covered by tests, for example significantly different response times between registered and unregistered emails (because only one path truly does password hashing or additional database queries). Tests pass, but the intent (preventing enumeration) isn’t truly achieved.

ANTI-PATTERN (passes tests, deviates from intent):
func RequestPasswordReset(email string) Response {
    if !emailExists(email) {
        return Response{Accepted: true} // returns immediately, fast
    }
    token := generateToken()           // slow process only if it exists
    saveTokenHash(token)
    sendEmail(email, token)
    return Response{Accepted: true}
}
// Response body is identical, but response time leaks information
// through a timing side-channel

CORRECT (intent truly satisfied):
func RequestPasswordReset(email string) Response {
    // Always run operations with similar duration, regardless of
    // whether the email is registered, to prevent a timing side-channel
    token := generateToken()
    if emailExists(email) {
        saveTokenHash(token)
        sendEmail(email, token)
    }
    // artificial delay if needed to equalize timing
    return Response{Accepted: true}
}

The most effective way to prevent this kind of drift isn’t adding more tests reactively after discovery, but writing constraints in the spec explicitly naming the threat to prevent — as discussed in Part 2, security constraints must be stated explicitly, not assumed. A spec stating “must prevent email enumeration, including through timing side-channels” gives the agent a clear signal that response time is also part of the contract, not just the response body.

Tests only verifying the final output are vulnerable to implementations that “technically pass” but are wrong in intent. For sensitive constraints (security, privacy), write tests verifying deeper properties than just output — for example timing consistency, not just response body similarity.

Tests as a Guardrail While the Agent Iterates

In Part 4, it was discussed that spec-driven execution runs incrementally per task group, with review checkpoints in between. The existing test suite from previous task groups functions as an automatic guardrail while the agent works on the next task group — changes accidentally breaking already-verified behavior will be immediately caught, without waiting for manual review to find them.

sequenceDiagram
    participant Agent
    participant TestSuite
    participant Reviewer

    Agent->>TestSuite: Implement Task Group 2
    TestSuite-->>Agent: All Task Group 1 tests still pass
    Agent->>Reviewer: Request Task Group 2 review
    Reviewer->>Agent: Continue to Task Group 3
    Agent->>TestSuite: Implement Task Group 3
    TestSuite-->>Agent: Task Group 1 test FAILS (regression detected)
    Agent->>Agent: Fix before continuing

This guardrail matters especially because agents, unlike human developers usually familiar with the codebase as a whole, don’t always have implicit understanding of which code parts depend on the behavior being changed. A comprehensive test suite replaces that implicit understanding with explicit signals: if a new task group’s change makes old task group tests fail, that’s a clear sign of a regression to handle before continuing — regardless of whether the agent “realizes” the relationship between the two code parts or not.

A recommended practice: run the entire test suite (not just tests for the newly worked task group) at every checkpoint, and make “all tests pass” a requirement before the agent is allowed to continue to the next task group — consistent with the definition of done discussed in Part 4.

Meaningful Coverage vs Coverage for the Number

Code coverage numbers are often used as a proxy for “how well the code has been verified”. In the SDD context, this number can be misleading when used uncritically, because coverage measures code lines executed while tests run — not whether the acceptance criteria in the spec are truly verified.

Two codes with 100% coverage can have very different verification quality:

// 100% coverage, but doesn't truly verify the acceptance criteria
func TestResetToken(t *testing.T) {
    token := generateToken()
    if token == "" {
        t.Fail()
    }
    // lines executed: yes. acceptance criteria verified: no.
    // No assertions about the 15-minute expiry, UUID v4 format, or
    // the token being stored in hashed form
}

// Coverage may be the same, but acceptance criteria are truly verified
func TestResetToken(t *testing.T) {
    token := generateToken()

    assert.True(t, isValidUUIDv4(token))
    assert.True(t, isStoredAsHash(token))

    expiry := getTokenExpiry(token)
    assert.WithinDuration(t, time.Now().Add(15*time.Minute), expiry, time.Second)
}

A more meaningful measure for SDD isn’t “what percentage of code lines are executed”, but “what percentage of acceptance criteria in the spec have explicit tests verifying them”. This mapping can be done simply — list each acceptance criterion line in the spec, and mark which test case verifies it, similar to a traceability matrix.

Traceability like this is more informative than a single coverage number, because it directly shows the specific gap — not just “15% less coverage” without knowing which part actually remains unverified.

Before considering a task group done, check the traceability from acceptance criteria to tests, not just the coverage number. High coverage with weak assertions gives a false sense of security — exactly like the formal-looking but actually ambiguous spec discussed in Part 2.

Testing Anti-Patterns in SDD

Several patterns weaken the test function as spec verification, even though the tests themselves “exist” and “pass”:

Tests written after code manually passes, not from acceptance criteria. This order reverses the basic SDD principle discussed at the article’s start — tests written to match the behavior of existing code tend to validate whatever happens to occur, not what should happen per the spec.

Tests too attached to implementation details. Tests asserting internal structure (for example internal variable names, private function call order) instead of externally visible behavior become fragile against legitimate refactoring, and create unnecessary friction every time the agent changes the implementation approach without changing the behavior actually required by the spec.

Ignoring tests for non-functional constraints. Constraints like performance (response time), security (timing side-channels as in the example above), or compatibility are often not tested at all because they’re considered “hard to test” compared to functional acceptance criteria. Yet these constraints are precisely the most common source of serious problems when assumed the agent will “automatically be correct”.

Running tests only at the end, not at every checkpoint. Consistent with the anti-pattern in Part 4 — postponing verification until all task groups finish removes the test function as an early guardrail against regressions.

Considering high coverage equal to a fully verified spec. As discussed above, coverage is a measure of executed code lines, not a measure of conformity with acceptance criteria. The two can correlate, but aren’t identical.

Summary

• SDD reverses the classic TDD order: acceptance criteria in the spec are written first as a reviewed contract, then tests are derived from them — not tests written based on informal assumptions
• The EARS notation from Part 2 can be almost directly mapped to a Given-When-Then structure, making the spec-to-test-case translation consistent across whoever reads it
• Every acceptance criterion should be mapped to the right testing layer — unit tests for single logic, integration tests for component interaction, contract tests for API schema conformity from Part 3, end-to-end tests for complete flows
• AI agent code can pass tests but still deviate from intent (similar to overfitting) — sensitive constraints like security must be stated explicitly in the spec, and tests for these constraints need to verify deeper properties than just the final output
• The existing test suite functions as a guardrail while the agent works on the next task group — regressions in previous task groups must be immediately detected before continuing
• High code coverage isn’t equal to fully verified acceptance criteria — traceability from each criterion to the relevant test case is more meaningful than a single percentage number
• Avoid writing tests after the code “looks correct” manually, tests attached to implementation details, and ignoring verification for non-functional constraints like performance and security