The Confirmation Bias Problem in Manual Testing

When an engineer writes a function and then writes tests for it, both activities share the same mental model of how the function should behave. The tests confirm what the engineer already believes — which is precisely why they miss the cases that actually cause production bugs.

This isn't a skill problem. It's a structural one. The person who knows the code best is the worst person to write adversarial tests for it. They'll test parseAmount("100") and parseAmount("100.50"), but not parseAmount(""), parseAmount(null), parseAmount("1e308"), or parseAmount(" 100 ") with leading and trailing whitespace.

The engineer who wrote the parser knows it expects a string. So they test strings. AI has no such assumption — it generates the full space of what a string can be.

In our experience, a large share of production bugs could have been caught by tests that were theoretically possible to write but were never written. Not because engineers are careless, but because the mental model that lets you build something also blinds you to how it can break.

What AI Generates Differently

When we feed a function to an AI model and ask for a test suite, it doesn't start from intent — it starts from the function's type signature, control flow, and any documented contracts. The output typically covers four categories that human-written tests consistently underweight:

1. Boundary values

Off-by-one errors are the oldest bug in software. AI-generated tests reliably hit n - 1, n, and n + 1 for every numerical boundary in the function, including ones the engineer didn't think of as boundaries because they're implicit in a downstream condition.

2. Type edge cases

In dynamically typed languages especially, functions receive inputs the author never intended. undefined instead of null, 0 instead of an empty array, a single object instead of an array of one. AI-generated tests enumerate these systematically.

3. Concurrency and ordering assumptions

For async code, AI will generate tests that verify the function handles resolved and rejected promises in different orders — something engineers rarely write explicitly because the happy path always resolves first in their mental model.

4. State mutation side effects

When a function modifies its input rather than returning a new value, AI-generated tests frequently catch that the input object is mutated in ways the caller doesn't expect — a common source of subtle bugs in shared state systems.

A Concrete Example

Here's a simplified version of a payment amount validator we worked with on a recent FinTech engagement. The human-written test suite had 6 tests, all of which passed. The AI-generated suite had 24 tests — 4 of which failed, revealing bugs that had been dormant in a function running in production for eight months. 

// The function under test
function parseTransactionAmount(input: string): number {
  const cleaned = input.replace(/,/g, '').trim();
  const amount = parseFloat(cleaned);

  if (isNaN(amount) || amount <= 0) {
    throw new Error('Invalid amount');
  }

  return Math.round(amount * 100); // store as cents
}

The 4 failing AI-generated tests exposed:

  • Scientific notation: "1e2" returns 10000 (100 dollars in cents), not an error. Whether that's correct depends on the product — but it was never a deliberate choice.
  • Negative zero: "-0" passes the amount <= 0 check because -0 <= 0 is true in JavaScript, but isNaN(-0) is false. It returns -0 in cents, which fails downstream integer assertions.
  • Very large values: "9999999999999999" returns a different value than expected due to floating-point precision loss before the Math.round call.
  • Whitespace-only input: " " passes the trim and replace, becomes an empty string, parses to NaN, and throws correctly — but the error message was missing required context for the upstream error handler.

None of these were edge cases an engineer would naturally think to test after writing this function. All four were caught by AI test generation in under 90 seconds.

Where AI Still Needs Human Review

AI-assisted testing is not a replacement for engineering judgment — it's a way to extend it. There are three areas where the generated tests require careful human review before they go into the suite:

  • Tests that assert wrong behaviour. AI infers expected outputs from the code itself, not the intent. If the code is wrong, some generated tests will assert that the wrong behaviour is correct. Every test that defines an expected value needs a human to verify that value is actually right.
  • Integration test setup. Unit tests for pure functions are excellent. Tests that require database state, external APIs, or message queues need human scaffolding — the AI generates the assertions but not the infrastructure setup.
  • Business rule edge cases. AI doesn't know that in your domain, a zero-dollar transaction is valid (for free trials) even though the validator above would reject it. Domain knowledge still has to come from the engineer.

How to Use AI for Unit Testing: Our Workflow

On every engagement, we use a two-pass approach to test generation:

  1. AI first pass: Run Claude or Copilot against each new function or module, generate a candidate test suite, and commit it to a review branch.
  2. Engineer review: A senior engineer reads each generated test, verifies expected values against the spec, removes tests that assert incorrect behaviour, and adds any domain-specific cases the AI missed.
  3. Coverage gap analysis: Run the combined (AI + manual) suite through a coverage report. AI-generated tests usually reach high branch coverage before the human review, and the review closes the remaining gaps.

The result: test suites that would take days to write by hand are reviewed and production-ready within hours. The engineers spend their time on judgment, not on typing expect(fn(null)).toThrow() for the fourteenth time.

Bottom Line

AI-assisted testing isn't about replacing engineers — it's about redirecting their attention. The mechanical work of enumerating inputs belongs to AI. The work of knowing what the correct output should be, understanding the domain, and deciding which edge cases are actually worth testing in your context — that belongs to the engineer.

Used correctly, it's one of the highest-leverage AI applications in the engineering workflow. Every hour spent reviewing AI-generated tests pays back in production bugs that never happen.