Why Your AI Keeps Reintroducing the Same Bugs

The dashboard package in a reference app we reviewed still had the default failing npm test placeholder. The root package had dependencies but no test scripts. There were no *.test.* or *.spec.* files anywhere. Quality lived in manual scripts — inspection harnesses, one-shot cleanup jobs, dedup scripts, database repair tools. Some previewed behavior. Some patched production-shaped data. None of them could answer the one question that matters on every pull request: "will this bug come back?"

It is a familiar shape. When an agent fixes the same bug twice, the tempting response is to write a clearer prompt. Sometimes that helps. More often, the repeated bug is telling you that the codebase has not learned the lesson yet.

AI tools move quickly through local context. They struggle when the product rule only lives in a chat thread, a commit message, a manual checklist, or one developer's memory. The agent sees the current code — it does not automatically know the incident history that taught the team why the current behavior matters.

Recurring bugs are rarely just prompt failures. They are missing executable memory: tests, types, schemas, and lint rules that encode lessons the team already learned once. (We make the types half of that argument in Types Are Memory for Your Codebase.) That gap is what a testing strategy has to close.

Why Agents Repeat Bugs

An agent repeats a bug when the codebase makes the local fix easy but does not preserve the reason behind it. The next session sees a function, a route, a script. It does not see the outage that came before, the customer complaint, the data repair, or the Slack thread where the product rule was finally pinned down.

A few patterns make this worse.

Duplicated behavior. If two import paths normalize the same field differently, a fix in one path does not teach the other. The bug looks fixed in the demo and survives in the batch job.

Manual scripts as quality gates. A script that prints a quality report helps during discovery, but it is not a regression test unless it has stable fixtures, clear assertions, and a place in CI. If humans have to remember to run it, the lesson has not been captured.

Over-mocked tests. A unit test that mocks the database, the queue, and the external adapter might prove that a function called its collaborators. It does not prove the write is idempotent, the migration supports the query, or the UI handles the actual response shape.

Prompt-only rules. "Remember to dedupe records by normalized name" is useful once. A test that fails when canonicalization changes is useful every time.

Turn the Bug into a Guardrail

The question worth asking after a repeated bug is not just "what broke?" — it is "what should keep this from breaking again?" The useful unit of work is an incident-to-guardrail loop: bug report, failing regression test, code fix or constraint, and an instruction update for the next agent.

Here is a copyable pattern for a common AI-app failure: an import job creates duplicate company records after a retry because one path normalizes names and another path writes the raw string.

1. Capture the bug report

Keep the report short enough that it can become a fixture.

Incident: duplicate company records from import retries
 
Observed:
 
- Importing "Northstar Labs, Ltd." creates one company.
- Retrying the same CSV with " northstar labs " creates a second company.
 
Expected:
 
- Within one workspace, both rows resolve to the same company identity.
- The retry updates the existing record instead of inserting a duplicate.
 
Guardrail required:
 
- Failing regression test for company import identity.
- One canonicalization path used by the importer and persistence layer.
- Database uniqueness constraint on workspace + canonical company name.
- Agent instruction update so future import edits preserve the invariant.

That report does more than describe the symptom. It names the product rule, the fixture, and the guardrails the fix must leave behind.

2. Write the failing regression test

Write this before the fix, or in the same commit if the team does not practice strict test-first development. The point is that the test should fail against the buggy behavior and pass only when the identity rule is encoded.

import { describe, expect, test } from 'vitest';
 
import { buildCompanyImportKey } from '@/imports/companyIdentity';
 
describe('company import identity', () => {
  test('deduplicates names across casing, whitespace, punctuation, and suffixes', () => {
    const rows = [
      { workspaceId: 'ws_123', companyName: 'Northstar Labs, Ltd.' },
      { workspaceId: 'ws_123', companyName: ' northstar labs ' },
    ];
 
    expect(new Set(rows.map(buildCompanyImportKey))).toEqual(new Set(['ws_123:northstar labs']));
  });
 
  test('keeps matching company names isolated by workspace', () => {
    const rows = [
      { workspaceId: 'ws_alpha', companyName: 'Northstar Labs' },
      { workspaceId: 'ws_beta', companyName: 'Northstar Labs' },
    ];
 
    expect(rows.map(buildCompanyImportKey)).toEqual([
      'ws_alpha:northstar labs',
      'ws_beta:northstar labs',
    ]);
  });
});

Run that against the buggy importer — the one that persists companyName as typed, with no canonicalization — and it fails the way the incident report predicts:

✗ company import identity > deduplicates names across casing, whitespace, punctuation, and suffixes
  AssertionError: expected Set{ 'ws_123:Northstar Labs, Ltd.', 'ws_123: northstar labs ' }
    to equal Set{ 'ws_123:northstar labs' }

That failure is the guardrail. buildCompanyImportKey doesn't exist yet at this point; writing it to canonicalize consistently is what turns the red above green.

Notice what the test name protects: the business identity rule. It does not assert that a particular regex was called. That gives an agent room to improve the implementation without erasing the lesson.

3. Fix the code and add a constraint

The code fix should create one import identity path and make every writer use it. Keep the domain-specific suffix list in one place.

export interface CompanyImportRow {
  workspaceId: string;
  companyName: string;
}
 
export function canonicalCompanyName(name: string): string {
  return name
    .normalize('NFKC')
    .trim()
    .toLowerCase()
    .replace(/&/g, ' and ')
    .replace(/\b(?:inc|llc|ltd|limited)\.?\b/g, '')
    .replace(/[^\p{L}\p{N}]+/gu, ' ')
    .replace(/\s+/g, ' ')
    .trim();
}
 
export function buildCompanyImportKey(row: CompanyImportRow): string {
  return `${row.workspaceId}:${canonicalCompanyName(row.companyName)}`;
}

If the data store can enforce the same rule, add a constraint too. Tests catch future code drift; constraints catch bad writes from paths the test did not exercise.

alter table companies add column canonical_name text;
 
-- Backfill with the same canonicalization rule before setting not null.
alter table companies alter column canonical_name set not null;
 
create unique index companies_workspace_canonical_name_unique
  on companies (workspace_id, canonical_name);

For a live table, the migration needs a real backfill and duplicate cleanup plan, and the index itself should be built with CREATE UNIQUE INDEX CONCURRENTLY so it doesn't hold a table-wide lock across every write while it builds. The important part is the invariant: the importer, the stored value, and the uniqueness rule all agree on company identity.

4. Update the agent instruction

Finally, put the lesson where the next coding session will actually read it. This belongs in the most local instruction file that owns the import area, not in a giant global prompt. (Where those instruction files live and what they should cover is the subject of Your App Needs a Constitution.)

## Import identity
 
When editing company import code, preserve `workspace_id + canonical_name` as
the company identity. Use `canonicalCompanyName` and `buildCompanyImportKey`;
do not add a second name-normalization path.
 
Bug-fix PRs for duplicate imports must include one of:
 
- a regression fixture in `tests/imports/companyIdentity.test.ts`
- a database constraint or migration that enforces the same invariant
- a short note explaining why the guardrail lives somewhere else

That instruction is useful because the test and constraint make it enforceable. The prompt points to the guardrail; it is not trying to be the guardrail.

The same pattern works for authorization, payments, queues, and data imports:

1. Reproduce the bug with the smallest stable fixture.
2. Write the failing test before or alongside the fix.
3. Name the assertion after the business rule, not the implementation detail.
4. Add the strongest cheap constraint the system can enforce.
5. Update local agent instructions with the invariant and the guardrail path.

Pick the Right Kind of Test

You do not need thousands of brittle tests. You need tests placed at the risk boundaries where repeated mistakes are expensive.

Unit tests are for deterministic logic: parsers, formatters, permission decisions, state reducers, idempotency-key builders, date and currency normalization. Vitest and Jest both make this straightforward. The discipline that matters is not which framework you pick — it is that the test runs fast, covers a specific behavior, and is on by default.

Integration tests belong where the bug depends on two pieces agreeing. Database writes, API contracts, queues, generated types, and third-party adapter shapes tend to break between modules rather than inside a single pure function. If a recurring bug required a cleanup script to fix, ask which integration boundary let bad data in. Put the fixture there.

End-to-end tests (Playwright is a good choice) protect browser-visible critical flows. They are slower, so they should guard the routes that matter most: sign-in, checkout, admin writes, destructive actions, anything where a false green build could hurt a customer. Playwright's locators and web-first assertions describe what the user sees, not which CSS selector happened to exist this week.

Property-based tests shine when a bug is about a rule that should hold across many inputs. Deduplication, slug generation, sorting, normalization, and state transitions are good candidates. Hand-picked examples might miss the next variation. A property says "this rule always holds" and lets the framework find the counterexample:

import { expect, test } from 'vitest';
import fc from 'fast-check';
 
test('normalizing twice equals normalizing once', () => {
  fc.assert(
    fc.property(fc.string(), (raw) => {
      expect(normalize(raw)).toBe(normalize(normalize(raw)));
    }),
  );
});

Choose the cheapest test that would have caught the bug before a human noticed it. Martin Fowler's practical test pyramid and Google's test-size taxonomy are both worth reading — not because the shapes are sacred, but because they force you to think about feedback speed, blast radius, and maintenance cost.

A Risk Matrix Beats Coverage Theater

Coverage percentage is a weak proxy for product risk. A codebase can have high line coverage and still miss the one behavior that keeps money, private data, or customer trust safe.

For AI-built apps, a risk-based matrix is more useful:

This table doubles as an agent instruction. Instead of telling the agent to "add more tests," point it at the row for the risk it just touched: "this PR touches background jobs, so add a queue/job test with retry and failure fixtures." That produces smaller diffs and better review conversations than a generic ask.

Mock Discipline Matters

Mocks themselves are fine. The problem is unowned mocks — test doubles that exist only to make the test pass, with no connection to a real contract.

When a test replaces every collaborator with whatever response makes assertions green, the agent can satisfy the test while breaking the product. Keep mocks close to stable contracts. Prefer typed fixtures over anonymous objects. When a mock represents an external service, make the fixture look like the real payload and name the fields the product actually depends on. Testing Library's guiding principle — tests should resemble the way the software is actually used — applies as much to a mocked queue consumer as it does to a rendered component.

For repeated bugs, ask whether the test lives at the wrong level. A parser bug belongs in a unit test. A database transaction bug belongs in an integration test. A lost confirmation message belongs in a browser test. A brittle deduplication rule may need both hand-picked examples and a property.

This is also where manual validation scripts should graduate. An ad-hoc inspection harness can stay useful for exploration, but its stable assertions should move into the test suite. And if you have a cleanup script that removes duplicate records, ask the harder question: why can those duplicates still be created? If the answer is known, it deserves a test, a constraint, or both.

How the Diagnostic Scores This

In a Post Code Labs diagnostic, repeated bugs surface as a testing and regression-control score — but they rarely stay in one lane. They usually connect to types, architecture, CI/CD, and agent instructions. These checkpoints are part of the full Post Code Labs consultation, which scores testing as one of its twelve categories.

Pass: the important test commands are real, documented, and wired into CI. Bug fixes routinely include a regression test or equivalent guardrail. Critical flows have an appropriate mix of unit, integration, and end-to-end coverage. Manual scripts are exploration tools or controlled jobs, not the primary quality gate.

Partial: some coverage exists, but the riskiest flows still rely on manual checks, local habits, or shallow mocks. The team has tests but no shared rule for what kind of test should accompany a bug fix.

Gap: the repository cannot answer basic regression questions. npm test fails by default, no test files exist, or quality depends on one-off scripts that someone runs after the damage is done.

The output is evidence, not vibes: file paths, commands, recent incidents, missing fixtures, and a risk-ranked 30-60-90 plan — starting with making the default test command meaningful, then protecting the highest-risk flows, then consolidating manual scripts into maintained test surfaces.

What to Do Next Monday

Start small. Asking an agent to "write a full test suite" across the app produces noisy coverage and review fatigue.

Make the default command honest. If the project uses pnpm, pnpm test should run the real suite or intentionally delegate to the documented command. A placeholder failure is useful on day one. After that, it just tells you nobody has made regression control part of the workflow.

Audit your recent bugs. List the last five that returned or required manual cleanup. For each, name the guardrail that would have caught it: a unit test, integration test, Playwright flow, property, type, schema, lint rule, database constraint, or CI gate.

Graduate one manual script. Pick a validation or inspection script and extract its stable checks into a test with proper fixtures and assertions. Keep the script for exploration if you like, but stop treating printed output as proof of correctness.

Update agent instructions. A good default rule: "Bug-fix PRs must include a regression guardrail, or explain why the guardrail is a type, schema, lint rule, database constraint, or operational check instead of a test."

Tag tests by risk and owner. The team should know which tests protect auth, payments, data writes, imports, background jobs, and admin actions. That map tells you more than a coverage percentage ever will.

Tests do not just prevent regressions — they communicate. Every regression test is a record of something the team already figured out the hard way. The goal is a codebase where the next contributor, human or AI, cannot accidentally ignore that knowledge.

References

• Vitest: Writing Tests

  [vitest.dev]
• Jest: Expect[jestjs.io]
• Playwright: Locators[playwright.dev]
• Playwright: Assertions[playwright.dev]

• fast-check: What is Property-Based Testing?

  [fast-check.dev]

• Testing Library: Guiding Principles

  [testing-library.com]

• Martin Fowler: The Practical Test Pyramid

  [martinfowler.com]

• Google Testing Blog: Test Sizes

  [testing.googleblog.com]