All Field Notes

CI/CD Is the Adult Supervision Your Vibe-Coded App Needs

How automated delivery checks turn AI coding from a risky interactive loop into a controlled path to production.

CI/CDTestingDeployment

Most of the AI-built codebases we review have automation. They have scripts, cron jobs, deploy wrappers, and build commands. What they rarely have is a release gate — something that actually decides whether a change is safe to ship before it reaches production.

That gap is easy to miss because work is visibly moving. A scheduled job pulls the latest branch and restarts the server. A data pipeline commits its own output and pushes to main. Someone points to a test command in package.json, but it is still the scaffolded placeholder that exits with an error code. The team has automation. What they do not have is supervision.

CI/CD fills that role. It gives the product an external standard — a set of checks the team agrees must pass before any change, generated or handwritten, moves toward production. For a codebase where most of the code came from an AI prompt, that standard is the difference between shipping with confidence and shipping on faith.

Automation Is Not the Same as a Gate

In one review we conducted, the app had no shortage of scripts. The dashboard package had a test command that was still the scaffolded "no test specified" placeholder. The root package listed runtime dependencies but had no project-level scripts at all. A few files had "test" in their names, but none of them added up to a test harness the release process could actually depend on.

The database told a similar story. The migrations directory had repeated numeric prefixes, lettered variants, date-named files, and one unnumbered SQL file mixed in with the rest. Any single migration might have been fine on its own. The real risk was that no tool or process could answer a straightforward question: in what order should a fresh environment apply this history?

There was also a production wrapper — a scheduled job that wrote a long shell script onto a server, pulled code, ran a multi-step data pipeline, tolerated some git sync failures, invoked an agent, consolidated results, and sent a notification. Separately, an apply script refused to run on a dirty checkout, then pulled the primary branch, applied pending writebacks, committed archive moves, and pushed with retries.

None of this was careless. It was the natural result of a team building automation to get work done without stepping back to ask whether any of it could block a bad change from reaching production. The work happened after the fact, on machines with production authority, instead of being reviewed before it could alter the product.

What Good CI Should Prove

A CI pipeline does not need to be elaborate on day one. It needs to be authoritative. When it passes, the team should know which risks were checked. When it fails, the change should not merge just because the local demo looked fine.

For a typical TypeScript and Next.js project, the first gate should cover at least these areas:

  1. Dependencies install from the lockfile, not from whatever happens to resolve today.
  2. TypeScript compilation, framework-generated types, linting, and formatting all agree with the patch.
  3. The test suite passes in a clean environment, not just on the developer's machine with warm caches and local state.
  4. The production build succeeds.
  5. Database migrations apply in a deterministic order and are validated against a disposable database or schema check.
  6. A dependency audit flags known vulnerabilities before they reach production.
  7. A preview deploy or equivalent artifact exists so someone can review runtime behavior before production changes state.

GitHub Actions is a common place to encode that contract. The exact commands should match your repo, but the required checks should be concrete enough that branch protection can enforce them:

name: ci
 
on:
  pull_request:
  push:
    branches: [main]
 
jobs:
  install-lockfile:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: corepack enable
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
 
  static-analysis:
    runs-on: ubuntu-latest
    needs: install-lockfile
    steps:
      - uses: actions/checkout@v4
      - run: corepack enable
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm audit --audit-level=high
      - run: pnpm run check
 
  production-build:
    runs-on: ubuntu-latest
    needs: static-analysis
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready --health-interval 10s --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - run: corepack enable
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm run build
      - run: pnpm run verify:migrations
        env:
          PGHOST: localhost
          PGUSER: postgres
          PGPASSWORD: postgres

Then make the status checks from those jobs required before merge. A branch protected without required checks is a suggestion, not a gate — anyone can merge past a red build. Use the status-checks-only endpoint rather than the general branch-protection PUT: the latter treats an omitted or null field as "disable this," so a payload built from the job names alone would silently strip any review requirements or push restrictions the branch already has.

gh api repos/OWNER/REPO/branches/main/protection/required_status_checks \
  --method PATCH \
  -F strict=true \
  -F 'contexts[]=install-lockfile' \
  -F 'contexts[]=static-analysis' \
  -F 'contexts[]=production-build'

A useful gate spells out the evidence, the automatic repair boundary, and the point where the agent must stop:

CheckProvesAgent may fixAgent must stop
install-lockfiledependencies resolve from the committed lockfile in a clean runnerrestore the package-manager command or remove an unused dependency it addedthe fix requires changing dependency versions, registry access, or security posture
static-analysistypes, lint, formatting, tests, and audit policy all agree with the patchapply formatting, add a missing import, tighten an obvious type, or fix a failing test with clear intenta green result would require changing product behavior, deleting a test assertion, or accepting a vulnerability
production-buildthe deploy artifact builds and the schema history can be applied deterministicallyfix build-only imports, route metadata, or a migration filename/order issue that has one obvious correctionthe failure touches schema design, secrets, deploy permissions, preview environments, or rollback behavior

pnpm run check and pnpm run verify:migrations are aggregates, not single commands — each should fan out to the checks it claims to cover:

{
  "scripts": {
    "check": "pnpm run typecheck && pnpm run lint && pnpm run test",
    "verify:migrations": "jiti scripts/verify-migrations.ts"
  }
}

The migration command is intentionally project-specific. A Rails app, a Supabase project, and a hand-rolled PostgreSQL migration folder will each validate migrations differently. The principle is the same: apply the full migration history, in order, to a database nobody depends on, and fail the build if any migration errors out:

// scripts/verify-migrations.ts
import { execSync } from 'node:child_process';
 
const db = `verify_migrations_${process.env.GITHUB_RUN_ID ?? 'local'}`;
execSync(`createdb ${db}`, { stdio: 'inherit' });
 
try {
  execSync(`for f in db/migrations/*.sql; do psql "${db}" -v ON_ERROR_STOP=1 -f "$f"; done`, {
    stdio: 'inherit',
    shell: '/bin/bash',
  });
} finally {
  execSync(`dropdb ${db}`, { stdio: 'inherit' });
}

That pull request should prove that a clean environment can apply the full migration history in a known order before the change merges.

Be explicit about framework-level checks too. Next.js treats next build as the production verification path, and newer versions support next typegen to generate route types before TypeScript validation runs. If the app deploys to Cloudflare Pages, pin the build environment and runtime versions through the Pages configuration rather than hoping the deploy environment happens to match your laptop.

Agents Need Permission Boundaries

CI works especially well with coding agents because it gives them a tight feedback loop. An agent can fix a formatting violation, resolve a lint error, add a missing import, or correct a type mismatch without asking a human to re-evaluate the product. The check failed, the fix is mechanical, the agent handles it. (Why those mechanical rejections matter so much in AI-heavy repos is the subject of Linting Is Not Pedantry When an AI Is Writing Your Code.)

Not every red check is that simple, though. A migration conflict could mean two features touched the same data model and someone needs to decide which design wins. A dependency audit failure might call for evaluating whether to patch, replace, or accept the risk. A failed preview deploy could point to an environment or permissions issue that has nothing to do with the code itself. And a job that writes production data is a different category of problem entirely.

It helps to write the boundary down:

  1. Agents fix local, mechanical failures — the kind where the intended behavior is obvious from the check output.
  2. Agents stop when the fix would change a schema, touch secrets, alter deploy permissions, write persistent data, affect rollback behavior, or change what customers see.
  3. When the same check fails repeatedly, that is a finding for the team to investigate — not a signal for the agent to keep trying variations until the pipeline turns green.

This is where CI earns its name as adult supervision. It does not replace engineering judgment. It surfaces the moment when judgment is needed.

Boundaries like these belong in the repo's committed agent instructions — the project constitution we describe in Your App Needs a Constitution. Whether those instructions exist, stay current, and give an agent a safe operating envelope is exactly what the free Post Code Labs repo audit checks.

Release Metrics Keep the Gate Honest

It is easy for a CI pipeline to drift in either direction. Too permissive, and changes slip through that cause incidents. Too strict, and the team starts working around the pipeline because nothing ships on time. DORA's delivery metrics help you spot which failure mode you are living in: deployment frequency and change lead time tell you whether the team can ship; change failure rate and recovery time tell you whether shipping is creating avoidable damage.

An AI-assisted team can have high deployment frequency and still have poor release health if changes regularly need rollbacks or emergency patches. The balance to aim for is a release system that keeps lead time short without leaving a trail of hotfixes and manual data repairs.

This is also why silent production automation deserves scrutiny. A cron job that mutates state, commits the result, and pushes it upstream may look efficient. But if the job fails halfway through, who reviews the partial result? If it succeeds with bad data, which gate should have caught it? If a server has permission to push directly to the primary branch, who approved that release?

The answer is usually not to remove the automation. It is to change its authority. Let the job write an artifact, open a pull request, upload logs, or pause for approval before touching production state. The automation still runs — the risky decision just becomes visible.

How a Diagnostic Scores This

In a Post Code Labs diagnostic, CI/CD is scored on evidence, not on intent. These checkpoints are part of the full Post Code Labs consultation, which scores CI/CD as one of its twelve categories. A repo gets credit for checks that actually run in the merge path, block unsafe changes, and produce clear failure signals.

A pass means required pull-request checks cover install, typecheck, lint, tests, build, migration validation, and preview deployment. Production deploys trace back to specific commits. Long-running jobs emit logs and status. Rollback is a documented procedure the team has practiced, not a scramble improvised during an incident.

A partial result is the most common finding. The app has GitHub Actions, but the test suite is thin. The build runs in CI, but migrations are still applied manually. Preview deploys exist, but production jobs still pull code directly onto servers and run unreviewed scripts. This does not mean the team needs to stop and rebuild everything. It means the highest-risk release paths should get gates first.

A gap means important production changes happen outside any review process: direct pushes to main, manual database edits, untracked cron scripts, deployment wrappers written onto servers, or test commands that do not actually test anything. In the first 30 days of remediation, the focus is on making those paths visible. Over the next 30, the goal is making them reviewable. By 90 days, the primary release path should be predictable enough that agents can move quickly without anyone having to trust them on faith.

What to Do Next Monday

Start with the paths that can hurt users or data.

  1. List every way code, schema, configuration, scheduled jobs, and generated artifacts currently reach production. Most teams find at least one path they had forgotten about.
  2. Pick one required pull-request check and make it authoritative. For many teams, pnpm run check or an equivalent combined verification command is the right starting point.
  3. Add migration ordering validation before adding the next migration. Repeated numeric prefixes and ad-hoc naming are cheap to detect and expensive to debug in production.
  4. Remove silent pushes from production jobs. If automation needs to persist a result, have it open a pull request, upload a build artifact, or route through a human approval step.
  5. Review your last failed deploy against four questions: how long did the change take to reach production, how often are you deploying, did it require manual intervention, and how long did recovery take?

CI/CD is not about slowing down AI-assisted development. It is about knowing the difference between moving fast and moving recklessly. A good release system lets agents iterate quickly inside a path the product can actually survive.

References