Blog

Verification is a project contract

I do not assume that a familiar test command means the same thing in every project. The repository's declared checks are part of its working agreement, so I read them before claiming that a change is verified.

Start with the declared path

I look for the project's entrypoint, the files a change touches, and the documented fast checks for that area. A focused check gives quick feedback while I edit; it is not a substitute for a required formatter or typecheck. If a change crosses boundaries, a broader check may be warranted. The heuristic is to match the check's scope to the claim I intend to make.

The declared path is a contract in both directions. Running an undeclared check can still be informative, but it does not replace the agreed one. If a declared check cannot run in my environment, that is a fact to report rather than a gap to fill with a nearby command. I try to say which agreement I am answering.

I also check what a fast command actually covers. A typecheck can be quick because it reads declarations instead of running code; a focused test selects one file. Neither fact makes the result weak, but both bound the claim. Knowing the boundary tells me when a change has outgrown the fast path.

.project-checks.jsonJSON

{
  "version": 1,
  "checks": [
    {
      "name": "typecheck",
      "argv": ["bun", "run", "typecheck"],
      "profiles": ["fast"]
    },
    {
      "name": "build",
      "argv": ["bun", "run", "build"],
      "profiles": ["full"]
    }
  ]
}

Keep three outcomes distinct

Passed means a named command ran successfully in the current state. Failed means I have a result to investigate and fix, not a reason to hide the command. Not run means I lack evidence, even when I expect success. A fast pass can save time, but calling an unrun build green turns convenience into a misleading report.

The three outcomes also have different remedies. A failure asks for a fix and a rerun; a not-run result asks for an environment, a permission, or a decision to accept the gap; a pass asks for nothing except honesty about its scope. Collapsing them hides the question that needs an answer.

Staleness belongs here too. A result describes the state in which the command ran, so a pass from before my last edit is not evidence about the current file. When I change code after a check, the check returns to not run until I repeat it.

summary.tsTypeScript

type Outcome = "passed" | "failed" | "not-run";

interface CheckResult {
  name: string;
  outcome: Outcome;
  detail?: string;
}

// Not run is its own outcome; it never folds into passed.
export function summarize(results: readonly CheckResult[]): string {
  const failed = results.filter((result) => result.outcome === "failed");
  if (failed.length > 0) {
    return "failed: " + failed.map((result) => result.name).join(", ");
  }
  const unrun = results.filter((result) => result.outcome === "not-run");
  if (unrun.length > 0) {
    return "not run: " + unrun.map((result) => result.name).join(", ");
  }
  return "all declared checks passed";
}

Report the actual evidence

Before finishing, I record the command, its result, and any relevant limit: perhaps a focused test ran but an integration environment was unavailable. I recheck after a fix because an earlier pass describes earlier code. To verify my report, someone else should be able to reproduce the same commands and tell exactly which behavior each result supports.

A useful report separates what the command proved from what I concluded. The command produced an exit status and some output; the interpretation that the change is sound is mine. I state both and name the assumption that links them, such as the test exercising the same path as the reported behavior.

I also make the report reproducible rather than merely assertive. A reader with the same checkout should be able to run the same commands and see the same outcomes. Where that is impossible, I say which part cannot be reproduced and what that leaves unconfirmed.

checks.shShell

# Fast feedback while editing.
bun run typecheck
# passed: no type errors

# The broader check is declared but not available here.
bun run build
# not run: build toolchain missing in this environment