Surveyor

Surveyor is Field's before/after verification system for bugs that need a real runtime and visible proof. It runs a scenario against a known-bad ref, captures evidence, runs the same scenario against a candidate ref, and writes a comparison a human can read and a machine can publish.

The oracle #

Verified means the baseline failed and the candidate passed. A candidate that passes on its own proves nothing: it may never have exercised the reported path.

A run has exactly four honest outcomes:

The distinction between not_verified and harness_failure is the single most important property here. A flaky environment must never read as a product verdict.

Commands #

# what can I run
npm run qa:surveyor -- list

# prove a fix: baseline must fail, candidate must pass
npm run qa:surveyor -- run \
  --scenario server-boot-smoke \
  --baseline <bug-ref> \
  --candidate HEAD

# check the harness itself, one lane only
npm run qa:surveyor:smoke

Output lands in .artifacts/surveyor/<scenario>-<timestamp>/:

surveyor-report.md        human-readable; safe to paste into a PR or issue comment
surveyor-summary.json     machine-readable source of truth
surveyor-evidence.json    publish manifest (schemaVersion 1)
comparison.json           the oracle result
run.log                   phase markers and lane output
baseline/checkout/        detached worktree for the baseline ref
candidate/checkout/       detached worktree for the candidate ref

How a run works #

1. Resolve baseline and candidate refs to SHAs.
2. Create a detached worktree per lane under the output directory. Never git checkout in the working tree — sync and dirty state mix proof across lanes, which is an upstream lesson we did not have to learn the hard way.
3. Install dependencies and build only what the scenario needs, per lane.
4. Run the scenario with an isolated data root, an isolated port, and a generated throwaway auth token.
5. Capture the oracle result and any artifacts.
6. Compare, write the summary, evidence manifest, report and log.
7. Remove the lane worktrees unless --keep-worktrees.

Every stage prints SURVEYOR_PHASE:<stage>:<lane> into run.log, so a failure names the stage instead of "the run failed".

Writing a scenario #

A scenario declares its whole contract. validateScenario() rejects a partial one, because a scenario that does not state its oracle cannot produce trustworthy proof.

export const quoteTotalRegression = {
  id: "quote-total-regression",
  title: "Quote total excludes hidden-amount nodes",
  transport: "http",                 // http | browser
  requiredCredentials: [],           // names only, never values
  baselineExpectation: "total includes the hidden node",
  candidateExpectation: "total excludes the hidden node",
  visualCaptureTargets: [],
  timeoutMs: 120_000,
  async run({ checkoutPath, laneDir, lane, log }) {
    // ... set up, stimulate, read the oracle
    return { status: "pass" | "fail" | "error", oracle: { ... }, notes: [] };
  },
};

Rules:

• error is for the harness, fail is for the product. A failed npm install, an occupied port or a missing config returns error. Only a real observation of wrong behaviour returns fail.
• Prefer a typed oracle over a screenshot. If an API response can prove the bug, that is the pass/fail signal; screenshots are for human confidence.
• Isolate state. Own data root, own port, own token. Never the operator's config or a real client dataset.
• A Surveyor run should become a regression test once the fix is understood. The scenario is scaffolding for diagnosis, not a permanent substitute for a cheap test.

Machines #

Today Surveyor runs local only. --provider libvirt|container|ssh fails closed rather than running locally while reporting a remote machine. Anything that executes steps supplied by a bug reporter belongs in an isolated VM.

Evidence is customer data #

Field screenshots and API payloads can contain client names, addresses, job sites and prices. Treat every artifact as customer data:

• never commit .artifacts/
• never attach raw evidence to a public issue
• report credential presence and length, never values
• prefer a link to a local artifact directory over an inline image when the audience is not the operator

What is not built yet #

Remote lanes, visual artifacts, motion previews, artifact hosting and a PR or issue comment publisher are deliberately deferred.