# Lessons Learned

Chronological log of persistent production bugs and the fixes that finally
resolved them. Newest entries at the top. Each entry names the user-facing
symptom, the wrong diagnoses that delayed the fix, the actual root cause, and
the commit that landed it — so future debugging has the full trail.

---

## 2026-10-13 — The jobs board stays on "Loading jobs…" and restarts do not help (fixed in #109 and #110)

**Symptom:** On a live installation, `GET /api/jobs` stopped answering. The
board stayed on "Loading jobs…", and the browser gave up with an AbortError
after 30–60 s. `/health`, `/api/auth/me` and `/api/ui-config` answered, but
slowly (`/api/storage/status` took 1.9 s). Later, reads of single jobs also
ran past 12 s. Restarts did not clear it, and no server error reached Sentry,
because the requests were slow, not failing.

**Wrong diagnoses:**
- A stuck in-flight request. Restarts ruled this out.
- A lock that outlived the process: a job generation lock, a durable-storage
  claim, or a PostgreSQL advisory lock. None of Field's locks survives a
  restart. The generation lock is reclaimed once its owner is dead, and it
  gives up after 30 s.
- A release regression. The hang began on one release and continued on the
  next.

**Root cause:** every read of a job (`resolveActiveJobGeneration`) called
`quarantineUncommittedGenerations`, which checked *every* generation's commit
marker. It did this by reading and hashing all of that generation's files,
twice. Each generation holds the job's files, so one read cost about the sum
of all bytes in the job's whole history. That grows with history, faster than
linearly for a job whose generations add PDFs. The jobs board reads each job
several times (meta, quote artifacts, lifecycle, issuance ledger, issued
artifact), so the list cost the sum over every job. It worked until enough
history had built up, then crossed the browser's limit all at once. Heavy
reading of many jobs in a row hastened it, and it stayed slow after restarts
because the cost is in the data.

**Fix:**
- #110: a full check of a committed generation is remembered against a stat
  signature (path, type, size, inode, change time and modification time of
  every entry, and the marker's). Any write, rename or replacement changes the
  signature, so the bytes are checked again. Changed, removed and added files
  are still caught. On a synthetic job with 40 generations, 5 reads went from
  7.7 s to 0.16 s.
- #109: `GET /api/jobs` gives each job 5 s and the list 25 s. A job that runs
  over is left out (the response sets `X-Field-Jobs-Skipped`) and is logged and
  reported to Sentry by job id and step only. A slow job can never hang the
  board again, and the slow job is named.

**Prevention:**
- `server/job-generation.test.js` has a regression test: 20 reads of a job
  with 60 generations must finish within a fixed time.
- Any per-read work that scales with a job's history needs a cache or a test
  like that one.
- A route that loops over every job needs a time limit for each job.

---

## 2026-10-12 — Jobs open by their link but are missing from the jobs board (#32, fixed in #34)

**Symptom:** On a live installation the board listed 57 jobs while the data
directory held 65. The 8 missing jobs, all created in April 2026, were not
archived and opened by direct link.

**Root cause:** `GET /api/jobs` lists the `field.db` job index and never
checked it against the job folders. The index was filled from disk only once,
on a boot with an empty index (`syncExistingJobs`, which also read only the
job root's `meta.json`, not the active generation). After that, a job got its
row only from the route that created it. From April 2026 recorded jobs were
indexed under `<day>/<id>` and then renamed to `<day>/<slug>--<id8>`, and the
row was never updated (`upsertJob` did not update `dir` on conflict, so even a
direct read that found the job did not fix it). The list read the row's
missing folder, found no meta, and skipped the job silently. A job created by
the recording-session finalize route got no row at all.

**Fix:** `reconcileJobIndex` runs on every boot (after durable storage has
restored the data directory): it reads each folder the index does not point at
through `readJobMeta`, adds jobs that have no row and repoints rows whose
folder moved. The list reconciles again when a row's folder is gone, and
`upsertJob` records the folder a job is in now.

---

## 2026-04-12 — "Make Current" on archived quote version freezes the page

This entry predates the open-source release. The commits it names are not in
this repository's history, and the files it names have since been removed.
The rules at the end still apply.

**Symptom:** Clicking **Make Current** on an archived quote version caused the
screen to dim and then become completely unresponsive. Refreshing did not
persist the intended version flip. Reported on two customer quotes.
Survived across page refreshes.

**Final root cause (commit `63a8dbe`):** `client/src/features/job-detail/components/versions-sheet.tsx`
— and three sibling sheet components — used `position: absolute, inset: 0` on
the top-level sheet container. The positioned ancestor is the job-detail page's
content column, which grows to the full height of the quote list (**3259 px**
on a long quote). That made the sheet 3259 px tall, which in turn made the
confirm dialog overlay (`position: absolute, inset: 0` inside the sheet) 3259 px
tall as well. The centered dialog card therefore rendered at document
`y ≈ 1529` — **~1100 px below the bottom of a 417-px-tall viewport**. The user
saw the overlay darken the visible sheet area but the actual Cancel / Make
Current buttons were literally off-screen below the fold, so every click just
hit the backdrop and did nothing.

**Fix:** Change the top-level sheet containers (and the two confirm-dialog
overlays inside `versions-sheet.tsx`) from `position: absolute` to
`position: fixed`. `inset: 0` then resolves against the viewport, so the sheet
is always viewport-sized and the centered dialog card always lands in the
visible area regardless of page scroll or quote length.

Files touched:
- `client/src/features/job-detail/components/versions-sheet.tsx` (sheet + 2 confirm overlays)
- `client/src/features/job-detail/components/soil-picker-sheet.tsx`
- `client/src/features/job-detail/components/archive-dialog.tsx`
- `client/src/features/chat/full-chat-overlay.tsx`

**Wrong diagnoses (and the commits that came out of them — each useful in its
own right but none actually fixed the freeze):**

1. **"It's a stale code-split chunk from an old deploy"** (`d3dbfc9`). The
   theory was that the tab still held references to a content-hashed Vite chunk
   the server no longer has, so the lazy `import()` rejected silently and the
   sheet never opened. Matched the `SyntaxError: Unexpected token '<'` pattern
   you get when the SPA fallback serves `index.html` as JS. The enhancement to
   `ErrorBoundary` (auto-reload on `ChunkLoadError` with a sessionStorage guard
   against loops) is worth keeping — it will prevent that *class* of bug — but
   it was not the active cause.

2. **"`findJob` returns rows missing `date_dir`, so `upsertJob` hits
   `NOT NULL constraint failed: jobs.date_dir` and every promote silently
   fails in the best-effort try/catch"** (`5e3a933`). This *was* a real latent
   bug uncovered by the promote-endpoint milestone logs — the DB index refresh
   was failing on every promote — but it wasn't the user-visible freeze
   either. Worth keeping fixed.

3. **"Backend is slow, user needs a loading state and error surface"**
   (`2e163df`, `ff3afb3`). The promote endpoint does a lot of filesystem work
   and had no client feedback. Adding `promoteVersionBusy`, a shared version-
   action error banner, and per-step server milestone logs is good hygiene but
   was not the cause — the backend actually completes in ~500 ms.

**Why the real cause took so long to find:**

- **Playwright reproduces the flow successfully on the exact same URL.**
  Playwright auto-scrolls to click targets, so when it clicked the off-screen
  dialog button it scrolled the page and the click landed. In a human's
  browser, the click hit-tests the coordinate the button *should* be at and
  finds the backdrop instead. The discrepancy wasted at least two debugging
  rounds: "the backend works, Playwright works, the user is wrong" is the
  exact kind of conclusion to distrust.
- **Console was silent.** No JS error, no network failure, no rejected
  promise. The freeze was purely a layout bug, not a runtime bug.
- **The backdrop overlay gave the user enough visual feedback** ("slightly
  darker") to convince me the dialog *had* opened — I just assumed the card
  was visible somewhere.

**How the bug was finally caught:** Scripted a Playwright DOM dump of every
element with `z-index ≥ 20`, including its `getBoundingClientRect`. The
versions sheet returned `51, -1477 / 640×3259` — absurd numbers for a
"fullscreen" sheet. Measuring the confirm dialog box the same way gave
`y = 1529` against `viewport.h = 417` and the problem was obvious.

**Rules to carry forward:**

1. **Don't trust Playwright-passing as proof the UI is reachable.** Playwright
   auto-scrolls. Humans do not. If a user reports a click that silently fails
   and automation says it works, the first thing to check is whether the
   target element is actually within the viewport at the user's scroll
   position — not whether the handler fires.
2. **Any full-screen sheet or modal must use `position: fixed`**, not
   `position: absolute`. `absolute` leaks into whatever the nearest positioned
   ancestor is, and if that ancestor grows (long quotes, infinite scroll,
   data-driven content), the sheet grows with it and its inner centered
   elements drift off-screen. This is non-obvious when the parent is
   viewport-sized in dev and only expands in production with real data.
3. **Dump bounding rects, not just "did it render."** `querySelector` finding
   an element proves it exists in the DOM. `getBoundingClientRect` against
   `window.innerHeight/innerWidth` proves the user can actually see and click
   it. These are different questions.
4. **When the symptom persists across refreshes, the stale-chunk theory is
   almost certainly wrong.** A refresh clears a stale dynamic-import cache
   immediately. If refresh doesn't fix it, the bug lives in the code being
   served *now*, not in the code the tab was holding.