← back to Homesonspec
TK-11364: version + golden-lock the TK-11125 guard's canonicalization contract
31581ff8316cd87b2f5ff61e99fe801864625ed0 · 2026-09-10 12:55:36 -0700 · Steve
Document and version the spurious-generation guard's two canonicalization
primitives (codex follow-up). canonicalKey is stored + used as the guard's
lookup key, so a rule change re-mints one generation per home (drift-SENSITIVE);
payloadHash is recomputed on both sides at compare time, so a rule change is
self-consistent and mints nothing (drift-IMMUNE). Adds CANONICAL_KEY_VERSION /
PAYLOAD_HASH_VERSION, a golden-vector contract test that breaks the build on any
silent rule change (forcing a conscious version bump), CANONICALIZATION-CONTRACT.md,
and a guard-comment cross-reference. Code/docs only — no schema, no data write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NdRDCrJcC2BcsYP7k3CxyA
Files touched
M apps/workers/src/pipeline.tsA packages/shared/CANONICALIZATION-CONTRACT.mdM packages/shared/src/canonical.tsA packages/shared/src/canonicalization-contract.test.tsM packages/shared/src/payload-hash.ts
Diff
commit 31581ff8316cd87b2f5ff61e99fe801864625ed0
Author: Steve <steve@designerwallcoverings.com>
Date: Thu Sep 10 12:55:36 2026 -0700
TK-11364: version + golden-lock the TK-11125 guard's canonicalization contract
Document and version the spurious-generation guard's two canonicalization
primitives (codex follow-up). canonicalKey is stored + used as the guard's
lookup key, so a rule change re-mints one generation per home (drift-SENSITIVE);
payloadHash is recomputed on both sides at compare time, so a rule change is
self-consistent and mints nothing (drift-IMMUNE). Adds CANONICAL_KEY_VERSION /
PAYLOAD_HASH_VERSION, a golden-vector contract test that breaks the build on any
silent rule change (forcing a conscious version bump), CANONICALIZATION-CONTRACT.md,
and a guard-comment cross-reference. Code/docs only — no schema, no data write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NdRDCrJcC2BcsYP7k3CxyA
---
apps/workers/src/pipeline.ts | 6 ++
packages/shared/CANONICALIZATION-CONTRACT.md | 98 ++++++++++++++++++++++
packages/shared/src/canonical.ts | 24 ++++++
.../shared/src/canonicalization-contract.test.ts | 97 +++++++++++++++++++++
packages/shared/src/payload-hash.ts | 20 +++++
5 files changed, 245 insertions(+)
diff --git a/apps/workers/src/pipeline.ts b/apps/workers/src/pipeline.ts
index a60248b4..cccbaea6 100644
--- a/apps/workers/src/pipeline.ts
+++ b/apps/workers/src/pipeline.ts
@@ -130,6 +130,12 @@ export async function extractStage(
// no schema column, no migration; history is untouched. Any real change
// (or a same-snapshot re-extract that adds a field) differs and falls
// through below, so the guard can only ever SKIP an exact duplicate.
+ //
+ // The canonicalization CONTRACT for `canonicalKey` (stored, drift-sensitive)
+ // and `payloadHash` (recomputed, drift-immune) — and the version-bump
+ // procedure for changing either — lives in
+ // packages/shared/CANONICALIZATION-CONTRACT.md, pinned by
+ // packages/shared/src/canonicalization-contract.test.ts.
const newPayloadHash = payloadHash(normalized.fields);
const latest = await prisma.stagedRecord.findFirst({
where: { sourceId: source.id, canonicalKey: key }, // uses @@index([canonicalKey])
diff --git a/packages/shared/CANONICALIZATION-CONTRACT.md b/packages/shared/CANONICALIZATION-CONTRACT.md
new file mode 100644
index 00000000..6eda2e47
--- /dev/null
+++ b/packages/shared/CANONICALIZATION-CONTRACT.md
@@ -0,0 +1,98 @@
+# Canonicalization contract — TK-11125 spurious-generation guard
+
+The TK-11125 guard (`apps/workers/src/pipeline.ts`, `extractStage`) prevents a
+new `RawSnapshot` whose only change is volatile page bytes (ad tokens, CSRF
+nonces, render timestamps) from minting a brand-new `StagedRecord` generation +
+a full fresh `SourceEvidence` set. It does that with two canonicalization
+primitives in `packages/shared`. This document is their **contract** and the
+procedure for changing them safely.
+
+Steve's ruling stands over all of this: **property history over time IS the
+product — no generation is ever deleted.** The guard only ever *skips writing a
+spurious duplicate*; it never suppresses or removes a genuine change.
+
+## The two primitives
+
+| Primitive | File | Role | Stored? | Drift behavior |
+|---|---|---|---|---|
+| `canonicalKey(hints)` | `src/canonical.ts` | **Identity.** Collapses the same physical home across national/division/community/feed surfaces to one record. | **Yes** — `StagedRecord.canonicalKey`, and it is the guard's **lookup key**. | **Drift-SENSITIVE** — see below. |
+| `payloadHash(fields)` | `src/payload-hash.ts` | **Change-detection.** Canonical hash of the extracted field payload; equal hash ⇒ byte-identical property data ⇒ spurious. | **No** — recomputed on both sides at compare time. | **Drift-IMMUNE** — see below. |
+
+## The guard, in code
+
+```ts
+const newPayloadHash = payloadHash(normalized.fields);
+const latest = await prisma.stagedRecord.findFirst({
+ where: { sourceId: source.id, canonicalKey: key }, // key = canonicalKey(hints)
+ orderBy: { createdAt: "desc" },
+ select: { snapshotId: true, payload: true },
+});
+if (latest && latest.snapshotId !== snapshotId &&
+ payloadHash(latest.payload) === newPayloadHash) {
+ continue; // spurious: no new StagedRecord, no new evidence
+}
+```
+
+Both `payloadHash(...)` calls run in the **same process with the same deployed
+code**. `latest.payload` is the prior generation's `normalized.fields` read
+verbatim from JSONB.
+
+## Why `payloadHash` is drift-IMMUNE
+
+Because the hash is **never stored** — both the incoming record and the prior
+generation are re-hashed at compare time — a change to `stableStringify` or
+`payloadHash` re-hashes *both sides identically*. The comparison stays
+self-consistent, so **changing the serialization rules cannot cause a spurious
+generation to mint.** `PAYLOAD_HASH_VERSION` is therefore documentation/audit
+only. (Still bump it on any rule change, so a shift in the golden hashes is
+traceable to an intentional edit.)
+
+## Why `canonicalKey` is drift-SENSITIVE
+
+`canonicalKey` **is** stored, and the guard looks prior generations up by it
+(`where: { canonicalKey }`). Old rows carry keys minted under the *old* rules.
+So if you change any `canonicalKey` rule:
+
+- the key a home hashes to changes;
+- the guard's lookup no longer matches that home's existing rows;
+- the guard finds no "latest" and falls through;
+- every affected home mints **one** fresh generation on its next extract.
+
+That is a **one-time, catalog-wide generation re-mint** — a benign but visible
+bump in generation counts, **not a regression**. Without a version marker a
+future operator watching generation counts jump could mistake it for the guard
+breaking. That is exactly the failure this contract exists to prevent.
+
+The `canonicalKey` rules that are part of identity (changing any of them
+triggers the re-mint):
+
+- `norm()` — lowercase, collapse whitespace, trim.
+- `normalizeAddress()` — lowercase, strip `# , .`, collapse whitespace, and the
+ `STREET_ABBREVIATIONS` table.
+- `latlon5()` — 5-decimal (`toFixed(5)`, ~1.1 m) rounding of lat/lon.
+- The field list **and its order** in `canonicalKey()` (`builderSlug`,
+ `communityName`, `address`, `lotNumber`, `builderInventoryId`, `lat`, `lon`,
+ `planName`) joined with `|`.
+- The hash algorithm (`sha256`, hex).
+
+## Procedure — changing a canonicalization rule
+
+1. Make the rule change in `src/canonical.ts` and/or `src/payload-hash.ts`.
+2. **Bump the matching version constant** — `CANONICAL_KEY_VERSION` and/or
+ `PAYLOAD_HASH_VERSION`.
+3. Run `pnpm --filter @homesonspec/shared test`. The golden test
+ (`canonicalization-contract.test.ts`) will fail on the old pinned hashes —
+ update the golden hex and the version assertions to the new values.
+4. For a **`canonicalKey`** change, state the expected one-time re-mint in the
+ PR/commit (roughly: one new generation per active home on next extract) so
+ nobody mistakes the generation-count bump for a guard regression.
+5. Ship `canonicalKey`/schema-affecting changes the same way as other prod DDL
+ for this repo — as a reviewed, gated migration. This contract change itself
+ is code-only (no schema, no data write).
+
+## Enforcement
+
+`canonicalization-contract.test.ts` pins exact sha256 vectors for both
+primitives, plus the lat/lon rounding, the address-abbreviation output, and the
+`stableStringify` form. Any silent change to a rule breaks the build; the only
+way past it is a conscious golden + version update — which is the point.
diff --git a/packages/shared/src/canonical.ts b/packages/shared/src/canonical.ts
index 8df7d413..b4007e4a 100644
--- a/packages/shared/src/canonical.ts
+++ b/packages/shared/src/canonical.ts
@@ -1,5 +1,29 @@
import { createHash } from "node:crypto";
+/**
+ * CANONICALIZATION CONTRACT — `canonicalKey` (identity)
+ * See packages/shared/CANONICALIZATION-CONTRACT.md for the full contract.
+ *
+ * `canonicalKey` is a STORED value (`StagedRecord.canonicalKey`) that is also
+ * used as the LOOKUP KEY by the TK-11125 spurious-generation guard
+ * (apps/workers/src/pipeline.ts). That makes its normalization rules
+ * discontinuity-sensitive in a way `payloadHash` is NOT:
+ *
+ * Changing ANY rule below (the normalize helpers, the abbreviation table,
+ * the field list/order, or the lat/lon rounding) changes the key a home
+ * hashes to. Because old rows carry keys minted under the OLD rules, the
+ * guard's `where: { canonicalKey }` lookup no longer matches them, so every
+ * affected home looks brand-new and mints ONE fresh generation on its next
+ * extract — a one-time catalog-wide re-mint (a benign but visible bump in
+ * generation counts), NOT a regression.
+ *
+ * Therefore: if you change a rule here, BUMP `CANONICAL_KEY_VERSION`. The
+ * golden test (canonicalization-contract.test.ts) fails until the version and
+ * the pinned hashes are updated together, forcing the change — and its re-mint
+ * consequence — to be conscious rather than silent.
+ */
+export const CANONICAL_KEY_VERSION = 1;
+
/**
* Canonical inventory identity. The same physical home may surface on a
* builder's national site, a division site, a community page, or a feed —
diff --git a/packages/shared/src/canonicalization-contract.test.ts b/packages/shared/src/canonicalization-contract.test.ts
new file mode 100644
index 00000000..a1a7040d
--- /dev/null
+++ b/packages/shared/src/canonicalization-contract.test.ts
@@ -0,0 +1,97 @@
+import { describe, expect, it } from "vitest";
+import {
+ CANONICAL_KEY_VERSION,
+ canonicalKey,
+ normalizeAddress,
+} from "./canonical";
+import {
+ PAYLOAD_HASH_VERSION,
+ payloadHash,
+ stableStringify,
+} from "./payload-hash";
+
+/**
+ * GOLDEN CONTRACT TEST for the TK-11125 spurious-generation guard's
+ * canonicalization. It pins the EXACT sha256 output for representative inputs.
+ *
+ * Why this test exists: the guard's identity key (`canonicalKey`) is stored and
+ * used as a lookup key, so any silent change to its normalization rules causes a
+ * one-time catalog-wide generation re-mint (see canonical.ts contract header).
+ * These golden vectors turn that silent change into a build failure — you cannot
+ * alter a normalization rule without also updating the pinned hash here, which
+ * forces you to bump the matching *_VERSION and acknowledge the consequence.
+ *
+ * When you INTENTIONALLY change a rule:
+ * 1. bump CANONICAL_KEY_VERSION and/or PAYLOAD_HASH_VERSION in the source,
+ * 2. update the golden hex below (and the version assertions),
+ * 3. for a CANONICAL_KEY change, note the expected one-time re-mint in the PR.
+ * See packages/shared/CANONICALIZATION-CONTRACT.md.
+ */
+
+// The canonical identity fixture (same shape as canonical.test.ts `base`).
+const IDENTITY_FIXTURE = {
+ builderSlug: "meridian-homes",
+ communityName: "Cedar Bend",
+ address: "123 Oak Street",
+ lotNumber: "42",
+ builderInventoryId: "MH-1001",
+ lat: 30.312345,
+ lon: -97.712345,
+ planName: "The Juniper",
+};
+
+// A FieldValue-shaped envelope, the real StagedRecord.payload shape.
+const fv = (value: unknown, raw: string | null = null) => ({
+ value,
+ raw,
+ evidenceText: raw,
+ sourceUrl: "fixture://x",
+ confidence: value === null ? 0 : 1,
+});
+const PAYLOAD_FIXTURE = { price: fv(500000), beds: fv(4), city: fv("Leander") };
+
+describe("canonicalization contract (versioned golden vectors)", () => {
+ it("canonicalKey golden vector is stable for CANONICAL_KEY_VERSION=1", () => {
+ expect(CANONICAL_KEY_VERSION).toBe(1);
+ expect(canonicalKey(IDENTITY_FIXTURE)).toBe(
+ "dad5b977ff94a3cbbe99ec37e59a43e4c5b0f91bb9abeb0101d6f8ab5015a1d3",
+ );
+ });
+
+ it("payloadHash golden vector is stable for PAYLOAD_HASH_VERSION=1", () => {
+ expect(PAYLOAD_HASH_VERSION).toBe(1);
+ expect(payloadHash(PAYLOAD_FIXTURE)).toBe(
+ "a1a3654ff4992a5170673f33ac4ff25f2927fae3990a5b6daec6e02e9f45021f",
+ );
+ });
+
+ it("lat/lon rounding is pinned (part of the identity contract)", () => {
+ // 5-decimal rounding is a canonicalKey rule; pin the exact toFixed output so
+ // a rounding change (which would re-fork GPS-jittered identities) trips here.
+ expect(IDENTITY_FIXTURE.lat.toFixed(5)).toBe("30.31235");
+ expect(IDENTITY_FIXTURE.lon.toFixed(5)).toBe("-97.71234");
+ });
+
+ it("normalizeAddress rule set is pinned", () => {
+ // The abbreviation table + punctuation stripping is part of identity.
+ expect(normalizeAddress("456 N. Mesquite Boulevard, #2")).toBe(
+ "456 n mesquite blvd 2",
+ );
+ });
+
+ it("stableStringify output form is pinned (sorted keys, dropped undefined)", () => {
+ expect(stableStringify({ b: { z: 1, a: 2 }, a: undefined, c: null })).toBe(
+ '{"b":{"a":2,"z":1},"c":null}',
+ );
+ });
+
+ it("guard invariant: canonical hashes are order-independent but change-sensitive", () => {
+ // Reordered-but-equal → same hash (the legitimate SKIP case).
+ const reordered = { city: fv("Leander"), beds: fv(4), price: fv(500000) };
+ expect(payloadHash(reordered)).toBe(payloadHash(PAYLOAD_FIXTURE));
+ // Any genuine field change → different hash (guard can only ever skip a
+ // true duplicate, never suppress a real change).
+ const priceDrop = { ...PAYLOAD_FIXTURE, price: fv(450000) };
+ expect(payloadHash(priceDrop)).not.toBe(payloadHash(PAYLOAD_FIXTURE));
+ });
+});
diff --git a/packages/shared/src/payload-hash.ts b/packages/shared/src/payload-hash.ts
index 663cbbcf..e49fc2d5 100644
--- a/packages/shared/src/payload-hash.ts
+++ b/packages/shared/src/payload-hash.ts
@@ -1,5 +1,25 @@
import { createHash } from "node:crypto";
+/**
+ * CANONICALIZATION CONTRACT — `payloadHash` / `stableStringify` (change-detection)
+ * See packages/shared/CANONICALIZATION-CONTRACT.md for the full contract.
+ *
+ * Unlike `canonicalKey`, this hash is NEVER stored. The TK-11125 guard
+ * recomputes BOTH sides at compare time — `payloadHash(normalized.fields)` for
+ * the incoming record and `payloadHash(latest.payload)` for the prior
+ * generation read back from `StagedRecord.payload` — with the same deployed
+ * code, in the same process. So a change to the serialization rules re-hashes
+ * both sides identically: the comparison stays self-consistent and a rule
+ * change does NOT cause a spurious generation to mint. `payloadHash` is
+ * drift-IMMUNE where `canonicalKey` is drift-sensitive.
+ *
+ * `PAYLOAD_HASH_VERSION` is therefore documentation/audit only — it does not
+ * gate any re-mint — but BUMP it on any rule change anyway, so a shift in the
+ * golden hashes is traceable to an intentional edit and the golden test
+ * (canonicalization-contract.test.ts) stays honest.
+ */
+export const PAYLOAD_HASH_VERSION = 1;
+
/**
* Canonical, key-sorted JSON serialization. Two structurally-equal values
* always produce the same string regardless of key insertion order — which is
← 3e4d499f O3b: ValidationEvent partition conversion script (corrected
·
back to Homesonspec
·
TK-11303: retire stale v1 apple-review launchd plist (supers 0200179c →