[object Object]

← back to Nationalrealestate

usre: coverage-gate the full-rescan marker — downgrade to 'partial' when a 'full' adapter covers <50% of known listings, so a truncated rescan can't fabricate withdrawals (Cody Cycle-2 gate) (TK-10155)

3709d7db9f451cdf61e08153d3baa67284a609b6 · 2026-08-03 07:01:25 -0700 · steve

Files touched

Diff

commit 3709d7db9f451cdf61e08153d3baa67284a609b6
Author: steve <steve@designerwallcoverings.com>
Date:   Mon Aug 3 07:01:25 2026 -0700

    usre: coverage-gate the full-rescan marker — downgrade to 'partial' when a 'full' adapter covers <50% of known listings, so a truncated rescan can't fabricate withdrawals (Cody Cycle-2 gate) (TK-10155)
---
 db/migrations/016_source_ingest_run_partial.sql | 10 ++++++++++
 src/ingest/listings/engine.ts                   | 24 ++++++++++++++++++++++--
 2 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/db/migrations/016_source_ingest_run_partial.sql b/db/migrations/016_source_ingest_run_partial.sql
new file mode 100644
index 0000000..db5bea1
--- /dev/null
+++ b/db/migrations/016_source_ingest_run_partial.sql
@@ -0,0 +1,10 @@
+-- TK-10155 Cycle-2 (Cody gate): allow a third ingest mode 'partial'.
+-- A 'full' adapter that only covered a fraction of its known listings (flaky/truncated rescan)
+-- is downgraded by the ingest engine to 'partial' so it does NOT arm gone-from-feed withdrawal
+-- (the detector only treats mode='full' as establishing absence). Prevents fabricated withdrawals.
+-- Idempotent: safe to re-run.
+
+ALTER TABLE source_ingest_run DROP CONSTRAINT IF EXISTS source_ingest_run_mode_check;
+ALTER TABLE source_ingest_run
+  ADD CONSTRAINT source_ingest_run_mode_check
+  CHECK (mode IN ('full','incremental','partial'));
diff --git a/src/ingest/listings/engine.ts b/src/ingest/listings/engine.ts
index d1ab256..2562718 100644
--- a/src/ingest/listings/engine.ts
+++ b/src/ingest/listings/engine.ts
@@ -155,18 +155,38 @@ async function runAdapter(a: ListingAdapter): Promise<{ upserted: number; skippe
     // TK-10155 fix 1: record this run's INGEST MODE so the lifecycle detector knows whether the
     // source was FULLY re-scanned. A 'full' run here is what lets classifyState later treat a
     // gone-from-feed listing as withdrawn; an 'incremental' run does NOT establish absence.
+    // TK-10155 Cycle-2 fix (Cody gate): a 'full' adapter that only covered a FRACTION of its known
+    // listings (flaky/truncated rescan, e.g. RealtyTexas 3/46) must NOT stamp a real 'full' marker —
+    // that would falsely arm gone-from-feed withdrawal for the un-refetched listings and fabricate
+    // withdrawals ~WITHDRAWN_DAYS later. Downgrade to 'partial' (which does NOT arm withdrawal)
+    // unless coverage >= FULL_RESCAN_MIN_COVERAGE of the source's known listings.
     // Non-fatal — a ledger failure must never fail the ingest.
+    const FULL_RESCAN_MIN_COVERAGE = 0.5;
+    let recordedMode: string = a.mode;
     try {
+      if (a.mode === 'full') {
+        const { rows } = await query<{ n: string }>(
+          `SELECT count(*)::text AS n FROM listing WHERE source=$1`,
+          [a.source],
+        );
+        const known = Number(rows?.[0]?.n ?? 0);
+        if (known > 0 && upserted < known * FULL_RESCAN_MIN_COVERAGE) {
+          recordedMode = 'partial';
+          console.warn(
+            `[${a.source}] full rescan covered only ${upserted}/${known} (<${FULL_RESCAN_MIN_COVERAGE * 100}%) — recording mode='partial' (withdrawal detection stays disarmed for this source this run)`,
+          );
+        }
+      }
       await query(
         `INSERT INTO source_ingest_run (source, mode, finished_at, listing_count)
          VALUES ($1,$2, now(), $3)`,
-        [a.source, a.mode, upserted],
+        [a.source, recordedMode, upserted],
       );
     } catch (e: any) {
       console.warn(`[${a.source}] source_ingest_run record skipped: ${e?.message || e}`);
     }
 
-    console.log(`[${a.source}] DONE upserted=${upserted} skipped=${skipped} firm_id=${firmId} mode=${a.mode}`);
+    console.log(`[${a.source}] DONE upserted=${upserted} skipped=${skipped} firm_id=${firmId} mode=${recordedMode}`);
     return { upserted, skipped };
   } catch (e: any) {
     await closeRun(runId, 'failed', { notes: String(e.message || e) });

← 37575d8 TK-10155 fix4: backfill writes ONE snapshot per listing (bas  ·  back to Nationalrealestate  ·  usre: loud+identifiable log on first-see lifecycle failure ( aeba005 →