RENTV: incremental corpus-crawl wrapper (LOCK-aware, cursor-reset-only, $0 local) — completes crawl→sync→serve chain, TK-10254 follow-up

45863c90288427492fc44219915fa270c997535c · 2026-08-05 14:17:54 -0700 · Steve Abrams

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

A scripts/crawl-corpus-incremental.sh

Diff

commit 45863c90288427492fc44219915fa270c997535c
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Aug 5 14:17:54 2026 -0700

    RENTV: incremental corpus-crawl wrapper (LOCK-aware, cursor-reset-only, $0 local) — completes crawl→sync→serve chain, TK-10254 follow-up
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/crawl-corpus-incremental.sh | 53 +++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/scripts/crawl-corpus-incremental.sh b/scripts/crawl-corpus-incremental.sh
new file mode 100755
index 00000000..7e3638ad
--- /dev/null
+++ b/scripts/crawl-corpus-incremental.sh
@@ -0,0 +1,53 @@
+#!/bin/zsh
+# ─────────────────────────────────────────────────────────────────────────────
+# crawl-corpus-incremental.sh  (TK-10254 follow-up)
+#
+# Catches NEWLY-published rentv.com articles into the article corpus so /closings
+# keeps enriching new closings with buyer/seller/broker. Runs the local-model
+# extractor (qwen3:14b via Ollama) — $0, no paid API.
+#
+# WHY A WRAPPER: pull-archive.mjs uses a ONE-PASS descending cursor
+# (nextId = min(prev.cursor, MAX)), so once the deep backfill has descended,
+# a plain re-run won't re-scan the NEW high ids. To catch them we reset ONLY the
+# cursor (rm archive-crawl-state.json) and scan a fresh top window. We do NOT use
+# --fresh (that would blank the whole corpus); already-ingested ids are skipped
+# off the corpus file itself, so nothing is re-extracted.
+#
+# SAFE BY DESIGN:
+#  - LOCK-aware: bails if a crawl (e.g. the deep backfill) holds a <6h lock, so we
+#    never disturb its cursor mid-run.
+#  - Corpus is append-only + skip-seen; the reset touches only the cursor file.
+#  - Window is floored at 20000 (below that is the completed historical backfill).
+#
+# Drafted for Steve's approval — the launchd install is gated.
+# ─────────────────────────────────────────────────────────────────────────────
+set -euo pipefail
+cd "$HOME/Projects/rentv"
+LOG="logs/corpus-incremental.log"
+LOCKF="data/.archive-crawl.lock"
+mkdir -p logs
+stamp() { date -u +%FT%TZ }
+
+# 1) don't interfere with an active crawl (backfill holds a <6h lock)
+if [ -f "$LOCKF" ]; then
+  AGE=$(node -e 'try{const fs=require("fs");const l=JSON.parse(fs.readFileSync("'"$LOCKF"'","utf8"));process.stdout.write(String(Date.now()-(l.at||0)))}catch{process.stdout.write("999999999")}' 2>/dev/null || echo 999999999)
+  if [ "$AGE" -lt 21600000 ]; then echo "$(stamp) SKIP: crawl lock active (age ${AGE}ms)" >> "$LOG"; exit 0; fi
+fi
+
+# 2) newest known article id (from the 20-min news pull + the current corpus)
+NEWEST=$(node -e 'const fs=require("fs");let m=0;try{const a=JSON.parse(fs.readFileSync("./data/news-archive.json","utf8"));const it=Array.isArray(a)?a:(a.items||a.articles||[]);for(const x of it){const n=+x.id;if(n>m)m=n}}catch{}try{for(const l of fs.readFileSync("./data/articles-corpus.jsonl","utf8").split("\n")){if(!l.trim())continue;try{const n=+JSON.parse(l).id;if(n>m)m=n}catch{}}}catch{}process.stdout.write(String(m))' 2>/dev/null || echo 0)
+[ "${NEWEST:-0}" -gt 0 ] || { echo "$(stamp) SKIP: could not determine newest id" >> "$LOG"; exit 0; }
+
+CEIL=$((NEWEST + 30))
+FLOOR=$((NEWEST - 250)); [ "$FLOOR" -lt 20000 ] && FLOOR=20000
+
+# 3) reset ONLY the cursor (never the corpus) → re-scan the fresh top window
+rm -f data/archive-crawl-state.json
+
+echo "$(stamp) RUN: incremental window ${CEIL}->${FLOOR}" >> "$LOG"
+if node scripts/pull-archive.mjs --max "$CEIL" --min "$FLOOR" --delay 90 >> "$LOG" 2>&1; then
+  echo "$(stamp) DONE: window ${CEIL}->${FLOOR}, corpus $(wc -l < data/articles-corpus.jsonl | tr -d ' ') lines" >> "$LOG"
+else
+  echo "$(stamp) FAIL: pull-archive error (Ollama down? check log above)" >> "$LOG"
+  exit 1
+fi

← f1e911ce Cody-gate fixes: by_metro bar routes to organizations (org-l · back to Rentv · PR intel: LinkedIn enrichment via Brave search (local, key s 1edf0c45 →