add refine-loop.sh — overnight loop that records every article into the archive + rebuilds, 90-min spacing, local-only (Steve: refine all night) (TK-10708)

37212766c31dea28ab61a6e9e9258183df96ce92 · 2026-08-20 00:08:00 -0700 · Steve Abrams

Files touched

A scripts/refine-loop.sh

Diff

commit 37212766c31dea28ab61a6e9e9258183df96ce92
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Thu Aug 20 00:08:00 2026 -0700

    add refine-loop.sh — overnight loop that records every article into the archive + rebuilds, 90-min spacing, local-only (Steve: refine all night) (TK-10708)
---
 scripts/refine-loop.sh | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/scripts/refine-loop.sh b/scripts/refine-loop.sh
new file mode 100755
index 0000000..45a6e68
--- /dev/null
+++ b/scripts/refine-loop.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+# TK-10708  "Refine all night long" (Steve). Each cycle: pull every feed -> record EVERY article into the
+# growing archive + deals into the store -> pull CA public-record deeds -> rebuild the flyers site -> commit.
+# LOCAL only (no auto-deploy — that stays Steve-gated). 90-min spacing so Google News rate-limits recover.
+export PATH=/opt/homebrew/bin:/opt/homebrew/opt/postgresql@14/bin:/usr/bin:/bin:/usr/sbin:/sbin
+cd "$HOME/Projects/re-flyer-aggregator" || exit 1
+LOG=data/refine-loop.log
+echo "=== refine-loop START $(date) ===" >> "$LOG"
+n=0
+while true; do
+  n=$((n+1))
+  echo "--- cycle $n $(date) ---" >> "$LOG"
+  node scripts/cre-news-monitor.mjs   >> "$LOG" 2>&1   # records every article -> article-archive.json (no cap)
+  node scripts/ingest-public-deals.mjs >> "$LOG" 2>&1  # CA deeds/DTT
+  node scripts/build-flyers-site.mjs  >> "$LOG" 2>&1   # rebuild flyers.json + index
+  git add data/ public/ scripts/ >/dev/null 2>&1
+  git commit -q -m "refine-loop cycle $n: +articles/deals $(date '+%m-%d %H:%M')" >/dev/null 2>&1
+  arch=$(node -e 'try{console.log(require("./data/article-archive.json").length)}catch(e){console.log(0)}' 2>/dev/null)
+  store=$(node -e 'try{console.log(require("./data/deal-store.json").length)}catch(e){console.log(0)}' 2>/dev/null)
+  echo "  cycle $n done $(date) — archive=$arch, store=$store" >> "$LOG"
+  sleep 5400   # 90 min — spacing to dodge Google News rate-limits
+done

← 6c18e62 news: record EVERY article to a growing article-archive.json · back to Re Flyer Aggregator · add global fast-search page (/search) — chat-style instant s 2cda3b3 →