Loop: reserve >=1 slot/tick for fast-consumer streams (homes/businesses/firm) so parcel backfills stop starving them (USRE_CONSUMER_SLOTS, default 1)

908d6036e5ea26ac9474e97f1b9d86a22f09f1f7 · 2026-07-30 18:34:43 -0700 · steve@designerwallcoverings.com

Files touched

M src/jobs/hourly_loop.ts

Diff

commit 908d6036e5ea26ac9474e97f1b9d86a22f09f1f7
Author: steve@designerwallcoverings.com <steve@designerwallcoverings.com>
Date:   Thu Jul 30 18:34:43 2026 -0700

    Loop: reserve >=1 slot/tick for fast-consumer streams (homes/businesses/firm) so parcel backfills stop starving them (USRE_CONSUMER_SLOTS, default 1)
---
 src/jobs/hourly_loop.ts | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/src/jobs/hourly_loop.ts b/src/jobs/hourly_loop.ts
index 2616f16..a37701b 100644
--- a/src/jobs/hourly_loop.ts
+++ b/src/jobs/hourly_loop.ts
@@ -25,6 +25,12 @@ const __dirname = dirname(fileURLToPath(import.meta.url));
 const ROOT = join(__dirname, '..', '..');
 const DRY = process.argv.includes('--dry');
 const MAX_JOBS_PER_TICK = Number(process.env.USRE_MAX_JOBS_PER_TICK || 2);
+// Fast-consumer streams (homes + businesses + firm enrichment) that the huge
+// paginating parcel/deed backfills would otherwise starve out of every slot.
+// Reserve >=1 slot/tick for the stalest eligible consumer so they get a
+// guaranteed cadence. Env-tunable (0 = old global-stalest behavior).
+const CONSUMER = new Set(['listings', 'places-seed', 'firm-crawl', 'discover']);
+const CONSUMER_SLOTS = Number(process.env.USRE_CONSUMER_SLOTS || 1);
 
 type Job = {
   name: string;
@@ -99,7 +105,20 @@ async function main() {
     console.log(`   ${mark} ${s.job.name.padEnd(12)} [${s.health.state.padEnd(11)}] age=${a.padStart(6)}  ${s.health.reason}`);
   }
 
-  const pick = plan.filter(s => s.eligible).slice(0, MAX_JOBS_PER_TICK);
+  // Pool-aware pick: reserve up to CONSUMER_SLOTS for the stalest eligible
+  // fast-consumer jobs (homes/businesses/firm) so paginating parcel/deed backfills
+  // can't monopolize every slot; fill the rest with the global stalest. `plan` is
+  // already sorted stalest/most-boosted first, so first-match = stalest.
+  const eligible = plan.filter(s => s.eligible);
+  const pick: typeof eligible = [];
+  for (const s of eligible) {
+    if (pick.length >= CONSUMER_SLOTS) break;
+    if (CONSUMER.has(s.job.name)) pick.push(s);
+  }
+  for (const s of eligible) {
+    if (pick.length >= MAX_JOBS_PER_TICK) break;
+    if (!pick.includes(s)) pick.push(s);
+  }
   if (!pick.length) { console.log('[loop] nothing eligible — idle tick (all fresh, blocked, or quarantined)'); await pool.end(); return; }
 
   await pool.end(); // children own their own pools

← 82774fe Make listings CAP env-tunable (USRE_LISTINGS_CAP, default 10 · back to Nationalrealestate · discover: fail-soft run recording (try/catch closeRun) so se 0321546 →