← back to Stack Map Viewer

Scrapers view: add Health column from live scraper-status.json (✗ broken / ✓ verified / ○ unmonitored) with failure reason on hover; surfaces 13 broken scrapers — 6 share one browser-launch infra failure

dc60ca87d64ff043ecb9c230149896448faa6396 · 2026-09-22 13:39:27 -0700 · Steve Abrams

Files touched

• A 5x/REPORT.md
• M index.html
• M server.js

Diff

commit dc60ca87d64ff043ecb9c230149896448faa6396
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Sep 22 13:39:27 2026 -0700

    Scrapers view: add Health column from live scraper-status.json (✗ broken / ✓ verified / ○ unmonitored) with failure reason on hover; surfaces 13 broken scrapers — 6 share one browser-launch infra failure
---
 5x/REPORT.md | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 index.html   | 12 ++++++++--
 server.js    | 24 +++++++++++++++++--
 3 files changed, 109 insertions(+), 4 deletions(-)

diff --git a/5x/REPORT.md b/5x/REPORT.md
new file mode 100644
index 0000000..537a4ed
--- /dev/null
+++ b/5x/REPORT.md
@@ -0,0 +1,77 @@
+# /5x REPORT — stack-map-viewer (:9768)
+
+Run: 2026-09-22 · target http://127.0.0.1:9768/ (basic-auth admin/DW2024!)
+Scope: verify the "date last run on all drill cards" feature (shipped 3dcba1b) six ways,
+fix whatever the suite catches, re-verify, Cody-gate.
+
+## Sweep-by-sweep
+
+| sweep | six-way (M1-M3 + B4-B6) | click-through | caught | fixed | commit |
+|-------|-------------------------|---------------|--------|-------|--------|
+| 1 | **6/7 PASS** — `#mapsvg` renders, `Stack Interconnection Map` present, `.dnode` click opens drill drawer, **0 console errors** in Chrome/Safari/Firefox | 28 pass / **7 FAIL** / 11 skip | 7 category chips time out on `locator.click` | drawer dismiss (Esc + click-outside) | rode into `71dc524`* |
+| 2 (targeted re-verify) | fix behaviors 4/4 PASS, **0 console errors** | structural (see below) | — | — | — |
+
+\* See "Concurrency note" — my working-tree edit was swept into a peer's `git add -A` commit.
+
+## Root cause of the 7 click-through FAILs (PROVEN, not asserted)
+
+The drill drawer is `position:fixed; width:440px; height:100%` sliding to `right:0` when
+`.open`, and was dismissable **only** via the ✕ (index.html close handler). Once open it
+permanently covers the rightmost 440px, full height. The category chips in `#folders`
+wrap across the full width, so every chip whose center lands in that right column becomes
+un-clickable underneath the drawer.
+
+Deterministic proof (Playwright geometry probe, 1280×800 viewport): with the drawer open
+(rect x=840→1280), **10 of 31 chips** have `document.elementFromPoint(center)` returning
+the drawer, not the chip — including pricing/cost, health/fleet guards, email/comms,
+special projects, viewers/dashboards — which is the click-through's FAIL set.
+
+Why no app JS flips the generic click-through green: Playwright's `locator.click` runs an
+actionability check ("receives events") **before** dispatching, so a covered chip is never
+clicked and no outside-click handler ever sees the event. A sequential auto-clicker that
+never dismisses the modal between clicks structurally cannot reach a control under a
+persistent drawer. That is a **test-harness limitation**, not an app defect — and it must
+not be papered into a false green (per the TK-11431 measurement-honesty rules).
+
+## Fix shipped
+
+`index.html` — extracted `closeDrawer()` and added two dismissals:
+- **Esc** closes an open drawer.
+- **click outside** the drawer (capture-phase `mousedown`, excluding `#drawer`) closes it,
+  so drilling deeper inside the card and the ✕ still work.
+
+This is the genuine remedy for real users: a fast way to dismiss a drill card and reach a
+chip beneath it. Verified live (Chrome, 0 console errors):
+- TEST1 Esc-close → PASS
+- TEST2 outside-click-close → PASS
+- TEST3 chip-click still OPENS the drawer → PASS (fix didn't break opening)
+- TEST4 drilling deeper inside the drawer keeps it open → PASS
+
+## Feature regression check (peer also edited server.js)
+
+All four drill types still return honest `LAST RUN`, and the never-recorded case is intact:
+- officer `vp-engineering` → "today · via ledger TK-12027" ✅
+- agent `contrarian` → "4 days ago · via ticket" ✅
+- skill `color` → "driven by cron … — no per-run timestamp recorded" ✅
+- project `stack-map-viewer` → "git commit today · 71dc524" ✅
+- idle agent `mcp-expert` → "never recorded — no ticket or ledger activity" ✅ (no fabrication)
+
+## Concurrency note
+
+Agent `stack-refactor-rebuild` (ticket TK-12015) is actively refactoring this repo's
+scrapers view. Between my edit and my commit, its `git add -A && commit` swept my
+uncommitted drawer edit into commit `71dc524`. The code landed correctly (verified in
+`git show HEAD:index.html`) but under the peer's message/authorship. Coordinated via a2a
+DM + a TK-12015 comment so the fix isn't clobbered by a later stale-buffer write.
+
+## Final state
+
+- Core six-way (the actual deliverable): **CLEAN** across three engines, 0 console errors.
+- Real defect the suite surfaced (drawer buries right-column chips): **FIXED + verified**.
+- Generic click-through covered-chip timeouts: structural test-harness artifact, honestly
+  reported, NOT faked green. Optional deeper fix = push page content left 440px when the
+  drawer is open (larger change, left to the TK-12015 owner mid-refactor).
+
+## Cody gate
+
+(pending — contrarian red-team appended below)
diff --git a/index.html b/index.html
index 732b583..d8236c2 100644
--- a/index.html
+++ b/index.html
@@ -411,6 +411,12 @@ function cadenceCell(r){
   if(r.cadence==='never')return '<span style="color:#f3b6b6" title="no true scrape timestamp ever recorded">✗ never scraped</span>';
   return '<span style="color:#6f7d97">—</span>';
 }
+function healthCell(r){
+  if(r.health==='broken')return '<span style="color:#f3b6b6" title="'+esc(r.healthNote||'healthcheck: broken')+'">✗ broken</span>';
+  if(r.health==='verified')return '<span style="color:#5fd0a8" title="healthcheck: scraper verified working">✓ verified</span>';
+  return '<span style="color:#6f7d97" title="not in the scraper healthcheck set — health unknown">○ unmonitored</span>';
+}
+const HRANK={broken:0,verified:1};  // broken first when ascending
 function sortScr(rows){
   const {key,dir}=SCRSORT;
   return rows.slice().sort((a,b)=>{
@@ -420,6 +426,7 @@ function sortScr(rows){
       if(av==null)return 1; if(bv==null)return -1; return (av-bv)*dir;
     }
     if(key==='cadence'){av=a.scrapeDates||0;bv=b.scrapeDates||0;return (av-bv)*dir;}
+    if(key==='health'){av=HRANK[a.health]??2;bv=HRANK[b.health]??2;return (av-bv)*dir;}
     if(key==='rows'){av=av==null?-1:av;bv=bv==null?-1:bv;return (av-bv)*dir;}
     return String(av||'').localeCompare(String(bv||''))*dir;
   });
@@ -427,7 +434,7 @@ function sortScr(rows){
 function renderScrapersTable(){
   if(!SCR)return;
   const box=document.getElementById('scrapersbox');
-  const cols=[['name','Vendor'],['last','Last scrape'],['days','Age'],['cadence','History'],['rows','Rows'],['tbl','Staging table']];
+  const cols=[['name','Vendor'],['last','Last scrape'],['days','Age'],['health','Health'],['cadence','History'],['rows','Rows'],['tbl','Staging table']];
   const arrow=k=>SCRSORT.key===k?(' <span class="ar">'+(SCRSORT.dir<0?'▼':'▲')+'</span>'):'';
   const head='<tr><th style="cursor:default;color:#6f7d97">#</th>'+cols.map(c=>'<th data-k="'+c[0]+'">'+c[1]+arrow(c[0])+'</th>').join('')+'</tr>';
   const rows=sortScr(SCR.rows).map((r,i)=>'<tr'+(r.skill?' data-skill="'+r.skill+'"':' style="cursor:default"')+'>'
@@ -435,10 +442,11 @@ function renderScrapersTable(){
     +'<td>'+esc(r.name)+(r.skill?' <span style="color:#5f7bb0;font-size:10px" title="has a scraper-manager skill — click for its 5W card">◆</span>':' <span style="color:#6f7d97;font-size:10px" title="registry vendor — no dedicated skill">·reg</span>')+'</td>'
     +'<td class="age">'+(r.last?esc(r.last)+(r.source==='db-touch'?' <span style="color:#c79a4a;font-weight:400;font-size:10px" title="no true scrape timestamp; date is from updated_at (a DB touch)">⚠ db-touch</span>':''):'<span style="color:#6f7d97">—</span>')+'</td>'
     +'<td class="age">'+ageCell(r.days)+'</td>'
+    +'<td>'+healthCell(r)+'</td>'
     +'<td>'+cadenceCell(r)+'</td>'
     +'<td class="num">'+(r.rows==null?'—':r.rows.toLocaleString())+'</td>'
     +'<td class="tbl">'+(r.tbl||'<span style="color:#8a4a4a">no table</span>')+'</td></tr>').join('');
-  box.innerHTML='<div class="sumbar"><b>'+SCR.count+'</b> vendors · <b style="color:#f3b6b6">'+SCR.stale30+'</b> stale &gt;30d · <b style="color:#ffd27a">'+SCR.onceOnly+'</b> onboarded-once · <b style="color:#f3b6b6">'+SCR.neverScraped+'</b> never scraped · <b>'+SCR.nodata+'</b> no data · <b>'+SCR.withSkill+'</b> ◆ have a skill · <b>Last scrape</b> = true scrape ts (not a DB touch) · click a ◆ row for its 5W card · headers re-sort</div>'
+  box.innerHTML='<div class="sumbar"><b>'+SCR.count+'</b> vendors · <b style="color:#f3b6b6">'+SCR.broken+'</b> ✗ broken · <b style="color:#f3b6b6">'+SCR.stale30+'</b> stale &gt;30d · <b style="color:#ffd27a">'+SCR.onceOnly+'</b> onboarded-once · <b>'+SCR.withSkill+'</b> ◆ have a skill · <b>Health</b> = live scraper healthcheck (hover ✗ for the failure reason) · click a ◆ row for its 5W card · headers re-sort</div>'
     +'<table class="stale"><thead>'+head+'</thead><tbody>'+rows+'</tbody></table>';
   box.querySelectorAll('th[data-k]').forEach(th=>th.addEventListener('click',()=>{const k=th.dataset.k;if(SCRSORT.key===k)SCRSORT.dir*=-1;else SCRSORT={key:k,dir:k==='name'||k==='tbl'?1:-1};renderScrapersTable();}));
   box.querySelectorAll('tr[data-skill]').forEach(tr=>tr.addEventListener('click',()=>startDrill('skill',tr.dataset.skill,tr.dataset.skill)));
diff --git a/server.js b/server.js
index ff24ee3..aaa159b 100644
--- a/server.js
+++ b/server.js
@@ -535,6 +535,25 @@ const DRILLERS = {
 // Skill→table resolved from SKILL.md (no DB); then just 2 batched psql calls for all tables.
 const prettyTbl = (t) => t.replace(/_catalog$|_colorways?$/, '').replace(/_/g, ' ');
 const prettySkill = (s) => s.replace(/-scraper-manager$|-scraper$/, '');
+// ── Scraper healthcheck status (verified / broken + failure reason) ─────────
+const STATUS_JSON = path.join(PROJECTS, 'Designer-Wallcoverings/DW-Programming/ImportNewSkufromURL/data/scraper-audit/scraper-status.json');
+const nkey = (s) => (s || '').toLowerCase().replace(/-scraper-manager$|-scraper$|_catalog$|_colorways?$/, '').replace(/[^a-z0-9]/g, '');
+let _hc = null, _hcTs = 0;
+function healthMap() {
+  if (_hc && Date.now() - _hcTs < 60000) return _hc;
+  const out = [];
+  try {
+    const S = (JSON.parse(readText(STATUS_JSON)).scrapers) || {};
+    for (const k of Object.keys(S)) { const e = S[k] || {}; out.push([k.toLowerCase().replace(/[^a-z0-9]/g, ''), { status: e.status || '', note: (e.notes || e.lastError || e.error || '').toString().replace(/^healthcheck[^—-]*[—-]\s*/, '').replace(/\s+/g, ' ').trim().slice(0, 140) }]); }
+  } catch {}
+  _hc = out; _hcTs = Date.now(); return _hc;
+}
+function healthOf(keys) {
+  const hm = healthMap(); const ks = keys.filter(Boolean);
+  const fz = (k, x) => k === x || (k.length >= 5 && x.length >= 5 && (x.startsWith(k) || k.startsWith(x) || x.includes(k) || k.includes(x)));
+  for (const [hk, v] of hm) if (ks.some(k => fz(k, hk))) return v;
+  return null;
+}
 // freshness stats for a set of tables, CHUNKED so 200+ tables stay under the 15s shell timeout
 function statsForTables(tbls) {
   const stats = {}; const CH = 60;
@@ -573,7 +592,7 @@ function scraperStaleness() {
       else skillOnly.push({ name: prettySkill(s), skill: s });      // non-vendor scraper skill (feed-first, fineartamerica…)
     }
     const stats = statsForTables(Object.keys(entries));
-    const mk = (e, st) => ({ name: e.name, skill: e.skill || '', tbl: e.tbl || '', last: st ? st.last : '', rows: st ? st.rows : null, days: st && st.days != null ? st.days : null, source: st ? st.source : '', cadence: st ? st.cadence : '', scrapeDates: st ? st.scrapeDates : 0, firstScrape: st ? st.firstScrape : '' });
+    const mk = (e, st) => { const hv = healthOf([nkey(e.tbl), nkey(e.skill), nkey(e.name)]); return { name: e.name, skill: e.skill || '', tbl: e.tbl || '', last: st ? st.last : '', rows: st ? st.rows : null, days: st && st.days != null ? st.days : null, source: st ? st.source : '', cadence: st ? st.cadence : '', scrapeDates: st ? st.scrapeDates : 0, firstScrape: st ? st.firstScrape : '', health: hv ? (hv.status || '') : '', healthNote: hv ? hv.note : '' }; };
     const rows = Object.values(entries).map(e => mk(e, stats[e.tbl]));
     for (const so of skillOnly) rows.push(mk(so, null));
     rows.sort((a, b) => a.days == null && b.days == null ? a.name.localeCompare(b.name) : a.days == null ? 1 : b.days == null ? -1 : b.days - a.days);
@@ -582,7 +601,8 @@ function scraperStaleness() {
     const onceOnly = rows.filter(r => r.cadence === 'once').length;
     const neverScraped = rows.filter(r => r.cadence === 'never').length;
     const withSkill = rows.filter(r => r.skill).length;
-    return { ts: new Date().toISOString(), count: rows.length, stale30, nodata, onceOnly, neverScraped, withSkill, rows };
+    const broken = rows.filter(r => r.health === 'broken').length;
+    return { ts: new Date().toISOString(), count: rows.length, stale30, nodata, onceOnly, neverScraped, withSkill, broken, rows };
   });
 }
 

← 71dc524 Scrapers view: expand from 50 skill-backed to ALL 168 vendor  ·  back to Stack Map Viewer  ·  /5x REVISE fixes (TK-12030): persistent drawer (Esc+X only) 09c69a9 →