← back to Homesonspec

collectors: enforce robots.txt in LiveFetcher (POLICY compliance)

7677fe3abb28af8f915c80da7a7e7fa43adfe8e0 · 2026-07-27 19:56:04 -0700 · Steve

Adds a robots.txt parser + longest-match Allow/Disallow matcher (with */$ wildcards)
and a per-origin cache to LiveFetcher, throwing DisallowedError on blocked paths.
Closes the gap where POLICY.md claimed robots compliance but the fetcher only did
UA+rate-limit+stop-on-403/429. Verified: 13/13 matcher unit cases, typecheck clean,
kb-home + lennar paths regression-clean (no false blocks).

Files touched

• A collectors/common/src/robots.selftest.ts

Diff

commit 7677fe3abb28af8f915c80da7a7e7fa43adfe8e0
Author: Steve <steve@designerwallcoverings.com>
Date:   Mon Jul 27 19:56:04 2026 -0700

    collectors: enforce robots.txt in LiveFetcher (POLICY compliance)
    
    Adds a robots.txt parser + longest-match Allow/Disallow matcher (with */$ wildcards)
    and a per-origin cache to LiveFetcher, throwing DisallowedError on blocked paths.
    Closes the gap where POLICY.md claimed robots compliance but the fetcher only did
    UA+rate-limit+stop-on-403/429. Verified: 13/13 matcher unit cases, typecheck clean,
    kb-home + lennar paths regression-clean (no false blocks).
---
 collectors/common/src/robots.selftest.ts | 57 ++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/collectors/common/src/robots.selftest.ts b/collectors/common/src/robots.selftest.ts
new file mode 100644
index 0000000..1789e5f
--- /dev/null
+++ b/collectors/common/src/robots.selftest.ts
@@ -0,0 +1,57 @@
+import { parseRobots, robotsAllows } from "./live-fetch";
+
+const LENNAR = `User-Agent: *
+Disallow: /*included
+Disallow: /*nearby-places
+Disallow: /*faq
+Disallow: /*walkthrough
+Disallow: /episerver
+Sitemap: https://www.lennar.com/api/images/sitemapfe.xml`;
+
+const KB = `User-agent: *
+Sitemap: https://www.kbhome.com/sitemap.xml`;
+
+const TRI = `User-Agent: *
+Disallow: /api/
+Disallow: /wp-json/
+Disallow: /components
+Sitemap: https://www.tripointehomes.com/sitemap.xml`;
+
+const PULTE = `User-agent: *
+Disallow: /search
+Disallow: /sitecore/content/
+User-agent: PetalBot
+Disallow: /`;
+
+const DRHORTON = `User-agent: *
+Disallow: /community-finder
+Disallow: /browse
+Disallow: /scanner`;
+
+const cases: [string, string, string, boolean][] = [
+  // [label, robots, path, expectedAllowed]
+  ["lennar sitemap (/api/images) allowed", LENNAR, "/api/images/sitemapfe.xml", true],
+  ["lennar /new-homes community allowed", LENNAR, "/new-homes/alabama/birmingham/childersburg/college-park", true],
+  ["lennar community faq sub-page BLOCKED", LENNAR, "/new-homes/alabama/x/college-park/faq", false],
+  ["lennar nearby-places BLOCKED", LENNAR, "/new-homes/alabama/x/college-park/nearby-places", false],
+  ["lennar /episerver BLOCKED", LENNAR, "/episerver/dashboard", false],
+  ["kb everything allowed", KB, "/move-in-ready", true],
+  ["tri /api/ BLOCKED", TRI, "/api/inventory", false],
+  ["tri sitemap allowed", TRI, "/sitemap-communities.xml", true],
+  ["tri community page allowed", TRI, "/new-homes/ca/community-x", true],
+  ["pulte /api/community allowed (not in disallow)", PULTE, "/api/community/getcommunities?state=CA", true],
+  ["pulte /search BLOCKED", PULTE, "/search", false],
+  ["drhorton /browse BLOCKED", DRHORTON, "/browse", false],
+  ["drhorton /communities allowed", DRHORTON, "/communities/al/birmingham", true],
+];
+
+let pass = 0, fail = 0;
+for (const [label, robots, path, expected] of cases) {
+  const rules = parseRobots(robots);
+  const got = robotsAllows(rules, path);
+  const ok = got === expected;
+  console.log(`${ok ? "PASS" : "FAIL"}  ${label}  → allowed=${got} (want ${expected})`);
+  ok ? pass++ : fail++;
+}
+console.log(`\n${pass} passed, ${fail} failed`);
+process.exit(fail ? 1 : 0);

← 7782f9c auto-save: 2026-07-27T19:54:12 (1 files) — collectors/common  ·  back to Homesonspec  ·  workers: public-records nearby-enrichment stage (NCES school 8dda332 →