← back to Wallco Ai

wallco.ai · Last-Modified + 304 on /sitemap.xml + /robots.txt — sitemap pinned to CATALOG_LAST_MODIFIED, robots.txt pinned to process start; both also override the global no-store middleware with cacheable Cache-Control (sitemap public+1h+must-revalidate, robots public+1d+must-revalidate) so crawlers actually issue conditional GETs; verified IMS-match → 304 0B, stale IMS → 200 32KB on /sitemap.xml

8d04faaf8c84ba8d00e8de5b5ae9cfd4d4527b11 · 2026-05-12 07:27:31 -0700 · SteveStudio2

Files touched

• M server.js

Diff

commit 8d04faaf8c84ba8d00e8de5b5ae9cfd4d4527b11
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date:   Tue May 12 07:27:31 2026 -0700

    wallco.ai · Last-Modified + 304 on /sitemap.xml + /robots.txt — sitemap pinned to CATALOG_LAST_MODIFIED, robots.txt pinned to process start; both also override the global no-store middleware with cacheable Cache-Control (sitemap public+1h+must-revalidate, robots public+1d+must-revalidate) so crawlers actually issue conditional GETs; verified IMS-match → 304 0B, stale IMS → 200 32KB on /sitemap.xml
---
 server.js | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/server.js b/server.js
index ecb9da1..85ed233 100644
--- a/server.js
+++ b/server.js
@@ -124,7 +124,13 @@ app.use('/designs/room', express.static(path.join(__dirname, 'data', 'rooms'), {
 }));
 
 // ── robots.txt — allow all + reference the sitemap.
+// Last-Modified pinned to process start (robots policy rarely changes; if it
+// does, the deploy/restart updates the timestamp). Override the global
+// no-store middleware with 1-day cacheable to maximize crawler win.
+const ROBOTS_LAST_MODIFIED = new Date();
 app.get('/robots.txt', (req, res) => {
+  res.setHeader('Cache-Control', 'public, max-age=86400, must-revalidate');
+  if (maybe304(req, res, ROBOTS_LAST_MODIFIED)) return;
   res.type('text/plain').send(
     `User-agent: *\nAllow: /\nDisallow: /admin/\nDisallow: /api/\nDisallow: /_devlogin\n\nSitemap: https://${SITE}/sitemap.xml\n`
   );
@@ -135,6 +141,9 @@ app.get('/robots.txt', (req, res) => {
 // designs with a room mockup get a +0.1 bonus capped at 0.9. Cached 1h.
 let SITEMAP_CACHE = { body: '', exp: 0 };
 app.get('/sitemap.xml', (req, res) => {
+  // Conditional GET — sitemap freshness = freshest design's created_at.
+  res.setHeader('Cache-Control', 'public, max-age=3600, must-revalidate');
+  if (maybe304(req, res, CATALOG_LAST_MODIFIED)) return;
   const now = Date.now();
   if (SITEMAP_CACHE.body && SITEMAP_CACHE.exp > now) {
     return res.type('application/xml').send(SITEMAP_CACHE.body);

← b74701c wallco.ai · Last-Modified + If-Modified-Since on /designs an  ·  back to Wallco Ai  ·  wallco.ai · all target=_blank now rel=noopener noreferrer (t d6cdb24 →