← back to Marketing Command Center
engine: image-grounded captions (vision) + always include DW SKU
d9e22d373eb5c89813779c54ef4cd639b6548756 · 2026-07-22 06:50:29 -0700 · Steve Abrams
Two changes to daily post-suggestion generation:
1. VISION-GROUNDED COPY — captionsFor() now runs a vision pass (qwen2.5vl:7b)
on the actual product image first, producing a factual visual brief (motif,
scale, color, texture) that grounds the qwen3:14b copy. Old captions were
text-only from title/tags and read generic; new ones describe what the image
actually shows. Graceful: image fetch / vision failure (incl. non-Mac hosts
that can't reach ollama) resolves to '' and captions fall back to text-only.
One vision call per product (not per channel). Source label gains '+vision'.
2. ALWAYS INCLUDE THE DW SKU — candidates.resolveSku() fills dwSku from
dw_sku || sku || variant_sku (strips the '-Sample' suffix; dw_sku is often
empty on the mirror). llm.withSku() then guarantees the SKU in every caption,
sized so channel limits never truncate it, and strips any dangling '· SKU'
tail when no code exists. SYSTEM/userPrompt updated to request it (belt) with
withSku as the deterministic backstop (suspenders).
Verified end-to-end on the real Onda image: vision read the wave motif, all 7
channels carried '· SKU DWTT-73419' within limits; empty-SKU + broken-image
edge cases degrade cleanly. $0 (local ollama).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M modules/engine/candidates.jsM modules/engine/llm.js
Diff
commit d9e22d373eb5c89813779c54ef4cd639b6548756
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Jul 22 06:50:29 2026 -0700
engine: image-grounded captions (vision) + always include DW SKU
Two changes to daily post-suggestion generation:
1. VISION-GROUNDED COPY — captionsFor() now runs a vision pass (qwen2.5vl:7b)
on the actual product image first, producing a factual visual brief (motif,
scale, color, texture) that grounds the qwen3:14b copy. Old captions were
text-only from title/tags and read generic; new ones describe what the image
actually shows. Graceful: image fetch / vision failure (incl. non-Mac hosts
that can't reach ollama) resolves to '' and captions fall back to text-only.
One vision call per product (not per channel). Source label gains '+vision'.
2. ALWAYS INCLUDE THE DW SKU — candidates.resolveSku() fills dwSku from
dw_sku || sku || variant_sku (strips the '-Sample' suffix; dw_sku is often
empty on the mirror). llm.withSku() then guarantees the SKU in every caption,
sized so channel limits never truncate it, and strips any dangling '· SKU'
tail when no code exists. SYSTEM/userPrompt updated to request it (belt) with
withSku as the deterministic backstop (suspenders).
Verified end-to-end on the real Onda image: vision read the wave motif, all 7
channels carried '· SKU DWTT-73419' within limits; empty-SKU + broken-image
edge cases degrade cleanly. $0 (local ollama).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
modules/engine/candidates.js | 16 ++++-
modules/engine/llm.js | 143 ++++++++++++++++++++++++++++++++++++++-----
2 files changed, 140 insertions(+), 19 deletions(-)
diff --git a/modules/engine/candidates.js b/modules/engine/candidates.js
index 7c0de42..d35b71e 100644
--- a/modules/engine/candidates.js
+++ b/modules/engine/candidates.js
@@ -48,7 +48,8 @@ function pool() {
function buildSql(lookbackDays) {
const days = Math.min(3650, Math.max(1, parseInt(lookbackDays, 10) || 14));
return `
- SELECT shopify_id, dw_sku, title, vendor, product_type, image_url, handle, tags,
+ SELECT shopify_id, dw_sku, sku, variant_sku, mfr_sku,
+ title, vendor, product_type, image_url, handle, tags,
created_at_shopify
FROM shopify_products
WHERE status = 'ACTIVE'
@@ -103,6 +104,16 @@ function recentlyFeaturedKeys(noRepeatDays) {
return keys;
}
+// Resolve the customer-facing DW SKU. dw_sku is often empty on the mirror; the
+// real SKU lives in sku/variant_sku carrying a '-Sample' suffix (the first variant
+// is the memo sample). Prefer dw_sku, fall back to sku → variant_sku, strip the
+// trailing sample suffix, and trim. Returns '' if nothing usable is present.
+function resolveSku(row) {
+ const raw = String(row.dw_sku || row.sku || row.variant_sku || '').trim();
+ if (!raw) return '';
+ return raw.replace(/[-_\s]*sample\s*$/i, '').trim();
+}
+
function seenKeyOf(row) {
const out = [];
if (row.dw_sku) out.push(`sku:${String(row.dw_sku).toLowerCase()}`);
@@ -171,7 +182,8 @@ async function pickCandidates() {
return picked.map(r => ({
shopifyId: r.shopify_id != null ? String(r.shopify_id) : '',
- dwSku: r.dw_sku || '',
+ dwSku: resolveSku(r),
+ mfrSku: r.mfr_sku || '',
handle: r.handle || '',
title: r.title || '',
vendor: r.vendor || '',
diff --git a/modules/engine/llm.js b/modules/engine/llm.js
index 9bbd10b..b4c9aa2 100644
--- a/modules/engine/llm.js
+++ b/modules/engine/llm.js
@@ -9,7 +9,10 @@ const { CHANNEL_LIMITS } = require('./config');
const PRIMARY = process.env.OLLAMA_PRIMARY || 'http://127.0.0.1:11434';
const FALLBACK = process.env.OLLAMA_FALLBACK || 'http://192.168.1.133:11434';
const MODEL = process.env.ENGINE_MODEL || 'qwen3:14b';
+const VISION_MODEL = process.env.ENGINE_VISION_MODEL || 'qwen2.5vl:7b';
const TIMEOUT_MS = 45000;
+const VISION_TIMEOUT_MS = 90000;
+const VISION_IMG_WIDTH = 640; // width-cap the fetched image for speed (Shopify CDN honors ?width=)
const CHANNELS = ['instagram', 'facebook', 'bluesky', 'youtube', 'tiktok', 'threads', 'linkedin'];
@@ -19,9 +22,13 @@ Voice rules (HARD):
- Say "wallcovering" — never "wallpaper" — when speaking generically.
- No generic superlatives: never "best", "amazing", "stunning", "gorgeous", "must-have".
- Use sensory, material language: texture, light, movement, hand, depth, sheen.
-- Describe ONLY qualities inferable from the given title, type, and tags. NEVER
- invent materials, colorway names, origins, or construction details not present
- in the input.
+- Ground every caption in the VISUAL DESCRIPTION provided (what the product image
+ actually shows — motif, scale, color, texture). Describe only qualities present
+ in that description, the title, type, or tags. NEVER invent colorway names,
+ origins, or construction details you cannot see or that aren't in the input.
+- If a DW SKU is provided, include it exactly as given, verbatim, in every
+ caption's text (a natural "· SKU <code>" tail is fine); never alter or
+ abbreviate it. If NO SKU is provided, do not write the word "SKU" at all.
- NEVER mention price, cost, sale, or discount.
Per-channel voice:
- instagram: polished and editorial, 5-8 hashtags.
@@ -33,16 +40,23 @@ Per-channel voice:
- threads: conversational and human, <= 400 characters.
Return ONLY valid JSON, no prose, no code fences.`;
-function userPrompt(product) {
+function userPrompt(product, brief) {
const p = {
title: product.title || '',
vendor: product.vendor || '',
productType: product.productType || '',
+ dwSku: product.dwSku || '',
tags: Array.isArray(product.tags) ? product.tags.slice(0, 12) : [],
};
+ const visual = (brief && String(brief).trim())
+ ? `\nVISUAL DESCRIPTION of the actual product image (ground your copy in this):\n${String(brief).trim()}\n`
+ : '';
+ const skuLine = p.dwSku
+ ? `\nHARD REQUIREMENT: include the exact SKU "${p.dwSku}" verbatim in every caption's text.\n`
+ : '';
return `Write social captions for this Designer Wallcoverings product:
${JSON.stringify(p, null, 2)}
-
+${visual}${skuLine}
Output EXACTLY this JSON shape (fill every field, obey the per-channel voice + limits):
{
"instagram": { "caption": "...", "hashtags": ["...", "..."] },
@@ -55,8 +69,78 @@ Output EXACTLY this JSON shape (fill every field, obey the per-channel voice + l
}`;
}
+// ── Vision pass: image → grounded visual brief ───────────────────────────────
+// Fetches the product image, sends it to a local vision model, and returns a
+// short factual description of what is ACTUALLY shown (motif, scale, color,
+// texture). This grounds the captions in the real image instead of the title.
+// Fully graceful: any failure (no image, fetch error, model down, non-Mac host
+// that can't reach ollama) resolves to '' so text generation proceeds unchanged.
+const VISION_PROMPT = `You are looking at a luxury wallcovering (or fabric) product image.
+In 2-3 factual sentences describe ONLY what you can actually see:
+- the motif / pattern and its scale (small repeat, large-scale, all-over, geometric, floral, textural plain, etc.)
+- the dominant colors and tonal feel
+- the surface texture or finish (matte, sheen, grasscloth, embossed, woven, metallic, flat, etc.)
+Do NOT name a brand, collection, price, or a material you cannot visually confirm. No preamble — just the description.`;
+
+async function fetchImageB64(url) {
+ if (!url) return null;
+ // Ask the CDN for a smaller render when it supports ?width= (Shopify/most CDNs do).
+ const sized = /\?/.test(url) ? `${url}&width=${VISION_IMG_WIDTH}` : `${url}?width=${VISION_IMG_WIDTH}`;
+ const ctrl = new AbortController();
+ const timer = setTimeout(() => ctrl.abort(), 20000);
+ try {
+ let res = await fetch(sized, { signal: ctrl.signal });
+ if (!res.ok) res = await fetch(url, { signal: ctrl.signal }); // fall back to the raw URL
+ if (!res.ok) throw new Error(`image HTTP ${res.status}`);
+ const buf = Buffer.from(await res.arrayBuffer());
+ if (!buf.length) throw new Error('empty image body');
+ return buf.toString('base64');
+ } finally {
+ clearTimeout(timer);
+ }
+}
+
+async function visionBriefAt(base, b64) {
+ const ctrl = new AbortController();
+ const timer = setTimeout(() => ctrl.abort(), VISION_TIMEOUT_MS);
+ try {
+ const res = await fetch(`${base}/api/chat`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ signal: ctrl.signal,
+ body: JSON.stringify({
+ model: VISION_MODEL,
+ stream: false,
+ options: { temperature: 0.3, num_predict: 180 },
+ messages: [{ role: 'user', content: VISION_PROMPT, images: [b64] }],
+ }),
+ });
+ if (!res.ok) throw new Error(`vision ${base} HTTP ${res.status}`);
+ const data = await res.json();
+ const content = data && data.message && data.message.content;
+ if (!content) throw new Error(`vision ${base} empty content`);
+ return String(content).trim();
+ } finally {
+ clearTimeout(timer);
+ }
+}
+
+// visualBrief(product) — primary → fallback → ''. Never rejects.
+async function visualBrief(product) {
+ let b64;
+ try { b64 = await fetchImageB64(product && product.imageUrl); }
+ catch (e) { console.error('[engine vision] image fetch failed —', e.message); return ''; }
+ if (!b64) return '';
+ try { return await visionBriefAt(PRIMARY, b64); }
+ catch (e1) {
+ console.error('[engine vision] primary failed —', e1.message);
+ try { return await visionBriefAt(FALLBACK, b64); }
+ catch (e2) { console.error('[engine vision] fallback failed —', e2.message); return ''; }
+ }
+}
+
// ── Ollama chat call ─────────────────────────────────────────────────────────
-async function ollamaChat(base, product) {
+async function ollamaChat(base, product, brief) {
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), TIMEOUT_MS);
try {
@@ -72,7 +156,7 @@ async function ollamaChat(base, product) {
options: { temperature: 0.8, num_predict: 900 },
messages: [
{ role: 'system', content: SYSTEM },
- { role: 'user', content: userPrompt(product) },
+ { role: 'user', content: userPrompt(product, brief) },
],
}),
});
@@ -177,6 +261,24 @@ function tidyHashtags(list) {
.filter(h => h && h.length > 1);
}
+// Guarantee the DW SKU appears in a caption without ever exceeding the channel
+// limit: if the model already wrote the SKU (case-insensitive), just tidy; else
+// reserve room for a " · SKU <code>" tail, tidy the body into the remaining space,
+// and append. This is the deterministic backstop behind the prompt instruction —
+// "always include our SKU" holds even when the model (or the template) omits it.
+function withSku(text, limit, sku) {
+ // Strip any DANGLING "· SKU" tail the model may add when no real code follows
+ // (only matches SKU at the very end, so a valid "SKU DWTT-73419" is preserved).
+ const cleaned = String(text == null ? '' : text).replace(/\s*[·•|–—-]?\s*SKU\s*$/i, '').trim();
+ const base = tidy(cleaned, limit);
+ if (!sku) return base;
+ if (base.toLowerCase().includes(String(sku).toLowerCase())) return base;
+ const suffix = ` · SKU ${sku}`;
+ if (!limit) return (base + suffix).trim();
+ const room = Math.max(0, limit - suffix.length);
+ return (tidy(cleaned, room) + suffix).trim();
+}
+
// Merge model output over the template so a missing channel/field is always
// filled, then post-process EVERY field regardless of source.
function finalize(modelOut, product) {
@@ -187,31 +289,32 @@ function finalize(modelOut, product) {
return (v != null && String(v).trim()) ? v : tmpl[ch][field];
};
const L = CHANNEL_LIMITS;
+ const sku = product && product.dwSku ? String(product.dwSku).trim() : '';
return {
instagram: {
- caption: tidy(pick('instagram', 'caption'), L.instagram),
+ caption: withSku(pick('instagram', 'caption'), L.instagram, sku),
hashtags: tidyHashtags((m.instagram && m.instagram.hashtags) || tmpl.instagram.hashtags),
},
facebook: {
- caption: tidy(pick('facebook', 'caption'), L.facebook),
+ caption: withSku(pick('facebook', 'caption'), L.facebook, sku),
},
bluesky: {
- caption: tidy(pick('bluesky', 'caption'), L.bluesky),
+ caption: withSku(pick('bluesky', 'caption'), L.bluesky, sku),
},
youtube: {
title: tidy(pick('youtube', 'title'), L.youtube),
- description: tidy(pick('youtube', 'description'), 5000),
+ description: withSku(pick('youtube', 'description'), 5000, sku),
},
tiktok: {
- caption: tidy(pick('tiktok', 'caption'), L.tiktok),
+ caption: withSku(pick('tiktok', 'caption'), L.tiktok, sku),
hashtags: tidyHashtags((m.tiktok && m.tiktok.hashtags) || tmpl.tiktok.hashtags).slice(0, 5),
},
threads: {
- caption: tidy(pick('threads', 'caption'), L.threads),
+ caption: withSku(pick('threads', 'caption'), L.threads, sku),
},
linkedin: {
- caption: tidy(pick('linkedin', 'caption'), L.linkedin),
+ caption: withSku(pick('linkedin', 'caption'), L.linkedin, sku),
hashtags: tidyHashtags((m.linkedin && m.linkedin.hashtags) || tmpl.linkedin.hashtags).slice(0, 3),
},
};
@@ -223,13 +326,18 @@ async function captionsFor(product) {
let modelOut = null;
let source = 'template';
+ // Vision pass first — grounds the copy in the ACTUAL product image. Resolves to
+ // '' on any failure (no image, model down, or a host that can't reach ollama —
+ // e.g. Kamatera), in which case captions fall back to metadata-only text.
+ const brief = await visualBrief(product);
+
try {
- modelOut = await ollamaChat(PRIMARY, product);
+ modelOut = await ollamaChat(PRIMARY, product, brief);
source = `${MODEL}@mac2`;
} catch (e1) {
console.error('[engine llm] primary failed —', e1.message);
try {
- modelOut = await ollamaChat(FALLBACK, product);
+ modelOut = await ollamaChat(FALLBACK, product, brief);
source = `${MODEL}@mac1`;
} catch (e2) {
console.error('[engine llm] fallback failed —', e2.message);
@@ -239,7 +347,8 @@ async function captionsFor(product) {
}
const out = finalize(modelOut, product);
- out._source = source;
+ out._source = brief ? `${source}+vision` : source;
+ out._brief = brief || null;
return out;
}
← 7ff5d7c deploy: ship Threads un-gate to Kamatera MCC (app 69 reloade
·
back to Marketing Command Center
·
engine: untrack runtime engine-queue.json + protect it on de d84dc2a →