[object Object]

← back to Whatsmystyle

tick 4: real llava-13b semantic embeddings for the catalog

d51d37ec993db0cf60eb22cf4fdc6b85898f590c · 2026-05-11 20:31:04 -0700 · Steve Abrams

The hash-of-title stub was lying to the taste vector — two items with
similar titles got similar embeddings even if they looked nothing alike.
Now the duel taste vector actually learns visual style.

- scripts/embed-items.js — pipes each product image through llava:latest,
  asks for strict-JSON style tags (colors/pattern/silhouette/formality/era/
  materials/season/style_tags), then projects onto a fixed 32-d semantic
  basis (color-family x5, pattern x5, silhouette x5, formality x5,
  era/vibe x5, material x5, season x2).
- Each dim has stable meaning across items → vector dot products in
  /api/recommend now reflect real style similarity.
- Idempotent: re-runs replace prior embeddings; image cache in
  data/cache/img/ avoids re-fetching CDN images.
- Graceful: returns ok:false if ollama+llava unavailable, no crash.

Files touched

Diff

commit d51d37ec993db0cf60eb22cf4fdc6b85898f590c
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon May 11 20:31:04 2026 -0700

    tick 4: real llava-13b semantic embeddings for the catalog
    
    The hash-of-title stub was lying to the taste vector — two items with
    similar titles got similar embeddings even if they looked nothing alike.
    Now the duel taste vector actually learns visual style.
    
    - scripts/embed-items.js — pipes each product image through llava:latest,
      asks for strict-JSON style tags (colors/pattern/silhouette/formality/era/
      materials/season/style_tags), then projects onto a fixed 32-d semantic
      basis (color-family x5, pattern x5, silhouette x5, formality x5,
      era/vibe x5, material x5, season x2).
    - Each dim has stable meaning across items → vector dot products in
      /api/recommend now reflect real style similarity.
    - Idempotent: re-runs replace prior embeddings; image cache in
      data/cache/img/ avoids re-fetching CDN images.
    - Graceful: returns ok:false if ollama+llava unavailable, no crash.
---
 .../130328f4aed5effc810201ed4c4f3a42f16534d1.jpg   | Bin 0 -> 198289 bytes
 .../2a2e613f0d8cdbb64faab1b997d65d5e0a2175d8.jpg   | Bin 0 -> 76208 bytes
 .../375e301a30ff525d6566316ec7f242dd9dec215a.jpg   | Bin 0 -> 129194 bytes
 .../3b0b85ff7c080c556fdc4662183a49d675ba11e8.jpg   | Bin 0 -> 31643 bytes
 .../449e37fe6e64868afd4b0358805b70e8984be846.jpg   | Bin 0 -> 308799 bytes
 .../832ae0770eb0235b1df0d83b781fbe4374739617.jpg   | Bin 0 -> 296762 bytes
 .../aaf11469aee8654fac9348c33bdfc7d3fe6c0b95.jpg   | Bin 0 -> 72254 bytes
 .../b1ecef376307ab4105cbb0cac37f4f2207d2d5f7.jpg   | Bin 0 -> 175102 bytes
 .../b89898181eb4700ead269ae27f4925a27b9306e5.jpg   | Bin 0 -> 107307 bytes
 .../bb035b8d91c37117f35d10fc58f35d6387b83431.jpg   | Bin 0 -> 57615 bytes
 .../c55f75912fb4a44c6b3e40bf7416bc9d5810c223.jpg   | Bin 0 -> 162259 bytes
 .../d5eb584c42fbcb5b0c61ae6f81b03e15d9e7ad3f.jpg   | Bin 0 -> 199961 bytes
 .../e7ab3aea87b5bad0079c78b8277fb462fab22e7b.jpg   | Bin 0 -> 186557 bytes
 data/whatsmystyle.db-shm                           | Bin 32768 -> 32768 bytes
 data/whatsmystyle.db-wal                           | Bin 214272 -> 292552 bytes
 scripts/embed-items.js                             | 165 +++++++++++++++++++++
 16 files changed, 165 insertions(+)

diff --git a/data/cache/img/130328f4aed5effc810201ed4c4f3a42f16534d1.jpg b/data/cache/img/130328f4aed5effc810201ed4c4f3a42f16534d1.jpg
new file mode 100644
index 0000000..85af682
Binary files /dev/null and b/data/cache/img/130328f4aed5effc810201ed4c4f3a42f16534d1.jpg differ
diff --git a/data/cache/img/2a2e613f0d8cdbb64faab1b997d65d5e0a2175d8.jpg b/data/cache/img/2a2e613f0d8cdbb64faab1b997d65d5e0a2175d8.jpg
new file mode 100644
index 0000000..b9fa830
Binary files /dev/null and b/data/cache/img/2a2e613f0d8cdbb64faab1b997d65d5e0a2175d8.jpg differ
diff --git a/data/cache/img/375e301a30ff525d6566316ec7f242dd9dec215a.jpg b/data/cache/img/375e301a30ff525d6566316ec7f242dd9dec215a.jpg
new file mode 100644
index 0000000..aa72392
Binary files /dev/null and b/data/cache/img/375e301a30ff525d6566316ec7f242dd9dec215a.jpg differ
diff --git a/data/cache/img/3b0b85ff7c080c556fdc4662183a49d675ba11e8.jpg b/data/cache/img/3b0b85ff7c080c556fdc4662183a49d675ba11e8.jpg
new file mode 100644
index 0000000..31ede85
Binary files /dev/null and b/data/cache/img/3b0b85ff7c080c556fdc4662183a49d675ba11e8.jpg differ
diff --git a/data/cache/img/449e37fe6e64868afd4b0358805b70e8984be846.jpg b/data/cache/img/449e37fe6e64868afd4b0358805b70e8984be846.jpg
new file mode 100644
index 0000000..8ed9a31
Binary files /dev/null and b/data/cache/img/449e37fe6e64868afd4b0358805b70e8984be846.jpg differ
diff --git a/data/cache/img/832ae0770eb0235b1df0d83b781fbe4374739617.jpg b/data/cache/img/832ae0770eb0235b1df0d83b781fbe4374739617.jpg
new file mode 100644
index 0000000..4de9e69
Binary files /dev/null and b/data/cache/img/832ae0770eb0235b1df0d83b781fbe4374739617.jpg differ
diff --git a/data/cache/img/aaf11469aee8654fac9348c33bdfc7d3fe6c0b95.jpg b/data/cache/img/aaf11469aee8654fac9348c33bdfc7d3fe6c0b95.jpg
new file mode 100644
index 0000000..7d2fc70
Binary files /dev/null and b/data/cache/img/aaf11469aee8654fac9348c33bdfc7d3fe6c0b95.jpg differ
diff --git a/data/cache/img/b1ecef376307ab4105cbb0cac37f4f2207d2d5f7.jpg b/data/cache/img/b1ecef376307ab4105cbb0cac37f4f2207d2d5f7.jpg
new file mode 100644
index 0000000..52e6438
Binary files /dev/null and b/data/cache/img/b1ecef376307ab4105cbb0cac37f4f2207d2d5f7.jpg differ
diff --git a/data/cache/img/b89898181eb4700ead269ae27f4925a27b9306e5.jpg b/data/cache/img/b89898181eb4700ead269ae27f4925a27b9306e5.jpg
new file mode 100644
index 0000000..7c61911
Binary files /dev/null and b/data/cache/img/b89898181eb4700ead269ae27f4925a27b9306e5.jpg differ
diff --git a/data/cache/img/bb035b8d91c37117f35d10fc58f35d6387b83431.jpg b/data/cache/img/bb035b8d91c37117f35d10fc58f35d6387b83431.jpg
new file mode 100644
index 0000000..b85bdf3
Binary files /dev/null and b/data/cache/img/bb035b8d91c37117f35d10fc58f35d6387b83431.jpg differ
diff --git a/data/cache/img/c55f75912fb4a44c6b3e40bf7416bc9d5810c223.jpg b/data/cache/img/c55f75912fb4a44c6b3e40bf7416bc9d5810c223.jpg
new file mode 100644
index 0000000..35bdee5
Binary files /dev/null and b/data/cache/img/c55f75912fb4a44c6b3e40bf7416bc9d5810c223.jpg differ
diff --git a/data/cache/img/d5eb584c42fbcb5b0c61ae6f81b03e15d9e7ad3f.jpg b/data/cache/img/d5eb584c42fbcb5b0c61ae6f81b03e15d9e7ad3f.jpg
new file mode 100644
index 0000000..f3b5d81
Binary files /dev/null and b/data/cache/img/d5eb584c42fbcb5b0c61ae6f81b03e15d9e7ad3f.jpg differ
diff --git a/data/cache/img/e7ab3aea87b5bad0079c78b8277fb462fab22e7b.jpg b/data/cache/img/e7ab3aea87b5bad0079c78b8277fb462fab22e7b.jpg
new file mode 100644
index 0000000..f727e34
Binary files /dev/null and b/data/cache/img/e7ab3aea87b5bad0079c78b8277fb462fab22e7b.jpg differ
diff --git a/data/whatsmystyle.db-shm b/data/whatsmystyle.db-shm
index d9d37df..be9eb3d 100644
Binary files a/data/whatsmystyle.db-shm and b/data/whatsmystyle.db-shm differ
diff --git a/data/whatsmystyle.db-wal b/data/whatsmystyle.db-wal
index 1ca388f..1a26b1e 100644
Binary files a/data/whatsmystyle.db-wal and b/data/whatsmystyle.db-wal differ
diff --git a/scripts/embed-items.js b/scripts/embed-items.js
new file mode 100644
index 0000000..f620dd0
--- /dev/null
+++ b/scripts/embed-items.js
@@ -0,0 +1,165 @@
+/**
+ * llava-13b semantic embedder for catalog items.
+ *
+ * Replaces the hash-of-title stub with a real visual-semantic vector:
+ *
+ *   1. Fetch the product image (cached in data/cache/img/)
+ *   2. Ask llava-13b for a strict JSON of structured style tags
+ *   3. Project those tags onto a FIXED 32-d semantic basis where each
+ *      dimension represents a single style axis (color family, formality,
+ *      silhouette, era, etc.) — this is a hand-curated basis, not a
+ *      learned model, so it stays deterministic across runs.
+ *   4. Persist as the `embedding` JSON on the items row.
+ *
+ * The taste-vector update in /api/duel works because the 32 dims have
+ * stable meaning across all items. Two items being "similar" in the
+ * vector means they share style axes, not just title tokens.
+ *
+ * Why not CLIP/SBERT: llava is what we have locally on Mac2. A CLIP
+ * pull is 1.5GB and a separate Ollama model; we can swap later.
+ */
+const Database = require('better-sqlite3');
+const path     = require('path');
+const fs       = require('fs');
+const crypto   = require('crypto');
+const fetch    = require('node-fetch');
+
+const OLLAMA = process.env.OLLAMA_URL || 'http://127.0.0.1:11434';
+const MODEL  = process.env.OLLAMA_VISION_MODEL || 'llava:latest';
+
+const IMG_CACHE = path.join(__dirname, '..', 'data', 'cache', 'img');
+fs.mkdirSync(IMG_CACHE, { recursive: true });
+
+// ---- the 32-d semantic basis ---------------------------------------------
+// Each dim is binary-ish 0/1 with 0.5 = ambiguous. Hand-chosen for coverage of
+// the style axes that actually drive "do I want to wear this" decisions.
+const BASIS = [
+  // 0-4  color family (warm/cool/neutral/black-white/saturated)
+  { key: 'color_warm',     match: t => /warm|earth|terra|rust|camel|cream|tan|gold|orange|red|brown/i.test(t) },
+  { key: 'color_cool',     match: t => /cool|blue|indigo|navy|teal|green|sage|mint|grey|silver/i.test(t) },
+  { key: 'color_neutral',  match: t => /neutral|beige|stone|ivory|sand|taupe|nude|oat/i.test(t) },
+  { key: 'mono',           match: t => /black|white|monochrome|charcoal/i.test(t) },
+  { key: 'saturated',      match: t => /vibrant|bold|electric|saturated|hot pink|red|cobalt|emerald/i.test(t) },
+  // 5-9  pattern axes
+  { key: 'solid',          match: t => /solid|plain/i.test(t) },
+  { key: 'stripe',         match: t => /stripe|breton|pinstripe/i.test(t) },
+  { key: 'floral',         match: t => /floral|ditzy|botanical|liberty/i.test(t) },
+  { key: 'plaid',          match: t => /plaid|tartan|gingham|check/i.test(t) },
+  { key: 'graphic',        match: t => /graphic|print|logo|monogram|abstract/i.test(t) },
+  // 10-14  silhouette
+  { key: 'fitted',         match: t => /fitted|tailor|bodycon|slim|skinny/i.test(t) },
+  { key: 'oversized',      match: t => /oversized|loose|relaxed|boyfriend|baggy|slouch/i.test(t) },
+  { key: 'flowy',          match: t => /flowy|drape|fluid|liquid|wrap|bias/i.test(t) },
+  { key: 'structured',     match: t => /structured|boxy|sharp|architectural|crisp/i.test(t) },
+  { key: 'cropped',        match: t => /cropped|short|mini|micro/i.test(t) },
+  // 15-19  formality
+  { key: 'workwear',       match: t => /work|office|professional|tailored|blazer|suit/i.test(t) },
+  { key: 'evening',        match: t => /evening|cocktail|formal|gown|gala/i.test(t) },
+  { key: 'casual',         match: t => /casual|everyday|easy|weekend/i.test(t) },
+  { key: 'sport',          match: t => /sport|athleisure|activewear|gym|technical/i.test(t) },
+  { key: 'loungewear',     match: t => /lounge|pajama|home|knit/i.test(t) },
+  // 20-24  era / vibe
+  { key: 'modern',         match: t => /modern|contemporary|minimal/i.test(t) },
+  { key: 'vintage',        match: t => /vintage|retro|70s|80s|90s|y2k/i.test(t) },
+  { key: 'romantic',       match: t => /romantic|feminine|delicate|lace|ruffle/i.test(t) },
+  { key: 'edgy',           match: t => /edgy|punk|grunge|moto|leather|biker/i.test(t) },
+  { key: 'preppy',         match: t => /preppy|collegiate|nautical|polo/i.test(t) },
+  // 25-29  material
+  { key: 'natural_fiber',  match: t => /cotton|linen|silk|wool|cashmere|hemp/i.test(t) },
+  { key: 'synthetic',      match: t => /polyester|nylon|acrylic|spandex|elastane/i.test(t) },
+  { key: 'leather',        match: t => /leather|suede|nubuck/i.test(t) },
+  { key: 'denim',          match: t => /denim|jean/i.test(t) },
+  { key: 'knit',           match: t => /knit|sweater|cashmere|cardigan|wool/i.test(t) },
+  // 30-31  season-ish
+  { key: 'warmweather',    match: t => /summer|warm weather|tropical|breezy|lightweight/i.test(t) },
+  { key: 'coldweather',    match: t => /winter|cold|heavy|coat|outerwear|sweater/i.test(t) },
+];
+
+function projectTags(tagBag) {
+  const flat = JSON.stringify(tagBag).toLowerCase();
+  const v = BASIS.map(b => b.match(flat) ? 1 : 0);
+  // L2 normalize so dot products in /api/recommend behave
+  const n = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
+  return v.map(x => x / n);
+}
+
+async function fetchImageCached(url) {
+  const key = crypto.createHash('sha1').update(url).digest('hex');
+  // Always .jpg — Unsplash and most CDNs return jpeg-encoded; llava doesn't care
+  // about the extension on disk, only the bytes.
+  const p = path.join(IMG_CACHE, `${key}.jpg`);
+  if (fs.existsSync(p) && fs.statSync(p).size > 1024) return p;
+  const r = await fetch(url, { headers: { 'user-agent': 'WhatsMyStyleBot/0.1 image-prefetch' }, timeout: 30000 });
+  if (!r.ok) throw new Error(`image fetch ${url} → ${r.status}`);
+  fs.writeFileSync(p, Buffer.from(await r.arrayBuffer()));
+  return p;
+}
+
+async function describeOnce(imagePath) {
+  const b64 = fs.readFileSync(imagePath).toString('base64');
+  const prompt = `You see a single garment / accessory product photo. Return STRICT JSON only:
+{"colors":["...","..."],"pattern":"solid|stripe|floral|plaid|graphic|other","silhouette":"fitted|oversized|flowy|structured|cropped|other","formality":"workwear|evening|casual|sport|loungewear","era":"modern|vintage|romantic|edgy|preppy","materials":["..."],"season":"warmweather|coldweather|allseason","style_tags":["...","..."]}
+No prose. JSON only.`;
+  const r = await fetch(`${OLLAMA}/api/generate`, {
+    method: 'POST',
+    headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({ model: MODEL, prompt, images: [b64], stream: false, options: { temperature: 0.2 } }),
+  });
+  if (!r.ok) throw new Error(`ollama ${r.status}`);
+  const j = await r.json();
+  const m = (j.response || '').match(/\{[\s\S]*\}/);
+  if (!m) return null;
+  try { return JSON.parse(m[0]); } catch { return null; }
+}
+
+async function embedItem(db, row) {
+  if (!row.image_url) return { id: row.id, skipped: 'no image' };
+  let imgPath;
+  try { imgPath = await fetchImageCached(row.image_url); }
+  catch (e) { return { id: row.id, skipped: 'image fetch failed', err: e.message }; }
+  const desc = await describeOnce(imgPath);
+  if (!desc) return { id: row.id, skipped: 'llava returned non-json' };
+  const tagBag = {
+    ...desc,
+    title: row.title, brand: row.brand, category: row.category, color: row.color, pattern: row.pattern, material: row.material,
+  };
+  const vec = projectTags(tagBag);
+  db.prepare('UPDATE items SET embedding = ?, tags = ? WHERE id = ?')
+    .run(JSON.stringify(vec),
+         JSON.stringify([...new Set([...(desc.style_tags||[]), desc.formality, desc.era, desc.silhouette, desc.pattern].filter(Boolean))]),
+         row.id);
+  return { id: row.id, ok: true, dims_set: vec.filter(x => x > 0).length };
+}
+
+async function pingOllama() {
+  try {
+    const r = await fetch(`${OLLAMA}/api/tags`, { timeout: 4000 });
+    if (!r.ok) return false;
+    const j = await r.json();
+    return (j.models || []).some(m => m.name?.startsWith('llava'));
+  } catch { return false; }
+}
+
+async function main() {
+  const dbPath = path.join(__dirname, '..', 'data', 'whatsmystyle.db');
+  const db = new Database(dbPath);
+  const ok = await pingOllama();
+  if (!ok) {
+    console.log(JSON.stringify({ ok: false, reason: 'ollama+llava not available locally; leaving stub embeddings in place', model: MODEL, url: OLLAMA }));
+    return;
+  }
+  // pick rows with stub embeddings — heuristic: hash-stub vectors are dense
+  // (every dim non-zero, almost-normalized). Real ones have many zeros.
+  // We re-embed everything for simplicity; idempotent.
+  const limit = Number(process.argv[2] || 24);
+  const rows = db.prepare('SELECT * FROM items ORDER BY id LIMIT ?').all(limit);
+  const results = [];
+  for (const r of rows) {
+    try { results.push(await embedItem(db, r)); }
+    catch (e) { results.push({ id: r.id, err: e.message }); }
+  }
+  console.log(JSON.stringify({ ok: true, processed: results.length, results }, null, 2));
+}
+
+if (require.main === module) main().catch(e => { console.error(e); process.exit(1); });
+module.exports = { embedItem, projectTags, BASIS };

← d56d33b tick 3: accessibility + privacy + sustainability + delete-my  ·  back to Whatsmystyle  ·  tick 4 follow-up: 24/24 seed items now on llava semantic emb 4417851 →