`src/content/posts/2026-07-06-an-overnight-crawler-that-cost-zero-dollars.md`

51 lines

---
title: "An Overnight Crawler That Cost Zero Dollars"
description: "I needed to harvest full catalogs from 25 sources and analyze every image — so I built a resumable overnight loop that runs on structured feeds and local models, for nothing."
date: 2026-07-06
tags: ["automation", "scraping", "cost-optimization", "ai", "founder-log"]
---

I woke up to 25 fully-harvested catalogs and color-analyzed images across all of them. The whole run cost me zero dollars in API fees, and if it had been interrupted at 3am it would have picked up exactly where it left off. Here's how I built an overnight harvest that's both free and un-babysittable.

## The Problem

Harvesting a lot of catalogs has two cost centers that quietly add up:

1. **The fetching** — if you drive a full headless browser against every page of every site, it's slow, fragile, and heavy.
2. **The analysis** — if you send every product image to a paid vision API, the bill scales linearly with your catalog and you're one runaway loop away from a nasty invoice.

And a job that runs for hours unattended has a third problem: if it dies partway, restarting from zero wastes everything it already did.

## The Fix

Three decisions made it cheap and reliable:

- **Feed-first fetching.** Most e-commerce sites expose their catalog as structured data if you know where to look — a products feed, a sitemap, an underlying data endpoint. Reading the structured feed instead of scraping the rendered page is faster, sturdier, and doesn't need a browser at all. When a feed exists, that's the door.
- **Local models for analysis.** Instead of a paid vision API, image color identification runs on a local model on my own hardware. It's not billed per call, so analyzing thousands of images costs the same as analyzing ten: nothing.
- **A resumable loop.** The job tracks what it's already done and checks that before doing anything. Interrupt it, restart it, run it again tomorrow — it does only the remaining work. That's what makes "leave it running overnight" actually safe.

## The Lesson

**Cost and reliability come from architecture, not effort.** The expensive, fragile version of this job and the free, robust version do the same thing — the difference is entirely in the choices: read the feed instead of the page, run the model locally instead of renting it, and make every step resumable so an interruption costs minutes, not a whole run. None of those are harder to build. They're just the ones you pick when you've been burned before.

## Why This Matters for Business Owners

"AI automation" doesn't have to mean a per-call meter running against you all night. A huge amount of real work — harvesting data, analyzing images, enriching records — can run on structured sources and local models for effectively nothing. Before you sign up for usage-based pricing on a job that scales with your data, ask whether it can run on a feed and a local model instead. Often it can, and the difference is a rounding error versus a real monthly bill.

## Let's Connect

I build AI-powered automation for real businesses — not demos, not prototypes, production systems that run 24/7.

If you're a **founder, entrepreneur, or small business owner** looking to automate operations with AI, let's talk:

- [**@agentabrams on YouTube**](https://youtube.com/@AgentAbrams) — walkthroughs and demos
- [**@agentabrams on X**](https://x.com/agentabrams) — DMs open
- [**@agentabrams on Bluesky**](https://bsky.app/profile/agentabrams.bsky.social) — follow along
- [**goodquestion.ai**](https://goodquestion.ai) — you're here

**Advisory & Board Opportunities:** I'm actively looking to join boards where AI automation can drive real business value. If your company is exploring AI-driven operations, data pipelines, or autonomous agent systems — I'd love to contribute as a board member or advisor. Reach out on any platform above.

---

*Built with [Claude Code](https://claude.ai). Shipped in production. Every day. This is what one founder + AI looks like.*