`src/content/posts/2026-02-25-parallel-agents-at-scale.md`

100 lines

---
title: "Parallel Agents at Scale: Growing a Catalog 25% in One Session"
description: "I dispatched 6 AI agents in parallel to build 15 data scrapers, fix 5 broken ones, and grow a product catalog by 25% in a single afternoon."
date: 2026-02-25
tags: ["automation", "ai", "scaling", "infrastructure"]
---

Sometimes you just have to send it.

## Watch the Video

[**Subscribe to @AgentAbrams on YouTube**](https://youtube.com/@AgentAbrams) for new videos every week.

![hero](/images/heroes/parallel-agents-at-scale.png)

## The Problem

I had a database with tens of thousands of catalog records spread across roughly 100 tables. But 26 of those tables were sitting at zero. Empty. Each one represented a different data source that needed its own custom scraper -- different websites, different APIs, different authentication flows.

Building them one at a time? That is a week of work. I had a few hours.

## The Approach: Run Everything at Once

The idea was simple: **dispatch 6 independent builder agents simultaneously**, each assigned a batch of sources to handle. Each agent:

1. Reads the existing scraper pattern from a reference implementation
2. Investigates the target site -- is it an e-commerce platform? A custom CMS? A REST API?
3. Writes a complete scraper following the established conventions
4. Runs it and reports results

The key insight: these tasks are embarrassingly parallel. One agent building an e-commerce scraper does not block another agent building an API connector. So run them all at once.

## What the Agents Discovered

The fun part is discovery. Every data source is a puzzle:

- **One site** turned out to be a completely different platform than expected. The agent pivoted to parsing sitemap XML instead.
- **Another** was a hosted e-commerce store masquerading as a custom website. Quick pivot to the standard product API.
- **Two sites** had their domains expire mid-project. One redirected to a parked page. Gone.
- **One brand** went bankrupt years ago. The agent bravely attempted three different strategies before confirming zero products.
- **A European manufacturer** had an incomplete SSL certificate chain, blocking all connections until the agent worked around it.
- **Another European brand** served thousands of products via API, but the initial data fetch was so slow it kept timing out at 10 minutes. Fix: extend the timeout to 15.

## Fixing What Was Already Broken

Three existing scrapers were completely non-functional -- they used an older browser automation library that hung indefinitely on sites with chatbots and analytics scripts. The fix was straightforward:

1. Switch to a modern browser automation tool
2. Change the page-load strategy to something less aggressive
3. Add automatic browser restarts every 15 pages to prevent memory buildup

One of those scrapers went from **0 products** to **734** after the conversion.

## The Results

| Metric | Before | After |
|--------|--------|-------|
| Total catalog records | Tens of thousands | **Over 100K** |
| Populated tables | ~74 | **91** |
| Empty tables | 26 | **9** |
| New scrapers built | -- | **15+** |
| Broken scrapers fixed | -- | **5** |
| Session duration | -- | **~4 hours** |

A **25% increase** in one session. The 9 remaining empty tables? Mostly dead domains, bankrupt brands, or luxury houses that do not publish product catalogs publicly.

## Lessons For Any Founder Scaling Data Operations

1. **Parallel beats sequential every time** when tasks are independent. Six agents finishing in 30 minutes beats one agent finishing in 3 hours.

2. **Every data source is different.** You cannot assume anything about how a website serves its product data. Build flexible agents that can adapt.

3. **Timeouts lie.** A scraper that "timed out" may have already saved its data. Always check your database before re-running something you think failed.

4. **Old tools rot.** Browser automation tools that worked last year may hang on today's websites. Budget time for migrations.

5. **The biggest unlock is not the AI -- it is the parallelism.** Running agents in parallel is a multiplier on everything else you have built.

## What Comes Next

- 29 background jobs now schedule monthly refreshes for every scraper automatically
- A command center dashboard shows all sources, record counts, and last-crawl dates in real time
- The remaining empty tables are being investigated one by one

If you are building data pipelines, the question is not whether to automate -- it is how many things you can automate at the same time.

## Follow Along

- [**@agentabrams on YouTube**](https://youtube.com/@AgentAbrams) -- subscribe for walkthroughs
- [**@agentabrams on X**](https://x.com/agentabrams) -- DMs open
- [**@agentabrams on Bluesky**](https://bsky.app/profile/agentabrams.bsky.social) -- follow along
- [**goodquestion.ai**](https://goodquestion.ai) -- you are here