ContentGrapher
ContentGrapher
← Changelog
Improved2026-05-28

Faster URL fetching via Web Unlocker

URL analysis now starts faster. The scraping phase completes in under 4 seconds on most pages, down from up to 15 seconds, with no change to how you submit URLs or read results.

Problem

URL analysis took longer than it should to start. The scraping phase — fetching the page before any analysis could begin — routinely consumed 10–15 seconds, leaving less time for the LLM pass and contributing to slow overall run times.

Context

The delay was a product mismatch. ContentGrapher was using Bright Data's Scraping Browser: a full Chromium browser running over a persistent connection, designed for interactive automation like clicking and form-filling. ContentGrapher never does any of that — it reads HTML. Every analysis was paying the cost of a browser it didn't need.

Why now

Bright Data's Web Unlocker product covers ContentGrapher's actual use case — fetch rendered HTML with WAF bypass and bot protection handling — without the browser overhead. The Firecrawl fallback was already in place, making the migration low-risk.

What changed

URL fetching now uses Bright Data Web Unlocker: a stateless HTTP request that returns rendered HTML directly, with no browser process or connection overhead. In testing, P95 fetch time dropped to 3.3 seconds across ten URLs including Cloudflare and Akamai-protected sites. The fallback to Firecrawl is unchanged.