Firecrawl getting blocked due to headlesness

3 points

7 months ago

I’ve noticed that Firecrawl struggles with certain sites that do better with headful browsers. So we made Teracrawl (https://github.com/BrowserCash/teracrawl), which is an open source crawler built on a headful chrome browser API. (Browser.cash)

I tested a few URLs that are very JS-heavy and anti-bot-ish:

1. Yahoo Finance – Currencies https://finance.yahoo.com/currencies

Teracrawl: got the full currencies table (it actually ran the React app and hit the live pricing API).

Firecrawl: only got the header/nav shell, no Forex table at all. Was getting completely blocked by Yahoo.

2. AT&T wireless phones https://www.att.com/buy/wireless/phones

Firecrawl: just the AT&T header + basic chrome, no phones catalog.

Teracrawl: full phones list + pricing, filters, etc., since it executed the React frontend and fetched the JSON endpoints.

3. GoDaddy domain search https://www.godaddy.com/en-ca/domainsearch/find?…domainToCheck=mydomain.io

Firecrawl: static shell (nav + footer), zero search results. Again, getting blocked.

Teracrawl: full domain search output – availability, pricing, premium/upsell info, all the JS/API-loaded stuff.

Firecrawl seems fine for static content, but on JS-heavy pages or headful checks, it often just gives me skeleton HTML or is blocked.

Teracrawl, because it’s running real headful Chrome, behaves much more like an actual user browser and consistently returned dynamic content and was blocked less.

Firecrawl isn’t bad overall, but for certain sites using a headful browser wins out.

Firecrawl getting blocked due to headlesness

3 points

maclarens