Rate limits
FairCrawl applies a Redis-backed 60-second counter window per org for the main crawl routes.
Current defaults
| Action | Limit per 60 seconds |
|---|---|
scrape |
1000 |
research |
200 |
map |
100 |
interact |
100 |
agent |
100 |
Every API key also has a 2,000 requests/minute ceiling across all FairCrawl routes (scrapes and job-status polling together). It sits above the per-org caps and only stops a runaway client.
What happens on limit hit
- the API returns
429 Retry-Afteris included in seconds- the current request is rejected before any billable work is dispatched
Capacity 429s
Separately from the per-org caps, FairCrawl sheds load when the shared scraping
pool is saturated. You get a 429 with Retry-After (and reason: "worker_saturated" in the problem details) instead of a request that waits and
times out. Nothing is billed. Retry after the indicated delay. Plain page scrapes
that do not need a browser keep working through these windows.
Backoff example
until curl https://api.faircompany.ai/v1/crawl/web/scrape \ -X POST \ -H "Authorization: Bearer $FAIRCRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com"}'; do sleep 5donefor (let attempt = 0; attempt < 5; attempt += 1) { try { return await fc.scrape({ url: "https://example.com" }); } catch (error) { const retryAfter = Number((error as any)?.details?.retryAfter || 0); await new Promise((resolve) => setTimeout(resolve, Math.max(retryAfter, 1) * 1000)); }}import timefrom faircrawl import FairCrawlfrom faircrawl.errors import FairCrawlError
fc = FairCrawl()
for _ in range(5): try: result = fc.scrape("https://example.com") break except FairCrawlError: time.sleep(5)Notes
- one-off org overrides exist in code for exceptional cases
- rate limiting is per org, not per API key
- read-only docs pages and the public product website are unaffected