Skip to content

Rate limits

FairCrawl applies a Redis-backed 60-second counter window per org for the main crawl routes.

Current defaults

Action Limit per 60 seconds
scrape 1000
research 200
map 100
interact 100
agent 100

Every API key also has a 2,000 requests/minute ceiling across all FairCrawl routes (scrapes and job-status polling together). It sits above the per-org caps and only stops a runaway client.

What happens on limit hit

  • the API returns 429
  • Retry-After is included in seconds
  • the current request is rejected before any billable work is dispatched

Capacity 429s

Separately from the per-org caps, FairCrawl sheds load when the shared scraping pool is saturated. You get a 429 with Retry-After (and reason: "worker_saturated" in the problem details) instead of a request that waits and times out. Nothing is billed. Retry after the indicated delay. Plain page scrapes that do not need a browser keep working through these windows.

Backoff example

Terminal window
until curl https://api.faircompany.ai/v1/crawl/web/scrape \
-X POST \
-H "Authorization: Bearer $FAIRCRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'; do
sleep 5
done
for (let attempt = 0; attempt < 5; attempt += 1) {
try {
return await fc.scrape({ url: "https://example.com" });
} catch (error) {
const retryAfter = Number((error as any)?.details?.retryAfter || 0);
await new Promise((resolve) => setTimeout(resolve, Math.max(retryAfter, 1) * 1000));
}
}
import time
from faircrawl import FairCrawl
from faircrawl.errors import FairCrawlError
fc = FairCrawl()
for _ in range(5):
try:
result = fc.scrape("https://example.com")
break
except FairCrawlError:
time.sleep(5)

Notes

  • one-off org overrides exist in code for exceptional cases
  • rate limiting is per org, not per API key
  • read-only docs pages and the public product website are unaffected