How to Self-Host Firecrawl: A Practical Setup Guide

Firecrawl is the open-source web scraping and crawling engine behind many AI data pipelines — it turns entire websites into clean, LLM-ready Markdown or structured JSON. While the hosted version at firecrawl.dev is the fastest way to get started, running a Firecrawl self host deployment gives you full control over your data, no per-credit costs, and no rate limits imposed by a third party. If you're building RAG systems, training datasets, or internal search indexes, self-hosting is often the more economical and privacy-conscious route. This guide walks through what self-hosting actually involves, how to get it running with Docker, what works and what doesn't compared to the cloud version, and the production considerations most tutorials skip.

Why Self-Host Firecrawl Instead of Using the Cloud API

Firecrawl's hosted service charges per credit, where a credit roughly corresponds to a scraped page. For occasional use, that's convenient and cheap. But if you're crawling large sites repeatedly — say, refreshing documentation for a retrieval system every week — costs scale quickly, and you're sending potentially sensitive URLs and content through someone else's infrastructure.

Self-hosting changes the equation in three ways:

• No usage costs. You pay only for your own compute. Crawling a 50,000-page site costs nothing beyond server time and bandwidth.
• Data stays with you. For compliance-sensitive environments or internal corporate content, keeping scraped data inside your own network can be a hard requirement.
• Full customization. You can modify the codebase, adjust concurrency, or integrate your own post-processing directly.

The tradeoff is operational responsibility. The hosted API handles proxy rotation, anti-bot evasion, caching, and scaling automatically. On a self-hosted instance, you get the core scraping engine, but some of the hosted service's conveniences — notably advanced proxy and stealth features — are tied to the cloud offering. Understanding that boundary upfront prevents the most common disappointment: expecting cloud-grade anti-bot performance from a plain self-hosted deployment.

Prerequisites and Architecture Overview

A self-hosted Firecrawl instance is not a single container. The project is composed of several cooperating services, and knowing what each does makes troubleshooting far easier:

• The API service — the main Node.js/TypeScript application exposing the REST endpoints (/v1/scrape, /v1/crawl, and related routes).
• A worker process — consumes crawl jobs from the queue so long-running crawls don't block API responses.
• Redis — used for job queuing and concurrency control. This is mandatory, not optional.
• Playwright service — a browser automation container that renders JavaScript-heavy pages.
• PostgreSQL (optional) — used by hosted features like billing and usage tracking; many self-hosted setups can run without it, though recent versions of the stack may expect a DATABASE_URL to be present.

For hardware, plan on at least 4 GB of RAM for light use — Playwright's headless Chromium instances are the heaviest component. For serious crawling, 8 GB or more and multiple CPU cores will noticeably improve throughput since pages render in parallel.

You'll also need Docker and Docker Compose, since that's the officially supported deployment path. Git is required to clone the repository.

Step-by-Step: Deploying with Docker Compose

The quickest path is cloning the official repository and using the provided self-host configuration:

git clone https://github.com/mendableai/firecrawl.git
cd firecrawl

# Copy the example environment file
cp apps/api/.env.example apps/api/.env

Open the .env file and set the essential variables. At minimum, point the application at your Redis instance and set a test API key so you can authenticate locally:

# Core configuration for a minimal self-hosted setup
REDIS_URL=redis://redis:6379
TEST_API_KEY=your-local-key-here
HOST=0.0.0.0
PORT=3002

# Optional: disable features that depend on hosted infrastructure
USE_DB_AUTHENTICATION=false

Setting USE_DB_AUTHENTICATION=false is the key step most guides emphasize — it tells the API to skip Supabase-based authentication and billing checks, which are only relevant to the hosted service. Without it, the container will fail or reject requests because there's no database backing user accounts.

Then bring the stack up:

# From the repository root, using the self-hosted compose file
docker compose -f self-hosted/docker-compose.yaml up -d

Once all containers are running, verify the deployment:

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-local-key-here" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

A successful response returns JSON containing the page title, raw HTML, and a markdown field with the cleaned content. If the request hangs, the most likely culprit is the Playwright container failing to start — check docker compose logs for browser launch errors, which on constrained servers usually mean insufficient memory.

Using the Self-Hosted API from Your Code

Because the self-hosted instance exposes the same v1 API shape as the cloud, client integration is nearly identical — you just change the base URL. Here's a Python example using the official SDK:

from firecrawl import FirecrawlApp

app = FirecrawlApp(
    api_key="your-local-key-here",
    api_url="http://localhost:3002"
)

# Scrape a single page to Markdown
result = app.scrape_url(
    "https://docs.example.com/getting-started",
    formats=["markdown"]
)
print(result["markdown"][:500])

# Crawl an entire site with a limit
crawl = app.crawl_url(
    "https://docs.example.com",
    limit=100
)

The same works with plain HTTP from any language, which is convenient for shell scripts or lightweight integrations. One practical tip: for large crawls, prefer the asynchronous pattern — kick off the crawl job and poll for status rather than holding a connection open, since long-running synchronous requests are vulnerable to timeouts through reverse proxies like Nginx.

Self-Hosted vs. Hosted: What You Give Up

Being honest about the gaps is what separates a smooth deployment from a frustrating one. On a self-hosted instance, you should expect these differences:

• Anti-bot capabilities are limited. The hosted service uses proxy networks and stealth infrastructure that aren't part of the open-source stack. Sites with aggressive bot protection (Cloudflare challenges, heavy fingerprinting) may fail or return block pages. Self-hosting works best for sites you own, internal tools, documentation, and moderately protected public sites.
• No automatic caching layer. Repeated crawls hit the target site again. Build your own caching — storing Markdown output in your database or object storage — to avoid hammering the same domains.
• You manage scaling. A single worker processes jobs sequentially per queue; scale by running more worker containers if you need throughput.

Conversely, self-hosting wins decisively on cost at volume, data privacy, and the ability to run in air-gapped or VPC-restricted environments. A reasonable hybrid strategy: self-host for your own sites and internal content, and use the hosted API only for the occasional heavily protected external target.

Production Considerations and Common Mistakes

Before pointing a self-hosted instance at anything important, address these recurring issues:

• Respect robots.txt and rate limits. Firecrawl will crawl as fast as you let it. Add delays and concurrency limits in your crawl parameters, and be a good citizen — getting your server's IP blocked by a CDN affects everything else running from it.
• Set TEST_API_KEY even internally. An unauthenticated scraping endpoint on your network is an open proxy for anyone who can reach it. If the instance is exposed beyond localhost, put it behind authentication or a VPN.
• Monitor memory. Chromium instances can leak or accumulate under sustained load. A container restart policy plus periodic health checks will save you from 3 a.m. debugging sessions.
• Pin your image versions. The project evolves quickly; pinning Docker image tags and the git commit you deploy from makes upgrades deliberate rather than accidental.

A common mistake is treating the self-hosted deployment as a drop-in replacement for the cloud API in every scenario. Audit your target sites first: if most of them render fine with a basic headless browser, self-hosting will serve you well. If they're behind enterprise-grade bot protection, you'll save time using the hosted service for those specific sources.

Frequently Asked Questions

Conclusion

Self-hosting Firecrawl trades convenience for control: you give up managed anti-bot infrastructure and automatic scaling, but gain unlimited crawling at zero per-page cost and keep your data entirely in-house. For teams building RAG pipelines over their own documentation, internal knowledge bases, or moderately protected public sites, the Docker Compose deployment path is mature enough to run reliably with modest maintenance. Start small — deploy the stack locally, scrape a site you own, and validate the Markdown output quality against your pipeline's needs. If it holds up, move the deployment to a small VPS behind proper authentication, and you'll have a scraping engine that costs nothing per page and answers to no one but you.

---

Transparency Note: Links in this article are not affiliate links. I recommend these tools and platforms based on their relevance to the topic, and no compensation is received for including them.