🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

---

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early access
We’ll be onboarding in phases and working closely with early users. Limited slots.

---

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.1

✨ New in v0.9.1: Patch release with 12 bug fixes across Docker, browser, and core. Adds preserve_classes/preserve_tags whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

• LLM ready output, smart Markdown with headings, tables, code, citation hints
• Fast in practice, async browser pool, caching, minimal hops
• Full control, sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence, learns site patterns, explores only what matters
• Deploy anywhere, zero keys, CLI and Docker, cloud friendly

🚀 Quick Start

1. Install Crawl4AI:

# Install the package
pip install -U crawl4ai

# For pre release versions
pip install crawl4ai --pre

# Run post-installation setup
crawl4ai-setup

# Verify your installation
crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually:

python -m playwright install --with-deps chromium

2. Run a simple web crawl with Python:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

3. Or use the new command-line interface:

# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown

# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"

💖 Support Crawl4AI

🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.

Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.

🤝 Sponsorship Tiers

• 🌱 Believer ($5/mo) — Join the movement for data democratization
• 🚀 Builder ($50/mo) — Priority support & early access to features
• 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
• 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
  Custom arrangements available - see SPONSORS.md for details & contact

Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.

See All Tiers & Benefits →

✨ Features

• 🧹 Clean Markdown: Generates clean, structured Markdown with accurate formatting.
• 🎯 Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
• 🔗 Citations and References: Converts page links into a numbered reference list with clean citations.
• 🛠️ Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
• 📚 BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.

• 🤖 LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
• 🧱 Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
• 🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
• 🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
• 🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.

• 🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
• 🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
• 👤 Browser Profiler: Create and manage persistent profiles with saved authentication states, cookies, and settings.
• 🔒 Session Management: Preserve browser states and reuse them for multi-step crawling.
• 🧩 Proxy Support: Seamlessly connect to proxies with authentication for secure access.
• ⚙️ Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
• 🌍 Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
• 📐 Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

• 🖼️ Media Support: Extract images, audio, videos, and responsive image formats like srcset and picture.
• 🚀 Dynamic Crawling: Execute JS and wait for async or sync for dynamic content extraction.
• 📸 Screenshots: Capture page screenshots during crawling for debugging or analysis.
• 📂 Raw Data Crawling: Directly process raw HTML (raw:) or local files (file://).
• 🔗 Comprehensive Link Extraction: Extracts internal, external links, and embedded iframe content.
• 🛠️ Customizable Hooks: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).
• 💾 Caching: Cache data for improved speed and to avoid redundant fetches.
• 📄 Metadata Extraction: Retrieve structured metadata from web pages.
• 📡 IFrame Content Extraction: Seamless extraction from embedded iframe content.
• 🕵️ Lazy Load Handling: Waits for images to fully load, ensuring no content is missed due to lazy loading.
• 🔄 Full-Page Scanning: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

• 🐳 Dockerized Setup: Optimized Docker image with FastAPI server for easy deployment.
• 🔑 Secure Authentication: Built-in JWT token authentication for API security.
• 🔄 API Gateway: One-click deployment with secure token authentication for API-based workflows.
• 🌐 Scalable Architecture: Designed for mass-scale production and optimized server performance.
• ☁️ Cloud Deployment: Ready-to-deploy configurations for major cloud platforms.

• 🕶️ Stealth Mode: Avoid bot detection by mimicking real users.
• 🏷️ Tag-Based Content Extraction: Refine crawling based on custom tags, headers, or metadata.
• 🔗 Link Analysis: Extract and analyze all links for detailed data exploration.
• 🛡️ Error Handling: Robust error management for seamless execution.
• 🔐 CORS & Static Serving: Supports filesystem-ba