Loading repository data…
Discover crawlers, extraction libraries, headless-browser tools and structured-data pipelines for the modern web.
firecrawl /
The API to search, scrape, and interact with the web at scale. 🔥
apify /
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
getmaxun /
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
apify /
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
alirezamika /
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
niespodd /
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
oxylabs /
Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.
geziyor /
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
vifreefly /
Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.
elixir-crawly /
Crawly, a high-level web crawling & scraping framework for Elixir.
scrapfly /
Scalable Python web scraping scripts for +40 popular domains
DataHenHQ /
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Repositories are ranked using live GitHub metadata, primarily stars unless another sort is selected. Stars are a popularity signal, not a quality guarantee.
Metadata is retrieved from the official GitHub API and may be cached briefly for reliability. Repository metrics can change at any time.