Welcome to the official repository overview for Oxylabs' Web Scraper API. This guide is a developer-focused, technical reference covering what Web Scraper API is, how it works, its key features, supported targets, and when it fits your web scraping pipeline. It also serves as the central hub linking to all Web Scraper API-related repositories, code examples, and integrations.

What is Web Scraper API?

Web Scraper API is an all-in-one web data collection solution designed for extracting real-time data at scale from any public website. It combines proxies, uninterrupted access, crawling, and parsing into a single API, so you don't have to manage separate tools or maintain scraping infrastructure yourself.

The product covers every stage of web scraping – from sending requests and handling interferences with IPs and CAPTCHAs, to rendering JavaScript, parsing the response, and delivering the result to your preferred storage. You send a single API request specifying a target URL or query, and Web Scraper API returns either raw HTML or structured JSON ready for use.

Web Scraper API is built to meet enterprise standards, including SOC 2 Type II compliance, and runs on fast-adapting infrastructure maintained by Oxylabs' in-house engineering team. This makes it suitable for high-volume, production-grade data extraction across search engines, e-commerce marketplaces, travel platforms, real estate sites, and generic web pages.

Key Features

Web Scraper API is designed around three core capabilities: reliable access request handling, accurate structured data extraction, and flexible automation for custom workflows.

All-in-One Web Data Collection

Web Scraper API consolidates the entire scraping stack – proxy rotation, automated access management, CAPTCHA handling, JavaScript rendering, parsing, and delivery – behind a single API endpoint. There is no need to combine separate proxy providers, uninterrupted access layers, headless browser setups, and parsers. One request returns the data.

The built-in proxy rotator uses Oxylabs' ethically sourced proxy network, and IP interruptions plus CAPTCHA challenges are handled automatically as part of the request lifecycle.

Dedicated Scrapers and Parsers

In addition to a universal source that works with any public website, Web Scraper API ships with dedicated sources tuned for popular targets. These return ready-to-use structured JSON without you having to write or maintain parsing logic.

Dedicated sources include search engines (Google Search, Google Ads, Google Trends, Google Lens, Bing), AI platforms (ChatGPT, Perplexity, Google AI Mode), and major e-commerce marketplaces (Amazon, Walmart, eBay, Etsy, Best Buy, Target, Costco, AliExpress, Alibaba, Lazada, Flipkart, MercadoLibre, and many more). See the full list under our Supported Targets.

For our targets, you can pass either a full URL or parametrized inputs like a search query, product ID, or video ID, or set "parse": true to receive structured JSON instead of HTML.

JavaScript Rendering and Browser Instructions

For dynamic, JavaScript-heavy pages, Web Scraper API supports JavaScript rendering through its Custom Browser feature. Set the render parameter to html to retrieve the rendered HTML, or to png to receive a Base64-encoded screenshot.

When pages require user-like interaction – clicks, form input, scrolling, or waiting for elements to load – you can pass a list of browser instructions in the browser_instructions parameter. Example: input "pizza boxes" into an eBay search field, click the submit button, then wait for results.

{
  "source": "universal",
  "url": "https://www.ebay.com/",
  "render": "html",
  "browser_instructions": [
    { "type": "input", "value": "pizza boxes", "selector": { "type": "xpath", "value": "//input[@class='gh-tb ui-autocomplete-input']" } },
    { "type": "click", "selector": { "type": "xpath", "value": "//input[@type='submit']" } },
    { "type": "wait", "wait_time_s": 5 }
  ]
}

Browser instructions can be generated and tested without manual coding through the Web Scraper API Playground in the Oxylabs dashboard.

Geolocation and Localization

The geo_location parameter lets you choose the proxy server location, returning data as if you were browsing from that country, state, or city. Its exact behavior differs by source category:

• For SERP sources, geo_location adjusts the search results page to reflect what users in that location would see.
• For E-commerce sources, specific values are required by certain marketplaces – check the per-target documentation.

Additional domain, locale, and results_language parameters customize the top-level domain, interface language, and result language for region-specific extraction.

Custom Parser

Custom Parser is a free feature that lets you define your own parsing logic on top of a raw scraping result, returning only the fields you care about in a structured format. It is configurable through the API or the dashboard, and Parser Presets let you save, manage, and reuse parsing configurations across jobs.

See the Custom Parser and Parser Presets documentation for details.

Scheduler

Scheduler is a complimentary feature that lets you automate recurring scraping jobs. Instead of issuing identical requests on a timer from your side, you submit a schedule once – with a cron expression, target parameters, an optional end time, and optional callback or cloud storage URLs – and Web Scraper API runs the jobs on the defined cadence.

Example payload running every hour until a specified end date:

{
  "cron": "0 * * * *",
  "items": [
    {
      "source": "universal",
      "url": "https://example.com",
      "parse": true,
      "render": "html"
    }
  ],
  "end_time": "2026-12-01 00:00:00"
}

POST to https://data.oxylabs.io/v1/schedules. See the Scheduler documentation for the full parameter list.

OxyCopilot

OxyCopilot is an AI-powered assistant inside the Scraper API Playground that generates scraping requests and parsing instructions from plain-English prompts. Describe the page you want to scrape and the fields you need, and OxyCopilot produces a ready-to-use API request along with a parsing template, including for nested or listed data. It is intended to reduce setup time for new targets and adapt quickly to layout changes.

How Does Web Scraper API Work?

You send an authenticated HTTP request specifying what to scrape (URL or query), which source to use, and any optional parameters (geolocation, rendering, parsing, callback, storage). Web Scraper API handles proxy rotation, retries, automated request management, JavaScript rendering, and parsing as part of the request lifecycle, then returns the result either synchronously or via callback/storage.

Integration Methods

Web Scraper API offers three integration methods, chosen per request:

Push-Pull supports batch requests – up to 5,000 query or url values in one POST to https://data.oxylabs.io/v1/queries/batch. Each entry is treated as a separate job and can be returned via callback or written directly to AWS S3 or Google Cloud Storage. Realtime and Proxy Endpoint do not support batch.

Authentication for all methods is HTTP Basic Auth with your dashboard USERNAME and PASSWORD.

Request Example

Realtime request against the universal source:

curl 'https://realtime.oxylabs.io/v1/queries' \
  --user "USERNAME:PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "universal",
    "url": "https://sandbox.oxylabs.io/",
    "geo_location": "United States"
  }'

Realtime request against a dedicated source returning parsed JSON:

curl 'https://realtime.oxylabs.io/v1/queries' \
  --user "USERNAME:PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
  }'

Push-Pull request with delivery to S3:

curl 'https://data.oxylabs.io/v1/queries' \
  --user "USERNAME:PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true,
    "callback_url": "https://your.callback.url",
    "storage_type": "s3",
    "storage_url": "s3://your.storage.bucket.url"
  }'

Each response includes a job_id (also returned in the x-oxylabs-job-id header), which is the identifier used for troubleshooting and result retrieval.

Supported Targets

Web Scraper API ships with dedicated source values for many high-priority targets, plus a universal source that w