How To Scrape Google Jobs

In this two-part tutorial, we're going to show how to scrape Google Jobs data. First, we'll show how to do it for free but if you need data at scale, please refer to the second part of the tutorial. There, we'll demonstrate how to gather large-scale data with Oxylabs API.

• Free Google Jobs Scraper
  • Prerequisites
  • Installation
  • Scraping Google Jobs
  • Notes
• Scraping Google Jobs with Oxylabs API
• Google Jobs website overview
• 1. Get a free trial and send a request
  • Install Python
  • Send a request for testing
• 2. Install and import libraries
• 3. Add your API user credentials
• 4. Set up queries and locations
  • URL parameters
  • Locations
• 5. Prepare the API payload with parsing instructions
  • Job title
  • Company name
  • Location
  • Date
  • Salary
  • Job posted via
  • URL
• 6. Define functions
  • Submit job
  • Check job status
  • Get job results
  • Save data to a CSV file
  • Scrape Google Jobs
• 7. Create the main() function
• 8. Run the complete code

Free Google Jobs Scraper

A free tool used to get data about jobs from Google Jobs for a provided search query.

Prerequisites

To run this tool, you need to have Python 3.11 installed in your system.

Installation

Open up a terminal window, navigate to this repository and run this command:

make install

Scraping Google Jobs

To scrape jobs from Google Jobs, simply run this command in your terminal with a search query for a job that you need data for:

make scrape QUERY="<job_query>"

For this example, let's try scraping for designer jobs. The command should look something like this:

make scrape QUERY="designer"

Make sure to enclose your query in quotation marks, otherwise the tool might have trouble parsing it.

After running the command, your terminal should look something like this:

After the tool has finished running, you should see a file named jobs.csv in your current directory.

The CSV file contains available jobs in your area for the position you entered in the query. The jobs are listed with these attributes:

• title - The title of position.
• company - The name of the company.
• location - The location of the position.
• url - The URL of the Google Jobs page for that job.

Here's an example of how the data can look like:

Notes

In case the code doesn't work or your project is of bigger scale, please refer to the second part of the tutorial. There, we showcase how to scrape public data with Oxylabs API.

Scraping Google Jobs with Oxylabs API

Google Jobs website overview

Once you visit the Google Jobs page, you'll see that all job listings for a query are displayed on the left side. Looking at the HTML structure, you can see that each listing is enclosed in the <li> tag and collectively wrapped within the <ul> tag:

In this guide, let’s scrape Google Jobs results asynchronously and extract the following publicly available data:

1. Job title
2. Company name
3. Job location
4. Job posted via [platform]
5. Job listing date
6. Salary

If you want to extract even more public data, such as job highlights, job description, and similar jobs, expand the code shown in this article to make additional API calls to the scraped job URLs.

1. Get a free trial and send a request

Visit the Oxylabs dashboard and create an account to claim your 1-week free trial for Google Jobs API, part of the Oxylabs Web Scraper API. It’s equipped with proxy servers, Custom Browser Instructions, Custom Parser, and other advanced features that’ll help you overcome blocks and fingerprinting. See this short guide that shows how to navigate the dashboard and get the free trial.

Install Python

If you don’t have Python installed yet, you can download it from the official Python website. This tutorial is written with Python 3.12.0, so ensure that you have a compatible version.

Send a request for testing

After creating an API user, copy and save your API user credentials, which you’ll use for authentication. Next, open your terminal and install the requests library:

pip install requests

Then run the following code that scrapes Google Jobs results and retrieves the entire HTML file:

import requests

payload = {
    "source": "google",
    "url": "https://www.google.com/search?q=developer&ibp=htl;jobs&hl=en&gl=us",
    "render": "html"
}

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=("USERNAME", "PASSWORD"),	# Replace with your API user credentials
    json=payload
)
print(response.json())
print(response.status_code)

Once it finishes running, you should see a JSON response with HTML results and a status code of your request. If everything works correctly, the status code should be 200.

2. Install and import libraries

For this project, let’s use the asyncio and aiohttp libraries to make asynchronous requests to the API. Additionally, the json and pandas libraries will help you deal with JSON and CSV files.

Open your terminal and run the following command to install the necessary libraries:

pip install asyncio aiohttp pandas

Then, import them into your Python file:

import asyncio, aiohttp, json, pandas as pd
from aiohttp import ClientSession, BasicAuth

3. Add your API user credentials

Create the API user credentials variable and use BasicAuth, as aiohttp requires this for authentication:

credentials = BasicAuth("USERNAME", "PASSWORD") # Replace with your API user credentials

4. Set up queries and locations

You can easily form Google Jobs URLs for different queries by manipulating the q= parameter:

https://www.google.com/search?q=developer&ibp=htl;jobs&hl=en&gl=us

This enables you to scrape job listings for as many search queries as you want.

Note that the q=, ibp=htl;jobs, hl=, and gl= parameters are mandatory for the URL to work.

Additionally, you could set the UULE parameter for geo-location targeting yourself, but that’s unnecessary since the geo_location parameter of Google Jobs Scraper API does that by default.

URL parameters

Create the URL_parameters list to store your search queries:

URL_parameters = ["developer", "chef", "manager"]

Locations

Then, create the locations dictionary where the key refers to the country, and the value is a list of geo-location parameters. This dictionary will be used to dynamically form the API payload and localize Google Jobs results for the specified location. The two-letter country code will be used to modify the gl= parameter in the Google Jobs URL:

locations = {
    "US": ["California,United States", "Virginia,United States", "New York,United States"],
    "GB": ["United Kingdom"],
    "DE": ["Germany"]
}

Visit our documentation for more details about geo-locations.

5. Prepare the API payload with parsing instructions

Google Jobs Scraper API takes web scraping instructions from a payload dictionary, making it the most important configuration to fine-tune. The url and geo_location keys are set to None, as the scraper will pass these values dynamically for each search query and location. The "render": "html" parameter enables JavaScript rendering and returns the rendered HTML file:

payload = {
    "source": "google",
    "url": None,
    "geo_location": None,
    "user_agent_type": "desktop",
    "render": "html"
}

Next, use Custom Parser to define your own parsing logic with xPath or CSS selectors and retrieve only the data you need. Remember that you can create as many functions as you want and extract even more data points than shown in this guide. Head to this Google Jobs URL in your browser and open Developer Tools by pressing Ctrl+Shift+I (Windows) or Option + Command + I (macOS). Use Ctrl+F or Command+F to open a search bar and test selector expressions.

As mentioned previously, the job listings are within the tags, which are wrapped with the tag.

As there is more than one <ul> list on the Google Jobs page, you can form an xPath selector by specifying the div element that contains the targeted list.

//div[@class='nJXhWc']//ul/li

You can use this selector to specify the location of all job listings in the HTML file. In the payload dictionary, set the parse key to True and create the parsing_instructions parameter with the jobs function:

payload = {
    "source": "google",
    "url": None,
    "geo_location": None,
    "user_agent_type": "desktop",
    "render": "html",
    "parse": True,
    "parsing_instructions": {
        "jobs": {
            "_fns": [
                {
                    "_fn": "xpath",
                    "_args": ["//div[@class='nJXhWc']//ul/li"]
                }
            ],
        }
    }
}

Next, create the _items iterator that will loop over the jobs list and extract details for each listing:

payload = {
    "source": "google",
    "url": None,
    "geo_location": None,
    "user_agent_type": "desktop",
    "render": "html",
    "parse": True,
    "parsing_instructions": {
        "jobs": {
            "_fns": [
                {
                    "_fn": "xpath", # You can use CSS or xPath
                    "_args": ["//div[@class='nJXhWc']//ul/li"]
                }
            ],
            "_items": {
                "data_point_1": {
                    "_fns": [
                        {
                            "_fn": "selector_type",  # You can use CSS or xPath
                            "_args": ["selector"]
                        }
                    ]
                },
                "data_point_2": {
                    "_fns": [
                        {
                            "_fn": "selector_type",
                            "_args": ["selector"]
                        }
                    ]
                },
            }
        }
    }
}

For each data point, you can create a