🕸️ Amazon Web Scraper — Automated Product Data Extraction
Automatically extract product data from Amazon search results using Python, BeautifulSoup & Pandas — and export it as a clean CSV in seconds.
🌟 Project Overview
This project is a Python-based Amazon Web Scraper that visits Amazon's product listing pages, parses the HTML, and extracts structured product data — all saved neatly into a CSV file for further analysis or use.
✅ Automatic product data extraction
✅ Clean CSV export via Pandas
✅ Custom HTTP request handling with User-Agent
✅ BeautifulSoup-powered HTML parsing
✅ Jupyter Notebook support for interactive use
The system scrapes Amazon India's PlayStation 5 search results as the data source:
🔗 Amazon PS5 Search Results
🧠 What is Web Scraping?
Web scraping is a technique to automatically access and extract large amounts of information from websites — saving enormous time and effort compared to manual data collection.
Amazon Product Page → HTTP Request → HTML Parsing → Data Extraction → CSV Export
🚀 Key Features
📥 1. HTTP Request Handling
- Sends GET requests to Amazon search URLs
- Custom User-Agent header to mimic browser requests
- Handles response status and raw HTML retrieval
🧹 2. HTML Parsing with BeautifulSoup
Parses raw HTML to locate and extract:
| Element | Description |
|---|
| Product Title | Full product name |
| Price | Listed selling price |
| Rating | Average customer star rating |
| Reviews | Total number of user reviews |
| Availability | In stock / Out of stock status |
🔍 3. Structured Data Extraction
Uses custom extraction functions to:
- Locate product listing containers
- Parse individual data fields
- Handle missing or inconsistent data gracefully
💾 4. CSV Export with Pandas
Exports all extracted data into a structured DataFrame:
- Clean column headers
- One row per product listing
- Saved as
Amazon Data.csv for further use
📓 5. Jupyter Notebook Support
- Interactive step-by-step execution
- Visual output at each stage
- Easy customization of target URLs and fields
🏗️ System Architecture
Import Libraries
↓
Send HTTP GET Request to Amazon URL
↓
Parse Raw HTML using BeautifulSoup
↓
Extract Product Details via Custom Functions
↓
Store Data in Pandas DataFrame
↓
Export to Amazon Data.csv
📂 Project Structure
amazon-web-scraper/
├── scraper.py
├── Amazon Web Scraper.ipynb
└── Amazon Data.csv
📖 File Descriptions
| File | Purpose |
|---|
scraper.py | Raw/experimental scraping code |
Amazon Web Scraper.ipynb | Final cleaned Jupyter Notebook |
Amazon Data.csv | Output — extracted product dataset |
📊 Extracted Data Fields
The scraper extracts the following information for each product listing:
| # | Field | Description |
|---|
| 1 | 📝 Title | Full product name/title |
| 2 | 💰 Price | Listed selling price |
| 3 | ⭐ Rating | Average customer star rating |
| 4 | 💬 Reviews | Total number of user reviews |
| 5 | 🏷️ Availability | In stock / Out of stock status |
All fields are saved together into Amazon Data.csv as a structured Pandas DataFrame.
⚙️ Technologies Used
Backend
Parsing
Data Processing
Interface
🔄 Complete Application Flow
Open Jupyter Notebook / Run scraper.py
↓
Import Required Libraries
↓
Send HTTP Request with User-Agent
↓
Receive Raw HTML Response
↓
Parse HTML with BeautifulSoup
↓
Extract Product Fields
↓
Build Pandas DataFrame
↓
Export to CSV
🛠️ Installation Guide
1️⃣ Clone Repository
git clone https://github.com/your-username/amazon-web-scraper.git
cd amazon-web-scraper
2️⃣ Install Dependencies
| Requirement | Install Command |
|---|
| BeautifulSoup | pip install beautifulsoup4 |
| Requests | pip install requests |
| Pandas | pip install pandas |
pip install beautifulsoup4 requests pandas
3️⃣ Set Your User-Agent
🔑 Important: You'll need your browser's User-Agent string.
Just Google "my user agent" and copy the result — paste it into the script where required.
4️⃣ Run the Scraper
Via Python Script
python scraper.py
Via Jupyter Notebook
jupyter notebook "Amazon Web Scraper.ipynb"
Output saved to:
Amazon Data.csv
📈 Future Improvements
- Multi-page scraping support
- Additional product categories
- Automated scheduling
- Price drop alert system
- Cloud storage integration
- Data visualization dashboard
🎯 Project Summary
One-Line Description
"A Python-based web scraper that automatically extracts structured product data from Amazon search results and exports it as a clean, analysis-ready CSV using BeautifulSoup and Pandas."
👨💻 Author
Devansh Rai
IT & Full-Stack Development Enthusiast
Python | Web Scraping | Data Engineering
⭐ If You Like This Project
Give it a ⭐ on GitHub and support the project!