🔥 Apache Spark RDD Workshop

A comprehensive hands-on workshop for learning Apache Spark RDD (Resilient Distributed Dataset) operations through practical exercises. This project contains three complete exercises demonstrating core Spark concepts including data filtering, aggregation, partitioning, and distributed word counting.

📋 Table of Contents

• Overview
• Features
• Prerequisites
• Installation
• Project Structure
• Exercises
• Usage
• Technologies
• Results
• Contributing
• License

🎯 Overview

This workshop provides hands-on experience with Apache Spark's foundational data structure: RDDs. Through three progressive exercises, you'll learn:

1. Log Analysis: Filter and count ERROR entries in log files
2. Sales Analytics: Aggregate purchase data across multiple stores with partitioning
3. Text Processing: Perform distributed word counting with RDD joins and caching

Perfect for data engineering students, big data enthusiasts, or anyone looking to master Spark fundamentals.

✨ Features

• Exercise 1: Error Log Analysis

  • RDD creation from text files
  • Data filtering with lambda functions
  • Action operations (count, take)

• Exercise 2: Sales Data Processing

  • Tab-separated value (TSV) parsing
  • Dynamic repartitioning
  • Map-Reduce aggregation
  • Group-by operations

• Exercise 3: Advanced Word Count

  • Multi-file RDD processing
  • Full outer joins
  • RDD caching for performance
  • Regular expression text cleaning
  • Sorted result generation

🔧 Prerequisites

• Python 3.7 or higher
• Java JDK 8 or 17 (required for Spark)
• Jupyter Notebook or JupyterLab
• Basic understanding of Python and distributed computing concepts

📦 Installation

1. Clone the Repository

git clone https://github.com/Dansoko22md/spark-rdd-workshop.git
cd spark-rdd-workshop

2. Set Up Python Environment

# Create virtual environment
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Install Apache Spark

On Windows:

# Download Spark from https://spark.apache.org/downloads.html
# Extract to C:\spark
# Set environment variables as shown in the notebook

On macOS/Linux:

# Using Homebrew (macOS)
brew install apache-spark

# Or download manually
wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark

4. Configure Java

Ensure JAVA_HOME is set:

# Windows
setx JAVA_HOME "C:\Program Files\Java\jdk-17"

# macOS/Linux
export JAVA_HOME=$(/usr/libexec/java_home)  # macOS
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk  # Linux

📁 Project Structure

spark-rdd-workshop/
│
├── atelier1.ipynb              # Main workshop notebook
├── requirements.txt            # Python dependencies
├── README.md                   # This file
├── LICENSE                     # MIT License
│
├── data/                       # Data files
│   ├── error_log.txt          # Log file for Exercise 1
│   ├── purchases.txt          # Sales data for Exercise 2
│   ├── README.md              # Spark documentation
│   └── CHANGES.txt            # Spark changelog
│
├── results/                    # Output directory
│   └── .gitkeep
│
└── screenshots/               # Visual documentation
    └── .gitkeep

📚 Exercises

Exercise 1: Error Log Analysis

Objective: Filter and analyze error entries in server log files.

Key Concepts:

• Creating RDDs from text files
• Filtering operations
• Count and collect actions

Code Snippet:

errors_rdd = rdd.filter(lambda line: "ERROR" in line)
error_count = errors_rdd.count()

Expected Output:

Nombre de lignes contenant 'ERROR' : 13

---

Exercise 2: Sales Data Aggregation

Objective: Calculate total sales per store from transaction data.

Key Concepts:

• RDD partitioning
• Map-Reduce pattern
• Key-value pair operations
• Aggregation with reduceByKey

Code Snippet:

total_par_magasin = rdd.map(lambda line: line.split('\t')) \
                       .map(lambda parts: (parts[2], float(parts[4]))) \
                       .reduceByKey(lambda a, b: a + b)

Sample Output:

Albuquerque: 3028079.62
Anaheim: 3085295.94
Cleveland: 3121222.68
Total général: 313270384.87

---

Exercise 3: Distributed Word Count with Joins

Objective: Count word frequencies across multiple documents using RDD joins.

Key Concepts:

• Multiple RDD processing
• Text tokenization with regex
• Full outer joins
• RDD caching
• Sorting and ranking

Code Snippet:

rdd_joint = word_count_readme.fullOuterJoin(word_count_changes)
rdd_joint.cache()
rdd_final = rdd_joint.mapValues(combiner_counts)

Top Results:

1. spark: 3918 occurrences
2. com: 3878 occurrences
3. pull: 2833 occurrences

🚀 Usage

Running in Jupyter Notebook

# Start Jupyter
jupyter notebook

# Open atelier1.ipynb
# Run cells sequentially

Running in Google Colab

1. Upload the notebook to Google Colab
2. Upload data files to Colab session
3. Install PySpark: !pip install pyspark
4. Run cells sequentially

Running as Python Script

# Convert notebook to script
jupyter nbconvert --to script atelier1.ipynb

# Run the script
python atelier1.py

🛠️ Technologies

• Apache Spark 3.5+: Distributed computing framework
• PySpark: Python API for Spark
• Jupyter Notebook: Interactive development environment
• Python 3.7+: Programming language
• Pandas: Data manipulation (optional)
• Matplotlib/Seaborn: Visualization (optional)

📊 Results

Performance Metrics

Exercise	Dataset Size	Processing Time	RDD Partitions
Exercise 1	~100 lines	< 1 second	2
Exercise 2	4,139 records	~2 seconds	4
Exercise 3	14,718 lines	~3 seconds	Default

Key Findings

• Exercise 2: Total sales across 107 stores = $313,270,384.87
• Exercise 3: 4,090 unique words processed from documentation files
• Caching Impact: ~30% performance improvement on repeated operations

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Guidelines

• Follow PEP 8 style guide for Python code
• Add comments for complex operations
• Update documentation for new features
• Include test cases when applicable

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

👥 Authors

• Moussa Dansoko - Initial work - Dansoko22md

🙏 Acknowledgments

• Apache Spark community for excellent documentation
• Databricks for educational resources
• All contributors and workshop participants

📞 Contact

• GitHub: @Dansoko22md
• Email: dansokomoussa42@gmail.com
• LinkedIn: Moussa Dansoko

🔗 Useful Links

• Apache Spark Documentation
• PySpark API Reference
• RDD Programming Guide
• Spark Examples

📈 Roadmap

• Add DataFrame API exercises
• Include Spark SQL examples
• Add streaming data processing
• Create visualization dashboard
• Add performance tuning section
• Include MLlib machine learning examples

---

⭐ Star this repository if you find it helpful! ⭐

Happy Spark Learning! 🎓