Project 4: Big Data Analysis with PySpark on Weather Data

Project Objective

This project involves analyzing and extracting insights from the NCEI Global Surface Summary of Day dataset. The data will be processed using PySpark, Jupyter Notebook, and various data engineering and analytics techniques, providing hands-on experience with big data. Key tasks include handling and analyzing weather data from Cincinnati and Florida for years 2015-2024.

Requirements and Setup

1. Anaconda and PySpark Installation

   • Download and install Anaconda and Apache Spark 3.5.3.
   • Set up environment variables and install the PySpark library:

     pip install pyspark
     

   • Launch Jupyter Notebook from Anaconda.

2. Dataset Download

   • Download weather data for Cincinnati (72429793812) and Florida (99495199999) from the NCEI database for 2015-2024.
   • Organize datasets by year and location (e.g., 2015/72429793812.csv for Cincinnati 2015).

Analysis Tasks

1. Setup Verification

   • Provide screenshots showing Anaconda and PySpark installation and Jupyter Notebook launch.

2. Data Loading

   • Load each CSV file and display the record count for verification (20 datasets).

3. Hottest Day Analysis

   • Find the hottest day (MAX temperature) for each year, providing station code, name, date, and temperature.

4. Coldest Day in March

   • Identify the coldest day in March across all years with station code, name, date, and temperature.

5. Annual Precipitation Comparison

   • Determine the year with the most precipitation for Cincinnati and Florida, including station code, name, year, and mean precipitation.

6. Missing Value Percentage

   • Calculate the percentage of missing values in the wind gust (GUST) column for Cincinnati and Florida in 2024.