Loading repository data…
awaisajaz1 / repository
A comprehensive data engineering pipeline has been established to coordinate the ingestion, processing, and storage of data. This pipeline utilizes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and FastDBs. All these components have been containerized with Docker to facilitate straightforward deployment and scalability
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
This project gives out as a component guide for building an end-to-end data engineering pipeline for newbies. It encompasses each stage, from data ingestion and processing to storage, employing a robust tech stack that includes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and fast databases like PostgreSQL, Redshift, etc. All components are containerized using Docker for ease of deployment and scalability.
The project is designed with the following tools:
randomuser.me API to generate random user data for our pipeline.Selected from shared topics, language and repository description—not editorial ratings.
tranhuy25 /
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline. It covers each stage from data ingestion to processing and finally to storage, utilizing a robust tech stack that includes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra.
36/100 healthanshulspider /
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline. It covers each stage from data ingestion to processing and finally to storage, utilizing a robust tech stack that includes Apache Airflow, Python, Apache Kafka,
27/100 healthmanhzeff /
Using virtual environment
To create a virtual environment in the virtual_env folder:
python -m venv venv
To activate the virtual environment
.\venv\Scripts\activate
With the virtual environment activated we can install the packages in the requirements.txt file:
pip install -r requirements.txt
Clone the repository:
git clone git@github.com:awaisajaz1/air_savant.git
Navigate to the project directory:
cd air_savant
Run Docker Compose to spin up the services:
docker-compose up
List all containers:
docker container ls -a
If you face the error entrypoint file not found then the solution is changing the entrypoint script file from CRLF -> LF.
This initiative aims to facilitate newcomers and novices in grasping the integration of diverse data engineering tools and data processing methodologies. you can fork and get the starting point from here!
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline. It covers each stage from data ingestion to processing and finally to storage, utilizing a robust tech stack that includes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. Everything is containerized using Docker for
Rahul-Patel321 /
A comprehensive collection of Data Engineering System Design concepts, architectures, best practices, and interview-ready designs using AWS, Azure, Spark, Databricks, Airflow, Kafka, SQL, and Python.
73/100 healthAllan-Kipkemei /
his project serves as a comprehensive guide to building an end-to-end data engineering pipeline. It covers each stage from data ingestion to processing and finally to storage, utilizing a robust tech stack that includes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. Everything is containerized using Docker for
36/100 health