REPOSITORY OVERVIEWLive repository statistics
★ 0Stars
⑂ 0Forks
◯ 0Open issues
◉ 0Watchers
27/100
OPENREPOHUB HEALTH SIGNALLimited signals
A transparent discovery signal based on current public GitHub metadata.
Recent activity35% weight
10 Community adoption25% weight
0 Maintenance state20% weight
100 License clarity10% weight
0 Project information10% weight
35 This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
README preview
Realtime Data Streaming | End-to-End Data Engineering Project
Table of Contents
Introduction
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline. It covers each stage from data ingestion to processing and finally to storage, utilizing a robust tech stack that includes Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. Everything is containerized using Docker for ease of deployment and scalability.
System Architecture
The project is designed with the following components:
- Data Source: We use
randomuser.me API to generate random user data for our pipeline.
- Apache Airflow: Responsible for orchestrating the pipeline and storing fetched data in a PostgreSQL database.
- Apache Kafka and Zookeeper: Used for streaming data from PostgreSQL to the processing engine.
- Control Center and Schema Registry: Helps in monitoring and schema management of our Kafka streams.
- Apache Spark: For data processing with its master and worker nodes.
- Cassandra: Where the processed data will be stored.
What You'll Learn
- Setting up a data pipeline with Apache Airflow
- Real-time data streaming with Apache Kafka
- Distributed synchronization with Apache Zookeeper
- Data processing techniques with Apache Spark
- Data storage solutions with Cassandra and PostgreSQL
- Containerizing your entire data engineering setup with Docker
Technologies
- Apache Airflow
- Python
- Apache Kafka
- Apache Zookeeper
- Apache Spark
- Cassandra
- PostgreSQL
- Docker
Getting Started
-
Clone the repository:
git clone https://github.com/airscholar/e2e-data-engineering.git
-
Navigate to the project directory:
cd e2e-data-engineering
-
Run Docker Compose to spin up the services:
docker-compose up
ALGORITHMICALLY RELATEDSimilar Open-Source Projects
Selected from shared topics, language and repository description—not editorial ratings.
The Voting System web application using Django is a project that serves as the automated voting system of an organization or school. This system works like the common manual system of election voting system whereas this system must be populated by the list of the positions, candidates, and voters. This system can help a certain organization or school to minimize the voting time duration because aside they can provide the voters an online platform to vote, the system will automatically count the votes for each candidate. The system has 2 sides of the user interface which are the administrator and voters side. The admin user is in charge to populate and manage the data of the system and the voter side which is where the voters will choose their candidate and submit their votes.
81/100 healthRecently updatedActive repository
PythonMIT#django#django-project#e-voting#otp
⑂ 86 forks◯ 3 issuesUpdated 4 days ago
This project serve HTML files (and a few more) saved in your computer with a UI suitable for Kindle web browser. On top of that, it include a Read Mode (thanks to ReadabiliPy) to display the text in a comfortable size without have to use the 'Article Mode' in Kindle web browser.
62/100 healthActive repository
PythonNo license#flask