Loading repository data…
vigviswa / repository
This project implements an ETL Pipeline using Python and schedules tasks with Apache Airflow
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
In this challenge, we will develop a simple Apache Airflow data pipeline.
Our data pipeline fetches data from News API, transforms the data into a tabular structure, and stores the transformed data on Amazon S3.
make init to download project dependencies.make test to make sure basic smoke tests are passing.make run with docker running to bring up airflow.
sample_dag. You should be able to view logs from the Task Instance Context Menu.<pipeline_execution_date>_top_headlines.csv<s3_bucket>/<source_name>Selected from shared topics, language and repository description—not editorial ratings.
caio-moliveira /
This repository implements a fully automated data pipeline integrating AWS S3, Snowflake, DBT, Apache Airflow, and Streamlit. It handles data ingestion, transformation, and visualization, providing a streamlined solution for building and analyzing datasets.
31/100 healthleeeboy /
This project implements an Airflow ETL pipeline that fetches finance news data from the Alpha Vantage API, transforms it, stores it in an Excel file, and uploads it to an AWS S3 bucket.
38/100 healthFrom the Apache Airflow documentation:
Airflow is a platform to programmatically author, schedule and monitor workflows.
Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.
In order to facilitate the use of Airflow, we have included a Dockerfile and a docker-compose.yml that can be used to set up a local airflow development environment. Make sure to have Docker and Docker Compose installed.
From the root folder, you can execute the following command to run airflow:
docker-compose up --build
The Airflow UI/Admin Console should now be visible on http://localhost:8080.
In order to build the data pipeline, it will be necessary to create a DAG. We have provided an example DAG, dags/sample_dag.py, that can be used as a reference. Further documentation can be found in the airflow tutorial and the airflow concepts pages.
To load a new DAG into airflow, simply create a new Python file in the dags folder that contains an airflow DAG object.
To install additional Python packages (boto3, pandas, requests, etc.), add them to requirements.txt.
A simple REST API that can be used to retrieve breaking headlines and search for articles. A free News API account is required to obtain an API key.
| Route | Description |
|---|---|
| /v2/top-headlines | Returns live top and breaking headlines for a country, specific category in a country, single source, or multiple sources. |
| /v2/sources | Returns the subset of news publishers that top headlines are available from. |
A simple cloud storage service run by Amazon Web Services (AWS). An AWS account is needed to use AWS S3. Furthermore, AWS has a free tier that can be used for this challenge.
Amazon provides a Python SDK (boto), that provides an easy to use API for interacting with AWS S3.
gargpuneet438 /
This project implements a data pipeline using Apache Airflow to extract data from the Twitter API, process it using pandas, and load it into an AWS S3 bucket in CSV format.
30/100 healthwlopezm-unal /
This project focuses on implementing an ETL pipeline using Apache Airflow to efficiently extract data from Reddit, transform it as needed, and load it into an AWS S3 bucket. The use of Airflow allows for robust orchestration of the data workflow, ensuring that each step of the ETL process is executed in a reliable and repeatable manner.
31/100 health