Loading repository data…
dlt-hub / repository
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
🚀 Join our thriving community of likeminded developers and build the future together!
dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.
pip install dlt
Load chess game data from chess.com API and save it in DuckDB:
import dlt
from dlt.sources.helpers import requests
# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='duckdb',
dataset_name='player_data'
)
# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
response = requests.get(f'https://api.chess.com/pub/player/{player}')
response.raise_for_status()
data.append(response.json())
# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
Try it out in our Colab Demo or directly on our wasm-based playground in our docs.
dlt is an open-source Python library that loads data from various, often messy data sources into well-structured datasets. It provides lightweight Python interfaces to extract, load, inspect, and transform data. dlt and dlt docs are built from the ground up to be used with LLMs: the LLM-native workflow will take your pipeline code to data in a notebook for over 5000 sources.
dlt is designed to be easy to use, flexible, and scalable:
For detailed usage and configuration, please refer to the official documentation.
You can find examples for various use cases in the examples folder, or in the code examples section of our docs page.
dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.
major means breaking changes and removed deprecationsminor new features, sometimes automatic migrationspatch bug fixesWe suggest that you allow only patch level updates automatically using the Compatible Release Specifier. For example dlt~=1.23.0 allows only versions >=1.23.0 and less than <1.24.0
Please also see our release notes for notable changes between versions.
The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:
Please read CONTRIBUTING before you make a PR.
Blacksmith is a drop-in replacement for GitHub-hosted runners that speed up our CI/CD pipelines by 2x and up to 75% cheaper. We're grateful to Blacksmith for sponsoring us with free CI/CD minutes--which helps us keep builds fast and our costs lower.
dlt is released under the Apache 2.0 License.
Selected from shared topics, language and repository description—not editorial ratings.
aleksandarskrbic /
Kafka data generator and load testing tool - generate fake messages, simulate producers and consumers, test broker failures, and run chaos engineering scenarios
jayzern /
Sample ELT project using Dagster, data load tool and Snowflake
chayansraj /
This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.
leorickli /
ETL project using the Spotify API to get the daily top 50 global songs in the app. This project will use the AWS cloud tools to load the data into AWS Athena for analysis.
gabrielmotablima /
✨ The current project is a basic process pipeline for extraction, transformation, loading, analysis and presentation. All of this was done using appropriate web scraping, data analysis/presentation and database tools.
DivineSamOfficial /
This project showcases an end-to-end ELT (Extract, Load, Transform) pipeline leveraging the TPCH orders table from Snowflake's sample database. The primary goal is to demonstrate modern data engineering practices using Snowflake, dbt (Data Build Tool), and Apache Airflow.