Loading repository data…

README preview

Data-centric AI (DCAI) is a development paradigm for ML-based solutions. The term was coined by Andrew Ng who gave the following definition:

Data-centric AI is the practice of systematically engineering the data used to build AI systems.

At Renumics, we believe DCAI is an important puzzle piece for building real-world AI systems that generate value. We like the following definition:

Data-centric AI means to improve training datasets systematically and iteratively by leveraging information from trained ML models.

Tools that can be efficiently used in day-to-day applications are the most important ingredient for the DCAI paradigm. This curated link collection is intended to help you discover useful open source tools for your data-centric AI workflows.

🔎 Scope

We include useful tools that have an open-source license and are actively maintained in this collection. All tools mentioned are useful for building DCAI workflows on unstructured data (e.g. images, audio, video, time-series, text).

We also collect workflow snippets into a data-centric AI playbook that show how typical tasks can be solved with open source tooling.

In order to keep a useful focus and to prevent duplicate work, we exclude some topics from this list such as tooling for tabular data, dedicated labeling tools, MLOps tooling as well as research papers. Please check out the further reading section to find awesome lists for these topics.

:open_hands: Contributing

Do you think something is missing? Please help contribute to this list by contacting us or adding a pull request.

🧰 Tooling

Data versioning

Logo	Name	Description	Popularity	License
	Data version control (DVC)	Data Version Control or DVC is a command line tool and VS Code Extension to help you develop reproducible machine learning projects.
	deeplake	Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets.
	Pachyderm	Pachyderm – Automate data transformations with data versioning and lineage.
	Delta Lake	An open-source storage framework that enables building a Lakehouse architecture.
	lakeFS	lakeFS is an open-source tool that transforms your object storage into a Git-like repository.

Embeddings and pre-trained models

Logo	Name	Description	Popularity	License
	towhee	Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
	Tensorflow Hub	TensorFlow Hub is a repository of reusable assets for machine learning with TensorFlow.
	Huggingface transformers	State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
	Lightly	Lightly is a computer vision framework for self-supervised learning.

Visualization and Interaction

Logo	Name	Description	Popularity	License
	Renumics Spotlight	Curation tool for unstructured data that connects your stack to the data-centric AI ecosystem.
	FiftyOne	The open-source tool for building high-quality datasets and computer vision models.
	refinery	The data scientist's open-source choice to scale, assess and maintain natural language data.
	Argilla	Argilla helps domain experts and data teams to build better NLP datasets in less time.
	Xtreme1	Xtreme1 is the world's first open-source platform for multisensory training data.
	YData Profiling	YData Profiling is a python package to perform Exploratory Data Analysis (EDA) for tabular and time-series data.

Outlier and noise detection

Logo	Name	Description	Popularity	License
	Cleanlab	Cleanlab facilitates machine learning with messy, real-world data by provid

awesome-open-data-centric-ai

Live repository statistics