Awesome Data Temporality

A curated list to help you manage temporal data across many modalities 🚀.

Generative Art Created By DALL·E!

• Awesome Data Temporality
  • Data Versioning for Machine Learning
  • Time Travel and Temporal Tables
  • Slowly Changing Dimensions Data Modeling
  • Bi-temporality Tools + Modeling
  • Change Data Capture (CDC) Tools
  • Soft Delete in ORM Frameworks
  • Contribution

Data Versioning for Machine Learning

Data versioning is the practice of storing multiple versions of the same data and providing a mechanism for accessing and managing these versions. This can be useful in a variety of situations, such as when data is accidentally deleted or corrupted, or when it is necessary to see how the data has changed over time. The vast majority of "data versioning" tools you see today are related to better managing your datasets for machine learning. The implementation paradigm used is to store versions of your data and models in Git commits. Therefore the following part of the awesome list is centered around machine learning. However, there are other ways to manage your temporal data covered in later sections.

• DVC Management and versioning of datasets and machine learning models.
• lakeFS Repeatable, atomic and versioned data lake on top of object storage.
• Dolt Dolt is Git for Data!
• Pachyderm Pachyderm data versioning technology
• wrgl Data version control for data projects
• neptune Log, organize, compare, register, and share all your ML model metadata in a single place
• Git LFS An open source Git extension for versioning large files
• DagsHub Where people build data science projects
• Neptune Best 7 Data Version Control Tools
• DagsHub Comparing Data Version Control Tools
• Aporia Best Data Versioning Tools for MLOps
• A Conceptual Framework A Conceptual Framework and Proposed Principles
• Australian Research Data Commons What is data versioning?
• Research Data Alliance Principles and best practices in data versioning for all data sets big and small
• MLFlow + LakeFS Data Versioning for Efficient Workflows with MLFlow and LakeFS
• LakeFS Data Versioning: All You Need to Know

Time Travel and Temporal Tables

Data time travel refers to the ability to go back in time and access previous versions of data. In order to enable data time travel, it is necessary to implement a system for versioning data, which involves storing multiple versions of the same data and providing a mechanism for accessing and managing these versions. Whereas temporal tables, also known as system-versioned temporal tables, are tables in a database that automatically track the history of data changes and allow you to query the data as it existed at any point in time. Both time travel an temporal tables often are used interchangablely to mean the same thing. Temporal tables are more of an implementation specific feature of some databases. These tables are useful for auditing, tracking changes to data over time, and performing point-in-time analysis. You can usually query a temporal table using the FOR SYSTEM_TIME clause in a SELECT statement.

• Postgres Postgres Temporal Tables Extension
• Azure MSSQL temporal tables
• MariaDB MariaDB Temporal Tables
• Cockroach Cockroach documentation about AS OF SYSTEM TIME
• Dremio Dremio Iceberg Time Travel
• Iceberg Apache iceberg time travel
• Teradata Teradata Vantage™ Temporal Table Support
• SAP HANA Temporal Tables (History, System-Versionined, Application-Time Period)
• Delta Lake Introducing Delta Time Travel for Large Scale Data Lakes
• Hopsworks Time travel operations in Hopsworks Feature Store
• BigQuery Access historical data using time travel
• JOOQ System and application versioned tables
• Apache Flink Temporal Table Function
• Apache Arrow Voltron as-of join (not exactly bi-temporal, but is temporal joining)
• Redpanda Time travel debugging through historical messages to identify and debug problems

Slowly Changing Dimensions Data Modeling

Slowly changing dimensions are those in which the attributes of the dimension change over time, and the changes need to be tracked in the data warehouse. For example, a customer's address or name might change over time, and the data warehouse needs to track these changes so that historical data can be analyzed correctly.

• VDK Versatile Data Kit (VDK) is an open source framework including help to manage SCD style data.
• dbtvault A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
• dataform Common data models for creating type-2 slowly changing dimensions tables from mutable data sources in Dataform.
• dbt snapshots DBT snapshots
• DeltaLake Databricks change data capture with Delta Live Tables
• 6 Kinds 6 Different Types of Slowly Changing Dimensions and How to Apply Them?
• Data Vault Loading Dimensions from a Data Vault Model
• SCD Data Warehouse Slowly Changing Dimension Handling in Data Warehouses Using Temporal Database Features
• Redshift Implement a slowly changing dimension in Amazon Redshift

Bi-temporality Tools + Modeling

Bitemporality is a concept in database management that refers to the ability of a database to store and manage data that is associated with multiple time periods. This can include historical data as well as data that is still in the process of being entered or updated. In a bitemporal database, data is stored in multiple versions, with each version corresponding to a specific point in time. This allows users to view and query the data as it existed at different points in time, which can be useful