database-lab-engine

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

#branching#ci#ci-cd#cicd#cloning#database

View Repository on GitHub ↗

REPOSITORY OVERVIEW

Live repository statistics

★ 2.5KStars

⑂ 72Forks

◯ 10Open issues

◉ 2.5KWatchers

91/100

OPENREPOHUB HEALTH SIGNAL

Strong signals

A transparent discovery signal based on current public GitHub metadata.

Recent activity35% weight

100

Community adoption25% weight

Maintenance state20% weight

100

License clarity10% weight

100

Project information10% weight

100

This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.

README preview

*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.

Why DBLab?

Build dev/QA/staging environments using full-scale, production-like databases.
Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

Visit Postgres.ai Console, set up your first organization, and provision a DBLab Standard Edition (DBLab SE) to any cloud or on-premises environment.
- Pricing (starting at $62/month)
- Documentation: How to install DBLab SE
Demo: https://demo.dblab.dev (use the token demo-token to access)
Looking for a free version? Install the DBLab Community Edition by following this tutorial.

How it works

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Speed & scale
- Blazing-fast cloning of PostgreSQL databases – clone in seconds, irrespective of database size
- Theoretical max of snapshots/clones: 264 (ZFS, default)
- Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
Support & technologies
- Supported PostgreSQL versions: 10–18
- Thin cloning (CoW) technologies: ZFS and LVM
- UI for manual tasks and API & CLI for automation
- Packaged in Docker containers for all components
PostgreSQL containers
- Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
- Customization capabilities for containers (docs)
- Docker container and PostgreSQL configuration parameters in the DBLab config
Source database requirements
- Location flexibility: self-managed PostgreSQL, AWS RDS, GCP Cloud SQL, Azure, etc.—no source adjustments needed.
- No ZFS or Docker requirements for source databases
Data provisioning & retrieval
- Simple-mode setup: paste source URL + password, get a probed configuration in one click (UI Configuration → Simple tab)
- Expert-mode form for both logical (dump/restore) and physical (WAL-G, pgBackRest) retrieval
- Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
- Partial data retrieval in logical mode (specific databases/tables)
- Database renaming during snapshot creation (databaseRename option)
- Continuous update in physical mode
- Periodic full refresh in logical mode without downtime
Recovery & management
- Fast Point in Time Recovery (PITR) for physical mode
- Auto-deletion of unused clones
- Time-based auto-deletion of unused branches and snapshots (retention config; safe-only, never force-deletes dependents)
- Snapshot retention policies in DBLab configuration
Clones, branches & snapshots
- "Deletion protection" for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
- Persistent clones withstand DBLab restarts
- "Reset" command for data version switching
- Resource quotas: CPU, RAM
Monitoring & security
- /healthz API endpoint (no auth), extended /status endpoint (API docs)
- Prometheus metrics endpoint (/metrics) for monitoring
- Netdata module for insights

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

database-lab-engine

Live repository statistics

Strong signals

README preview

Why DBLab?

How it works

Where to start

Case studies

Features

How to contribute

Support us on GitHub/GitLab

Spread the word

Share your experience