DiskDB

🚀 Lightning-fast, Redis-compatible persistent database built in Rust with RocksDB. Experience the power of in-memory performance with the reliability of disk persistence.

┌─────────────────────────────────────────────────────────────────────┐
│                    DiskDB Performance Evolution                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Original  ████████                                    796K ops/s   │
│                                                                     │
│  v0.2      ████████████████████████████████████████    4.2M ops/s   │
│                      5.3x Faster                                    │
│                                                                     │
├─────────────────────────────────────────────────────────────────────┤
│                        Feature Highlights                           │
├─────────────────────────────────────────────────────────────────────┤
│  • 5.3x faster protocol parsing     • 70% less memory usage         │
│  • 2.2x faster memory allocation    • Request pipelining            │
│  • Zero-copy I/O on Linux           • Connection pooling            │
└─────────────────────────────────────────────────────────────────────┘

Table of Contents

• Overview
• Architecture
• Key Features
  • All Your Favorite Data Types
  • Performance That Scales
  • Enterprise-Grade Reliability
  • Developer Experience
• Redis Compatibility
  • What Does "Redis Compatible" Mean?
  • Client Libraries
  • Supported Operations
  • Protocol Details
  • Migration Path
  • Key Differences
• Installation
  • Server Installation
  • Python Client Installation
• Usage Examples
  • Python Client
  • Error Handling
  • Advanced Usage
• Python Package Features
  • Real-World Examples
  • Session Management
  • Rate Limiting

Overview

DiskDB is a modern, high-performance database that brings you the best of both worlds:

• ⚡ Blazing Fast: Near-instant operations with RocksDB's LSM-tree architecture
• 💾 Rock-Solid Persistence: Your data survives restarts, crashes, and power outages
• 🔄 Redis-Inspired: Familiar commands and data types with a similar syntax
• 🌍 Multi-Language: Native clients for Python, Go, and more
• 🔒 Enterprise-Ready: TLS encryption, atomic operations, and production-tested

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Python Client  │     │    Go Client    │     │  Other Clients  │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┴───────────────────────┘
                                 │
                          TCP/TLS Protocol
                                 │
                        ┌────────┴────────┐
                        │  DiskDB Server  │
                        │   (Port 6380)   │
                        └────────┬────────┘
                                 │
                 ┌───────────────┴───────────────┐
                 │      Optimized Engine         │
                 │  ┌─────────────────────────┐  │
                 │  │  Network I/O Layer      │  │
                 │  │  • Buffer Pool          │  │
                 │  │  • Connection Pool      │  │
                 │  │  • Request Pipeline     │  │
                 │  └───────────┬─────────────┘  │
                 │  ┌───────────┴─────────────┐  │
                 │  │  Command Processor      │  │
                 │  │  • Fast Parser (5.3x)   │  │
                 │  │  • Batch Executor       │  │
                 │  └───────────┬─────────────┘  │
                 │  ┌───────────┴─────────────┐  │
                 │  │  Storage Engine         │  │
                 │  │  • RocksDB Backend      │  │
                 │  │  • Write Batching       │  │
                 │  │  • Read Cache           │  │
                 │  └─────────────────────────┘  │
                 └───────────────────────────────┘

🎯 Key Features

🗄️ All Your Favorite Data Types

• Strings: The bread and butter - SET, GET, INCR, APPEND
• Lists: Lightning-fast queues - LPUSH, RPUSH, LPOP, RPOP
• Sets: Unique collections - SADD, SREM, SISMEMBER, SMEMBERS
• Hashes: Object storage made easy - HSET, HGET, HDEL, HGETALL
• Sorted Sets: Leaderboards & rankings - ZADD, ZRANGE, ZSCORE
• JSON: Modern data structures - JSON.SET, JSON.GET with path queries
• Streams: Event sourcing & logging - XADD, XRANGE, XLEN

⚡ Performance That Scales

• Microsecond Latency: Optimized for speed with zero-copy operations
• 5.3x Faster Parsing: Advanced protocol parsing with optional C acceleration
• Request Pipelining: Process up to 100 requests in a single batch
• Buffer Pooling: Reusable memory buffers reduce GC pressure by 70%
• Concurrent by Design: Lock-free reads, minimal write contention
• Smart Caching: Hot data stays in memory, cold data on disk
• Compression: Automatic Snappy/LZ4/Zstd compression saves 50-80% disk space

🛡️ Enterprise-Grade Reliability

• ACID Guarantees: Your data is always consistent
• Crash Recovery: Automatic WAL-based recovery
• Atomic Operations: No partial updates, ever
• TLS Encryption: Bank-level security for client connections

🔧 Developer Experience

• Zero Configuration: Works out of the box
• Redis-Like Commands: Familiar syntax for Redis users
• Rich Client Libraries: First-class Python and Go support
• Docker Ready: One command to launch

🔄 Redis Compatibility Explained

What Does "Redis Compatible" Mean?

DiskDB is Redis-inspired and implements a subset of Redis commands with the same syntax. While it uses a Redis-like protocol, it currently requires its own client libraries:

1. Redis-Like Command Syntax - Commands use familiar Redis syntax (SET, GET, etc.)
2. Similar Wire Protocol - Based on Redis protocol concepts but not fully RESP-compatible
3. Own Client Libraries - Currently requires DiskDB-specific clients (Python, Go)
4. Subset of Redis Commands - Implements the most commonly used Redis operations

Client Libraries

DiskDB provides its own client libraries that use Redis-like syntax:

# Python - Using DiskDB client (NOT redis-py)
from diskdb import DiskDB
db = DiskDB(host='localhost', port=6380)
db.set('key', 'value')  # Redis-like syntax

# Go - Using DiskDB Go adapter
import "github.com/transybao1393/diskdb/client"
db, _ := client.Connect("localhost:6380")
db.Set("key", "value")

# Direct TCP connection also works
# You can send commands via telnet or netcat

Supported Operations

DiskDB currently implements these Redis-like commands:

✅ Implemented:

• String Operations: SET, GET, INCR, DECR, INCRBY, APPEND
• List Operations: LPUSH, RPUSH, LPOP, RPOP, LRANGE, LLEN
• Set Operations: SADD, SREM, SISMEMBER, SMEMBERS, SCARD
• Hash Operations: HSET, HGET, HDEL, HGETALL, HEXISTS
• Sorted Set Operations: ZADD, ZREM, ZRANGE (with WITHSCORES), ZSCORE, ZCARD
• Key Operations: EXISTS, DEL, TYPE
• Connection: PING, ECHO
• Server: INFO, FLUSHDB

➕ DiskDB Unique Features:

• JSON Operations: JSON.SET, JSON.GET, JSON.DEL (native JSON support)
• Stream Operations: XADD, XRANGE, XLEN (event streaming)
• Automatic Persistence: All data persisted to disk automatically

🚧 Planned Features:

• Additional String Ops: STRLEN, GETSET, MGET, MSET, DECRBY (in enum but not parser)
• Additional List Ops: LINDEX, LSET, LTRIM, LINSERT
• Additional Set Ops: SINTER, SUNION, SDIFF, SRANDMEMBER
• Additional Hash Ops: HLEN, HKEYS, HVALS, HMGET, HMSET, HINCRBY
• Additional Sorted Set Ops: ZREVRANGE, ZCOUNT, ZRANK, ZREVRANK
• Key Management: EXPIRE, TTL, PERSIST, KEYS, SCAN, RENAME
• Pub/Sub: PUBLISH, SUBSCRIBE, UNSUBSCRIBE
• Transactions: MULTI, EXEC, WATCH, DISCARD
• Connection: SELECT, AUTH, DBSIZE
• Lua Scripting: EVAL, EVALSHA

❌ Not Planned:

• Redis Cluster: Single-node focus
• Redis Modules: Built-in features instead
• Dangerous Ops: FLUSHALL, SHUTDOWN, CONFIG (for safety)

Protocol Details

DiskDB uses a text-based protocol similar to Redis:

# Using telnet or netcat
$ telnet localhost 6380
SET mykey "Hello DiskDB"
OK
GET mykey
Hello DiskDB

# Simple text protocol format
# Send: COMMAND arg1 arg2 ...
# Receive: Response

# Note: DiskDB's protocol is Redis-inspired but not fully RESP-compatible
# Some Redis tools may work, but full compatibility is not guaranteed

Using DiskDB in Your Application

DiskDB Client Example:

# Using DiskDB Python client
from diskdb import DiskDB

def cache_user(user_id, data):
    db = DiskDB()
    db.set(f"user:{user_id}", json.dumps(data))
    # Note: No built-in expiration yet (EXPIRE not implemented)
    
def get_cached_user(user_id):
    db = DiskDB()
    data = db.get(f"user:{user_id}")
    return json.loads(data) if data else None

Important Note: DiskDB requires its own client libraries and is not compatible with Redis ORMs or frameworks like:

• ❌ redis-py, Flask-Redis, Django-Redis
• ❌ node-redis, ioredis, Bull
• ❌ Jedis, Lettuce, Spring Data Redis
• ✅ Use DiskDB's Python client: pip install diskdb
• ✅ Use DiskDB's Go adapter

Migration Path

Migrating from Redis to DiskDB requires code changes since DiskDB uses its own client:

# Before - Using Redis
import redis
r = redis.Redis(port=6379)
r.set("key", "value")

# After - Using DiskDB
from diskdb import DiskDB
db = DiskDB(port=6380)
db.set("key", "value")

Migration Strategy:

# Gradual migration with separate clients
import redis
from diskdb import DiskDB

class MigrationCache:
    def __init__(self):
        self.redis = redis.Redis(port=6379)
        self.diskdb = DiskDB(port=6380)
    
    def set(self, key, value):
        # Write to both during migration
        self.redis.set(key, value)
        self.diskdb.set(key, value)
    
    def get(self, key):
        # Try DiskDB first, fallback to Redis
        value = self.diskdb.get(key)
        if value is None:
            value = self.redis.get(key)