slothdb

Website · Playground · Discord · Blog · Docs · Benchmarks · Python · SQL Guide

---

Status

SlothDB is early-stage and experimental.

• No vendor PR to ClickBench is open or planned while the handler dispatch is still there.

Treat performance numbers in this README as anecdotal until the architecture work lands. The project is a learning exercise; expect rough edges.

---

Ask in any language. Get SQL.

Type .ask at the slothdb> prompt. A rules parser handles catalog questions and common English shapes in under 10 ms with no model. Anything else falls through to a local Qwen2.5-Coder (0.5B for simple, 1.5B for analytic; lazy-downloaded on first use under -DSLOTHDB_ASK_MODEL=ON), which speaks 29 natural languages: English, Chinese, Spanish, French, German, Japanese, Korean, Russian, Arabic, Portuguese, Italian, Hindi, and more. Every generated statement is shown before it runs. Nothing leaves the machine. Set SLOTHDB_ASK_CONFIRM=1 to add a [Y/n] prompt before each run.

tier	what	cost	covers
1	Rules parser (default)	sub-10 ms, no model	catalog, COUNT/SUM/AVG/GROUP BY/TOP-N, file-source
2	Local Qwen 2.5-Coder 0.5B Q4_K_M	~200 ms, ~310 MB	open-ended SELECT/GROUP BY/filter
3	Local Qwen 2.5-Coder 1.5B Q4_K_M	~500 ms, ~986 MB	window functions, ranking within groups, LAG/LEAD, joins

Both model tiers download lazily in parallel on first .ask (total ~1.3 GB). Router is a pure function of the question: no LLM call involved in routing. Cumulative / running / moving aggregates refuse cleanly (engine gap, not model gap). Full spec, router signals, refusal policy: docs/ASK.md.

Try it in 60 seconds

In your browser - no install, no account: slothdb.org/playground. Full SlothDB compiled to WebAssembly, with a pre-loaded 1,000-row demo CSV + matching Parquet to compare format performance. Files you add stay on your machine.

In Python - CPython 3.8+ on Linux / macOS / Windows (see latest release for published wheel tags; falls back to source build if no wheel matches):

pip install slothdb
python -c "import slothdb; slothdb.demo()"

That generates a 100 000-row CSV, runs three queries, and prints the side-by-side with DuckDB shown above. No files to find, no setup.

# Your own files, same API. pandas round-trip in two lines:
import slothdb, pandas as pd
db = slothdb.connect()
df = db.sql("SELECT region, SUM(revenue) AS rev FROM 'sales.parquet' GROUP BY region").fetchdf()

In Node/JS - npm install @slothdb/wasm:

import { SlothDB } from '@slothdb/wasm';
const db = await SlothDB.create();
const { columns, rows } = db.query("SELECT 1 AS n");

In the shell - download or build from source; then slothdb analytics.slothdb for a persistent single-file DB.

---

What's new in 0.2.6

• Engine work for high-cardinality GROUP BY and parquet decode. Radix-partitioned aggregators for high-cardinality GROUP BY, a bounded hash-table COUNT path, parquet decode improvements (zero-copy VARCHAR, batched RLE unpack), and TopN pushdown into the aggregate. See the Status section above for the context around the previous ClickBench framing.
• Portable build. The GCC and Clang builds compile cleanly again: the MSVC-only getenv_s is replaced with std::getenv, and the vendored snappy source is committed so CI can build it.
• bench/clickbench/. All 43 ClickBench queries verbatim from the ClickBench repo, plus the runner and a chart generator. python bench/clickbench/official_bench.py reproduces the head-to-head against DuckDB.

424 tests pass.

What's new in 0.2.5

• Nested aggregates work everywhere. ROUND(AVG(x)), AVG(x) + 1, SUM(x) / COUNT(*), CAST(SUM(y) AS DOUBLE) and other shapes that wrap an aggregate inside a scalar function or arithmetic used to throw "Function execution for: AVG". The planner now walks the whole expression tree and hoists every aggregate it finds, no matter how deep, so you can write the SELECT list the way you'd write it in DuckDB or Postgres.
• ORDER BY by aggregate alias works. SELECT region, COUNT(*) AS cnt FROM ... GROUP BY region ORDER BY cnt DESC used to default the sort to column 0 and silently sort by region. PhysicalOrderBy now precomputes per-row order keys via the expression executor whenever any clause isn't a plain column ref. Both the full-sort and top-N heap paths use the precomputed keys.
• Arithmetic type promotion fixed. A pre-existing bug surfaced by the hoist: AVG(x) + 1 would lose the +1 and AVG(x) / COUNT(*) would return inf, both because the typed arithmetic kernel reinterpreted operand bytes instead of converting them. Now coerces both operands to the result type with typed fast paths (int to double, int to bigint, float to double).
• bench/ directory at the repo root. All 43 ClickBench queries verbatim from ClickHouse/ClickBench plus a 16-query mixed suite, behind a generic Python runner. Reproducible side-by-side timing against DuckDB. See bench/README.md.

408 tests, 131,537 assertions, green on Windows / Linux / macOS.

What's new in 0.2.3

• GZIP and ZSTD Parquet decode. miniz handles codec=2 with a hand-rolled RFC 1952 header peel (the gzip wrapper miniz refuses by default). Vendored libzstd 1.5.6, decompression-only subset, adds about 50 KB to the binary and unblocks codec=6 files written by Spark / pyarrow / parquet-mr defaults.
• read_parquet glob in the browser. SELECT * FROM '/data/shard_*.parquet' fans out across every match in the playground's MEMFS instead of failing with "Cannot open Parquet". Same path the native CLI has had for a while.
• Single-threaded WASM stops crashing on GROUP BY. std::thread spawn is now routed through a HWThreads() helper that returns 1 under __EMSCRIPTEN__ && !__EMSCRIPTEN_PTHREADS__ and runs the worker inline.
• HTTPS Parquet from the browser playground. Tiny Cloudflare Worker proxy (cloudflare/cors-proxy/, 60 lines, free-tier-friendly) routes around CORS for buckets that don't set Access-Control-Allow-Origin. FROM 'https://host/file.parquet' works in the playground for any host.
• Discord server. discord.gg/XJWyGmX5G. Bug reports, weird query plans, perf threads, anything.

Previously in 0.2.0 - 0.2.2

• Top-N pushdown for ORDER BY ... LIMIT N on Parquet. Bounded-heap operator instead of full sort then truncate. 10M-row ORDER BY q DESC LIMIT 10: 420,344 ms to 119 ms.
• Predicate pushdown via row-group stats. Filters that don't intersect a row group's min/max skip the whole group. Selective queries on big files get the I/O reduction without an index.
• HTTP Range requests for Parquet. Reader pulls the footer, then only the bytes for the row groups it actually needs. No full-file download for most queries.
• Typed batch C API. New slothdb_column_int32_buffer / int64_buffer / double_buffer / varchar_buffer / validity_buffer. The Python wrapper reads one buffer per column instead of two ctypes calls per cell. SELECT 2 columns x 10M rows: 46 s to 16 s.
• Direct string_t emit + lazy QueryResult. Result chunks stay alive until the user pulls them; fetchnumpy() and fetchdf() skip the per-cell Value boxing entirely.
• .ask natural-language sub-REPL. Rules parser in every build (sub-10 ms, no model). Optional local Qwen2.5-Coder GGUF fallback under -DSLOTHDB_ASK_MODEL=ON: 0.5B and 1.5B tiers picked by a deterministic keyword router. 29 languages. Generated SQL is shown before it runs. docs/ASK.md.
• JOIN hot path. Typed int64 hash, parallel CSV pre-parse, build-side projection pushdown, COUNT(*)-over-JOIN fused into the aggregate.
• CREATE LIVE VIEW with incremental CSV append. Useful when a dashboard tails a log that keeps growing.
• Edge build (-DSLOTHDB_EDGE=ON) for sub-MB WASM bundles under Cloudflare Workers' 1 MB cap.

Per-commit history with bench deltas in CHANGELOG.md.

---

Why SlothDB?

SlothDB is an embedded analytical database in C++20. You link it into your application (or run the shell) and point SQL at files on disk. No server process, no import step, no "load the extension first." That's the same model as DuckDB and SQLite, but the defaults are different.

-- No CREATE TABLE. No COPY FROM. Just point at the file.
SELECT department, COUNT(*), AVG(salary)
FROM 'employees.parquet'
WHERE hire_year >= 2020
GROUP BY department
ORDER BY AVG(salary) DESC;

-- Local, HTTP(S), or public S3 - same SQL.
SELECT region, SUM(revenue) FROM 'https://host/data.csv' GROUP BY region;
SELECT * FROM 's3://public-bucket/events.parquet';

If you're already using DuckDB

DuckDB is great and a lot of users should keep using it. SlothDB overlaps on the embedded-columnar-SQL story but makes different default choices. Four of those choices are what we've heard asked for most:

• Live views over growing files. CREATE LIVE VIEW caches a query result and, for the common SELECT * FROM 'file.csv' shape on CSV/TSV, appends only the new bytes on the next query. Useful when a dashboard tails a log that keeps growing.
• Smaller WASM for edge workers. The -DSLOTHDB_EDGE=ON build (CSV/JSON/Parquet only) targets Cloudflare Workers' 1 MB script budget; the full WASM bundle is around 1.3 MB.
• Everything in core, no extensions. HTTP(S), S3 (anonymous public reads), Avro, Excel, Arrow, and SQLite read through the same core binary - no separate install/load step.
• Stable C ABI + numeric error codes. ErrorCode::TABLE_NOT_FOUND = 2000 does not shift between releases; bindings built against 0.1.x keep working.

Same idea as DuckDB otherwise: embedded, columnar, vectorized