🇬🇧 English Version
HIBP Offline Downloader (C++ Core + Python API)
A lightweight, high-performance offline downloader and API server for Have I Been Pwned? passwords.
It allows you to download, verify, repair, and serve the entire Pwned Passwords database locally - with no dependency on the public API.
Written in C++ (core logic) with a Python Flask REST interface, designed to be fully compatible with PLD Linux Distribution, and easily portable to Debian, Arch, and other Linux systems.
🧩 About the Project
This project was developed from scratch as an independent, high-performance alternative to the
official HaveIBeenPwned/PwnedPasswordsDownloader repository,
originally created under the Have I Been Pwned initiative by Troy Hunt and contributors.
It also draws inspiration from several related community discussions and improvements:
🚀 Features
-
🧠 C++ Core - highly optimized multithreaded engine for downloading, verifying, repairing, and optimizing HIBP data.
-
🌐 Python REST API (Flask) - exposes the functionality as a simple HTTP service.
- Flask was chosen because FastAPI is not packaged in PLD Linux,
and this project uses only system-maintained libraries.
- On Debian/Ubuntu or Arch, you can easily replace Flask with FastAPI + Uvicorn.
-
🕒 Custom Cron Scheduler - implemented in pure Python (stdlib only).
- Replaces
apscheduler, which is not available in PLD Linux repositories.
-
💾 SQLite3 backend - local, transactional, indexed database for fast offline queries.
-
🧱 Fully self-contained - no pip, no virtualenvs, no network dependencies.
-
🧵 Thread-safe operations - safe multithreaded downloads, verifications, and DB updates.
-
🧮 Detailed logging and progress tracking with persistent logs in SQLite.
-
🧪 Tested on PLD, Debian, and Arch Linux.
⚙️ System Requirements
- Linux (tested on PLD, Debian 13, Arch Linux)
- Python 3.11+
- g++ / clang with C++17
- Libraries:
sqlite3, libcurl, openssl
- Optional (for HTTP API):
flask or fastapi + uvicorn
🧰 Installation
No pip is required - everything uses system libraries and compilers.
git clone https://github.com/KrisKros123/hibp-offline-downloader.git
cd hibp-offline-downloader
1️⃣ Build the C++ module
g++ -O3 -std=c++17 -fPIC -shared hibp_module.cpp \
-o hibp_module$(python3-config --extension-suffix) \
$(python3 -m pybind11 --includes) \
-lsqlite3 -lcurl -lssl -lcrypto -lpthread
2️⃣ Run the server
python3 hibp_server.py
By default it listens on:
👉 http://0.0.0.0:5000
🧠 API Overview
| Endpoint | Method | Description |
|---|
/api/health | GET | Health check and DB stats |
/api/stats | GET | Database statistics |
/api/status | GET | Current operation status |
/api/check | POST | Check if password is pwned |
/api/download | POST | Full cycle: Download + Verify + Repair + Optimize |
/api/verify | POST | Verify database integrity only |
/api/verify-and-repair | POST | Verify + Repair + Optimize |
/api/optimize | POST | Optimize the database |
/api/logs | GET | Get operation logs |
/api/scheduler/status | GET | Scheduler info and next run time |
Incremental Updates via ETags
Unlike a full re-download of the entire Pwned Passwords dataset (≈50-55 GB raw),
this implementation performs incremental updates based on ETag range comparison,
as discussed in Issue #29.
When /api/download is called:
- The downloader checks all 1,048,576 prefix ranges against their current ETags.
- Only ranges with new or changed ETags are fetched and merged into the local database.
- Unchanged ranges are skipped, drastically reducing bandwidth and update time.
- Afterward, the database is verified, repaired if necessary, and optimized.
This keeps the local dataset synchronized with Have I Been Pwned
while avoiding redundant downloads - turning multi-hour full imports
into short and efficient delta updates.
🕒 Scheduler (Cron Replacement)
This project implements its own lightweight cron scheduler in Python, using only the standard library.
It was created because PLD Linux does not include apscheduler in its repositories.
Example configuration (via environment variables):
export HIBP_AUTO_UPDATE=true
export HIBP_AUTO_UPDATE_CRON="0 2 * * 0"
export HIBP_TIMEZONE="Europe/Warsaw"
On other systems, you may easily replace it with APScheduler, systemd timers, or cron jobs.
💾 Database Size & Storage Explanation
| Format / Storage Type | Logical Size | Physical Size | Description |
|---|
Plain text (.txt) | ~50-55 GB | ~50-55 GB | Raw SHA-1 hash list downloaded from HaveIBeenPwned. No indexing, direct range data. |
SQLite3 database (.db) | ~130-135 GB | ~130-135 GB | Fully indexed version of the password dataset (additional storage used for indexes and integrity metadata). |
Btrfs image (.img, compress=zstd) | ~130 GB (logical) | ~35 GB + 4 GB Btrfs overhead | SQLite database stored on a Btrfs filesystem image with zstd compression enabled - effective ~4× size reduction with minimal performance loss (~100 RPS). |
SQLite3 takes roughly 2-2.5× more disk space.
This is due to:
- B-tree indexing for O(1) hash prefix lookups
- Page alignment (4 KiB) and internal fragmentation
- WAL journaling and ACID transaction metadata
- Binary-to-text conversion overhead for SHA1 strings
- Row headers, padding, and page-level slack space
This trade-off ensures:
- Instant offline
check_password queries
- Fully resumable and recoverable operations
- Strong data integrity guarantees
- Reduced CPU overhead during hash verification
Disk space optimization with Btrfs compression
To further reduce disk space usage, the SQLite database can be stored on a compressed Btrfs volume.
In testing, creating a 64 GB Btrfs image and mounting it with zstd compression reduced the actual disk usage by over 4× compared to the raw .db file.
Example setup:
fallocate -l 64G hibp_data.img
mkfs.btrfs hibp_data.img
mount -o compress=zstd hibp_data.img /path/to/mountpoint
mv pwned_passwords.db /path/to/mountpoint/
After moving the SQLite database to the Btrfs filesystem:
- Reported database size: ~130 GB (logical)
- Actual disk space used: ~35 GB + 4 GB Btrfs metadata overhead
- Effective compression ratio: ≈ 4× smaller footprint
This solution is completely transparent to the application - it works without any code changes,
only by adjusting the database location (e.g. via the HIBP_DB_PATH environment variable or at the top of the Python file).
🧪 Tested On
| Distribution | Status | Notes |
|---|
| PLD Linux | ✅ | Fully self-contained (Flask + built-in scheduler) |
| Debian 13 | ✅ | Fully compatible with system packages |
| Arch Linux | ✅ | Fully compatible with system packages |
🧾 License / Usage Rights
This code is public domain - you are free to use, modify, edit, and redistribute
it without any restrictions.
Attribution is appreciated but not required.
Everything is provided as-is, for educational and practical use.
🧭 Credits
The project was developed as an original, high-performance implementation
inspired by the concepts of the official HaveIBeenPwned PwnedPasswordsDownloader
(created by Troy Hunt and contributors), as well as ideas shared by the community in related discussions.
Written entirely from scratch in C++ and Python,
optimized for Linux systems (PLD, Debian, Arch),
with focus on portability, efficiency, and long-term maintainability.
🇵🇱 Wersja Polska
HIBP Offline Downloader (rdzeń C++ + API w Pythonie)
Lekki, wydajny serwer API i narzędzie offline do obsługi bazy haseł Have I Been Pwned?.
Umożliwia pobieranie, weryfikację, naprawę i optymalizację lokalnej kopii bazy Pwned Passwords - bez korzystania z publicznego API.
Całość została napisana w C++ (rdzeń) oraz Pythonie z interfejsem REST (Flask),
z pełną zgodnością z PLD Linux Distribution, i łatwą przenośnością na Debiana i Archa.
🧩 O projekcie
Projekt został napisany od podstaw jako niezależna, wysokowydajna alternatywa dla
oficjalnego projektu HaveIBeenPwned/PwnedPasswordsDownloader,
rozwijanego w ramach inicjatywy Have I Been Pwned przez Troy’a Hunta i współtwórców.
Zainspirowany został również dyskusjami i poprawkami społeczności:
🚀 Funkcje
-
🧠 Rdzeń w C++ - zoptymalizowany wielowątkowy silnik pobierania, weryfikacji i naprawy danych.
-
🌐 REST API w Pythonie (Flask) - udostępnia funkcje przez prosty interfejs HTTP.
- Flask został użyty, ponieważ FastAPI nie jest dostępne w PLD Linux,
a projekt korzysta wyłącznie z utrzymywanych bibliotek systemowych.
- W Debianie lub Archu można łatwo zastąpić Flaska przez FastAPI + Uvicorn.
-
🕒 Własny scheduler (cron) - czysty Python, bez zewnętrznych zależności.
-
💾 Baza SQLite3 - szybkie, lokalne i transakcyjne przechowywanie danych.
-
🧱 Brak potrzeby pip - korzysta wyłącznie z bibliotek systemowych.
-
🧵 Bezpieczna wielowątkowość - synchroniczne operacje z blokadami mutex.
-
🧮 Trwałe logowanie postępu i błędów.
-
🧪 Przetestowany na PLD, Debianie i Archu.
⚙️ Wymagania
- Linux (PLD, Debian 13, Arch)
- Python 3.11+
- g++ / clang z obsługą C++17
- Biblioteki:
sqlite3, libcurl, openssl
- Opcjonalnie: `flask