⚙️ Configuration

Configuration is handled via environment variables plus a user config file under the platformdirs user config path (<user-config>/abx/config.env). Runtime-derived cache entries such as resolved binary paths are stored separately in <user-config>/abx/derived.env:

abx-dl config                        # show all config (global + per-plugin)
abx-dl config --get WGET_TIMEOUT     # get a specific value
abx-dl config --set TIMEOUT=120      # set persistently (resolves aliases)

Output is grouped by section:

# GLOBAL
TIMEOUT=60
USER_AGENT="Mozilla/5.0 ..."
...

# plugins/wget
WGET_BINARY="wget"
WGET_TIMEOUT=60
...

# plugins/chrome
CHROME_BINARY="chromium"
...

Common options:

• TIMEOUT=60 - default timeout for hooks
• USER_AGENT - default user agent string
• {PLUGIN}_BINARY - path or name of the binary to use (e.g. WGET_BINARY=wget or CHROME_BINARY=/usr/bin/chromium)
• {PLUGIN}_ENABLED=True/False - enable/disable specific plugins
• {PLUGIN}_TIMEOUT=120 - per-plugin timeout overrides

Aliases are automatically resolved (e.g. --set USE_WGET=false saves as WGET_ENABLED=false).

One-off config is easy via env vars or CLI args:

env \
  TIMEOUT=120 \
  WGET_TIMEOUT=120 \
  abx-dl \
    --dir=./config-example \
    --plugins=title,wget \
    --timeout=90 \
    'https://example.com'

📦 Install

uv tool install abx-dl
abx-dl version

uvx abx-dl version

abx-dl install wget title

🔠 Usage

abx-dl --plugins=title,wget --dir=./downloads --timeout=120 'https://example.com'

# Default command - a bare URL archives with all enabled plugins:
abx-dl 'https://example.com'

# Select plugins by output type (mimetypes, categories, or file extensions):
abx-dl --output=html,pdf,video/ 'https://example.com'
abx-dl -o text -o image -o mp4 'https://example.com'

# Limit work to a subset of plugins by name:
abx-dl --plugins=wget,title,screenshot,pdf 'https://example.com'

# Skip auto-installing missing dependencies (emit warnings instead):
abx-dl --no-install 'https://example.com'

# Specify output directory (default is current working dir):
abx-dl --dir=./downloads 'https://example.com'

# Set timeout:
abx-dl --timeout=120 'https://example.com'

Commands

abx-dl <url>                              # Download URL (default shorthand)
abx-dl plugins                            # Check + show info for all plugins
abx-dl plugins wget ytdlp git             # Check + show info for specific plugins
abx-dl install wget ytdlp git             # Pre-install plugin dependencies
abx-dl config                             # Show all config values
abx-dl config --get TIMEOUT               # Get a specific config value
abx-dl config --set TIMEOUT=120           # Set a config value persistently

Installing Dependencies

Many plugins require external binaries (e.g., wget, chrome, yt-dlp, single-file).

By default, abx-dl lazily installs missing dependencies as needed when you download a URL. Use --no-install to skip plugins with missing dependencies instead. install runs only the pre-run dependency pipeline (required_binaries → BinaryRequestEvent → BinaryEvent) without starting crawl setup or snapshot extraction:

abx-dl install wget title
abx-dl plugins wget title

abx-dl 'https://example.com'              # auto-installs missing deps on-the-fly
abx-dl --no-install 'https://example.com' # skips plugins with missing deps and emits warnings
abx-dl install wget singlefile ytdlp      # installs dependencies for specific plugins only
abx-dl plugins                            # checks which dependencies are available/missing

Successful preflight installs are cached for 24 hours in derived.env under ABX_INSTALL_CACHE, keyed by binary name. If a binary was installed successfully recently, abx-dl skips re-running the install preflight for that binary. Cached abspaths are still validated at use time, and stale cache entries fall back to abxpkg provider resolution.

The normal runtime flow after dependency preflight is:

• CrawlEvent (internal lifecycle root)
• CrawlSetupEvent → plugin on_CrawlSetup__* hooks
• CrawlStartEvent → SnapshotEvent
• SnapshotEvent → plugin on_Snapshot__* hooks
• SnapshotCleanupEvent / CrawlCleanupEvent

Hook output contract:

• binary preflight is driven by plugin required_binaries and handled by abxpkg, not by plugin hooks
• on_CrawlSetup__* hooks emit no stdout JSONL records
• on_Snapshot__* hooks emit only ArchiveResult, Snapshot, and Tag
• the TUI and services consume structured events derived from those hook records

Dependencies are installed to <user-config>/abx/lib/{arch}/ using the appropriate package manager:

• pip packages → <user-config>/abx/lib/{arch}/pip/venv/
• npm packages → <user-config>/abx/lib/{arch}/npm/
• brew/apt packages → system locations

You can override the install location with ABXPKG_LIB_DIR=/path/to/lib abx-dl install wget.

Output Structure

By default, abx-dl writes results into the current working directory. Each run creates an index.jsonl manifest plus one subdirectory per plugin that produced output. If you want to keep runs isolated, cd into a scratch directory first or pass --dir=/path/to/run.

mkdir -p /tmp/abx-run && cd /tmp/abx-run
uvx --from abx-dl abx-dl --plugins=title,wget 'https://example.com'

./
├── index.jsonl             # Snapshot metadata and results (JSONL format)
├── title/
│   └── title.txt
├── favicon/
│   └── favicon.ico
├── screenshot/
│   └── screenshot.png
├── pdf/
│   └── output.pdf
├── dom/
│   └── output.html
├── wget/
│   └── example.com/
│       └── index.html
├── singlefile/
│   └── output.html
└── ...

All Outputs

• index.jsonl - snapshot metadata and plugin results (JSONL format, ArchiveBox-compatible)
• title/title.txt - page title
• favicon/favicon.ico - site favicon
• screenshot/screenshot.png - full page screenshot (Chrome)
• pdf/output.pdf - page as PDF (Chrome)
• dom/output.html - rendered DOM (Chrome)
• wget/example.com/... - mirrored site files
• singlefile/output.html - single-file HTML snapshot
• ... and more via plugin library ...

Available Plugins

See the abx-plugins marketplace.

Snapshot / Extraction Plugins

• ytdlp - downloads media plus sidecars: audio, video, images/thumbnails, subtitles (.srt, .vtt), JSON metadata, and text descriptions.
• gallerydl - downloads gallery/media sets as images, videos, JSON sidecars, text sidecars, and ZIP archives.
• forumdl - exports forum/thread archives as JSONL, WARC, and mailbox-style message archives.
• git - clones repository contents including text, binaries, images, audio, video, fonts, and other tracked files.
• wget - mirrors pages and requisites as HTML, WARC, images, CSS, JavaScript, fonts, audio, and video.
• archivedotorg - saves a Wayback Machine archive link as plain text.
• favicon - saves site favicons and touch icons as image files.
• modalcloser