Loading repository data…
Loading repository data…
pomagrenate / repository
A context squeezer tools for AI Agent, reduce token consuption
Deterministic document compression — squeeze tokens, not meaning.
Context-Squeezer is a high-performance, local CLI tool and library designed to compress large documents to fit within LLM token budgets while maintaining semantic structure and determinism.
make build
Creates a standalone binary with no external C++ library dependencies.
make build-static
Context-Squeezer is designed to be easily integrated into other projects.
import "contextsqueezer/pkg/api"
opt := api.Options{MaxTokens: 2000, Profile: "api"}
squeezed, err := api.SqueezeBytes(input, opt)
find_package(ContextSqueeze REQUIRED)
target_link_libraries(your_app PRIVATE ContextSqueeze::contextsqueeze)
csq_last_error().Set CSQ_DEBUG=1 to enable detailed stage-by-stage debug logging.
Context-Squeezer implements a streaming chunked processor for large files. Use MaxMemoryMB in the pipeline configuration to control peak resident memory usage.
The tool includes a built-in benchmark suite to verify performance and determinism:
./build/bin/contextsqueeze bench --suite default --runs 5
This project is licensed under the terms of the LICENSE file included in the repository.
| Format | Extension |
|---|---|
.pdf | |
| 📝 Word | .docx |
| 🌐 HTML | .html |
| 📃 Plain Text / Markdown | .txt, .md |
./scripts/build.sh
# Compress a PDF to 8 000 tokens, API profile
./build/bin/contextsqueeze --max-tokens 8000 --profile api doc.pdf
# Output as JSON
./build/bin/contextsqueeze --json page.html > out.json
# Show compression stats for a DOCX
./build/bin/contextsqueeze stats --source docx --max-tokens 2000 report.docx
Runtime linking (Linux)
export LD_LIBRARY_PATH="$(pwd)/build/native/lib:${LD_LIBRARY_PATH}"Runtime linking (macOS)
export DYLD_LIBRARY_PATH="$(pwd)/build/native/lib:${DYLD_LIBRARY_PATH}"
| Stream | Content |
|---|---|
stdout | Squeezed text (or JSON when --json) |
stderr | Errors, warnings, stats, profiling |
Flags:
| Flag | Effect |
|---|---|
--quiet | Suppress warnings |
--verbose | Print per-stage timing |
CSQ_DEBUG=1 | Include stack traces on failure |
When using --json, the output contains these stable fields (schema locked at version 1):
{
"schema_version": 1,
"engine_version": "1.0.0",
"build": { ... },
"bytes_in": 0,
"bytes_out": 0,
"tokens_in": 0,
"tokens_out": 0,
"reduction_pct": 0.0,
"aggressiveness": "",
"profile": "",
"budget_applied": true,
"truncated": false,
"source_type": "",
"warnings": [],
"text": "..."
}
Token approximation formula:
approx_tokens = ceil(bytes / 4) + whitespace_word_count
# Run with determinism checks (SHA-256 per run)
./build/bin/contextsqueeze bench --suite default --runs 5 --warmup 1 --aggr 0..9
# Benchmark a single file
./build/bin/contextsqueeze bench --file testdata/bench/large.txt --runs 10
# Benchmark an entire directory
./build/bin/contextsqueeze bench --dir testdata/bench --pattern "*.txt"
Bench checks deterministic SHA-256 digests per run and exits non-zero on mismatch.
# Capture CPU + heap profiles
./build/bin/contextsqueeze profile \
--cpu out/cpu.pprof \
--heap out/heap.pprof \
--seconds 10 \
testdata/bench/large.txt
# View interactive profile UI
go tool pprof -http=:0 out/cpu.pprof
See docs/perf.md for full benchmark and profiling workflows.
# Run tests
./scripts/test.sh
# Format code
./scripts/format.sh
# Verify no committed binaries
./scripts/verify_no_binaries.sh
# Run sanitizer gate
./scripts/sanitize.sh
Generate release artifacts locally (not committed to git):
./scripts/release.sh
Outputs are placed in dist/:
dist/
├── contextsqueeze_linux_amd64
├── contextsqueeze_darwin_amd64
├── contextsqueeze_darwin_arm64
├── SHA256SUMS
└── VERSION
| Code | Meaning |
|---|---|
0 | Success |
2 | Usage error |
3 | Input error |
4 | Parse error |
5 | Timeout |
6 | Internal error |
word/document.xml entry.LD_LIBRARY_PATH as shown in the Quick Start above.DYLD_LIBRARY_PATH as shown in the Quick Start above.Context-Squeezer · v1.0.0 · License