Recursive Language Models (RLM)

Python implementation of Recursive Language Models for efficient long-context processing. Context stays in a Python REPL so models can inspect relevant parts and reduce token usage on large-context tasks.

Based on the RLM paper by Alex L. Zhang, Tim Kraska, and Omar Khattab | Official implementation

What is RLM?

RLM enables language models to process extremely long contexts (100k+ tokens) by:

• Storing context as a Python variable instead of in the prompt
• Allowing the LM to recursively explore and partition the context
• Using local search and computation to reduce model token usage on suitable long-context tasks
• Avoiding "context rot" (performance degradation with long context)

Instead of this:

llm.complete(prompt="Summarize this", context=huge_document)  # Context rot!

RLM does this:

rlm = RLM(model="gpt-5-mini")
result = rlm.complete(
    query="Summarize this",
    context=huge_document  # Stored as variable, not in prompt
)

The LM can then peek, search, and recursively process the context adaptively.

Installation

Note: This package is not yet published to PyPI. Install from source:

# Clone the repository
git clone https://github.com/grishahq/recursive-llm.git
cd recursive-llm

# Install in editable mode
pip install -e .

# Or install with dev dependencies
pip install -e ".[dev]"

Future: Once published to PyPI, you'll be able to install with pip install recursive-llm

Requirements

• Python 3.9 or higher
• An API key for your chosen LLM provider (OpenAI, Anthropic, etc.)
• Or a local model setup (Ollama, llama.cpp, etc.)

Quick Start

from rlm import RLM

def main():
    # Initialize with any LLM
    rlm = RLM(model="gpt-5-mini")

    # Process long context
    result = rlm.complete(
        query="What are the main themes in this document?",
        context=long_document,
    )
    print(result)

if __name__ == "__main__":
    main()

RLM uses a spawned worker process for isolated REPL execution. Executable Python scripts must use the standard if __name__ == "__main__": entry-point guard, as shown above. This is required by Python multiprocessing on spawn-based platforms; all repository examples follow this pattern.

Usage and Cost Statistics

RLM.stats aggregates model calls across the complete recursion tree. Token usage comes from provider responses, while cost is calculated on a best-effort basis using LiteLLM's model pricing metadata.

rlm = RLM(
    model="gpt-5-mini",
    recursive_model="deepseek/deepseek-v4-flash",
)
result = rlm.complete(query="Summarize this", context=document)

print(rlm.stats)
# {
#     "llm_calls": 11,
#     "root_calls": 3,
#     "recursive_calls": 8,
#     "leaf_calls": 4,
#     "prompt_tokens": 12500,
#     "completion_tokens": 3200,
#     "cached_tokens": 6000,
#     "estimated_cost_usd": 0.0047,
#     "by_model": {
#         "gpt-5-mini": {"calls": 3, ...},
#         "deepseek/deepseek-v4-flash": {"calls": 8, ...},
#     },
# }

Each root completion receives fresh statistics, so they describe one recursion tree rather than lifetime usage. estimated_cost_usd is None when LiteLLM has no pricing metadata for any completed call. Compare priced_calls with llm_calls before treating the estimate as the full run cost.

When the same RLM instance runs concurrent completions, RLM.stats describes whichever root run completed most recently. Use the structured result API for exact per-run statistics and trajectory:

result = rlm.complete_result(query="Summarize this", context=document)
print(result.answer)
print(result.stats)
for event in result.trajectory:
    print(event.kind, event.depth, event.node_id, event.parent_id)

# Append one complete, versioned run record for later comparison
result.write_jsonl("runs.jsonl")

acomplete_result is the asynchronous equivalent. Trajectories include the complete root, child RLM, and leaf-call tree. Query, context, model response, code, and output content are represented by character counts by default. Set capture_trajectory_content=True only when the resulting logs are allowed to contain that data. An optional event_handler receives events as they occur; handler failures do not interrupt model completion.

Use the non-raising result API when failed runs must be persisted or compared alongside successful runs:

run = rlm.try_complete_result(query="Summarize this", context=document)
if run.succeeded:
    print(run.answer)
else:
    print(run.error_type, run.error)

# Success and failure records share the same versioned JSONL schema.
run.write_jsonl("runs.jsonl")

atry_complete_result is the asynchronous equivalent. Ordinary run failures return a FailedCompletionResult with exact per-run statistics, secret-free configuration, and the complete partial trajectory, including its terminal run_error event. Cancellation and other process-control exceptions are not converted. The existing complete, acomplete, complete_result, and acomplete_result APIs keep their exception behavior.

Completed JSONL records contain the final answer; failed records contain answer: null and a typed error. Query, context, intermediate model responses, code, and REPL output follow the same redaction setting as the in-memory trajectory.

Live Model Comparison

The comparison script uses the same model for both root and recursive calls. It runs one small recursive smoke test by default and reports latency, calls, tokens, and estimated cost:

python benchmarks/compare_same_model.py gpt-5-mini
python benchmarks/compare_same_model.py deepseek/deepseek-v4-flash

Use repeated runs and save raw records before comparing configurations:

python benchmarks/compare_same_model.py gpt-5-mini --full --runs 3 --jsonl results.jsonl
python benchmarks/compare_same_model.py gpt-5-mini --full --runs 3 --mode direct
python benchmarks/compare_same_model.py gpt-5-mini --generated-chars 100000 --seed 2026

--max-depth compares recursion capabilities; --mode direct sends task and context in one normal long-context model request. The script reports pass rate, p50/p95 latency, calls, tokens, and best-effort cost. Task-specific graders require exact IDs, numeric boundaries, and explicit labeled counts rather than accepting arbitrary substrings. Live benchmarks make paid API calls and require the corresponding provider keys. --trace includes sensitive content-bearing trajectories in the JSON output and should be used deliberately.

The repository also includes a SHA-pinned real-document benchmark over the public-domain English translation of War and Peace. The book itself is downloaded separately and is not committed:

curl -L https://www.gutenberg.org/files/2600/2600-0.txt \
    -o /tmp/war-and-peace-2600-0.txt
python benchmarks/war_and_peace.py gpt-5-mini /tmp/war-and-peace-2600-0.txt
python benchmarks/war_and_peace.py deepseek/deepseek-v4-flash \
    /tmp/war-and-peace-2600-0.txt

benchmarks/multi_document.py extends this to three independently sourced large corpora: War and Peace, the official 9/11 Commission Report, and the official Python 3.14 text documentation. It verifies every prepared artifact by SHA-256 and uses exact labeled-field graders. Download and PDF extraction commands, hashes, task definitions, and measured candidate results are in BENCHMARK_RESULTS.md.

API Keys Setup

Copy the example environment file and add keys only for the providers you use:

cp .env.example .env

OPENAI_API_KEY=sk-...
DEEPSEEK_API_KEY=...
MOONSHOT_API_KEY=...

Use the LiteLLM provider prefix for non-OpenAI models, for example deepseek/deepseek-v4-flash or moonshot/kimi-k2.6. This lets a hybrid RLM select the correct API key for each model automatically.

Or pass directly in code:

rlm = RLM(model="gpt-5-mini", api_key="sk-...")

Supported Models

Works with 100+ LLM providers via LiteLLM:

# OpenAI
rlm = RLM(model="gpt-5")
rlm = RLM(model="gpt-5-mini")

# Anthropic
rlm = RLM(model="claude-sonnet-4")
rlm = RLM(model="claude-sonnet-4-20250514")

# Ollama (local)
rlm = RLM(model="ollama/llama3.2")
rlm = RLM(model="ollama/mistral")

# llama.cpp (local)
rlm = RLM(
    model="openai/local",
    api_base="http://localhost:8000/v1"
)

# Azure OpenAI
rlm = RLM(model="azure/gpt-4-deployment")

# And many more via LiteLLM...

Advanced Usage

Two Models (Optimize Cost)

Use a cheaper model for recursive calls:

rlm = RLM(
    model="gpt-5",              # Root LM (main decisions)
    recursive_model="gpt-5-mini"  # Recursive calls (cheaper)
)

Async API

For better performance with parallel recursive calls:

import asyncio

async def main():
    rlm = RLM(model="gpt-5-mini")
    result = await rlm.acomplete(query, context)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

Configuration

rlm = RLM(
    model="gpt-5-mini",
    max_depth=2,                 # One child RLM level, then a plain-LM fallback
    max_iterations=20,           # Maximum REPL iterations per RLM
    repl_timeout=5,              # Hard timeout for each local Python step
    max_output_chars=2000,       # Observation truncation limit
    max_concurrent_subcalls=4,   # Bound batch concurrency
    max_total_calls=24,          # Exact provider-call cap for the full recursion tree
    max_total_tokens=100_000,    # Stop after reported usage crosses this value
    max_total_cost_usd=0.10,     # Stop after reported cost crosses this value
    max_elapsed_seconds=300,     # Deadline shared by root and child calls
    max_retries=2,                # Retry transient provider failures; default is 0
    retry_backoff_seconds=1.0,   # Exponential retry delay; Retry-After is respected
    # Optional LiteLLM params: temperature, timeout, etc.
)

Call limits are reserved atomically before provider requests, including batched and recursive subcalls. Token and cost limits are evaluated after each response because providers only report those values after generation; the crossing response is included in partial statistics attached to BudgetExceededError. A deadline also bounds in-flight provider requests. All limits are optional and are reset for each root completion.

Retries are opt-in and share the same call and elapsed-time budgets as the rest of the recursion tree. Every retry is counted as a provider call and recorded in statistics and trajectories. RLM disables hidden LiteLLM retries to preserve exact accounting; configure max_retries on RLM instead of passing num_retries or max_retries as a LiteLLM option. Null provider content is normalized to an empty response so the existing repair iteration can recover, while malformed response structures fail with ProviderResponseError or use the bounded retry path when enabled.

Final Answer Validation

Applications can reject a syntactically valid final answer and give deterministic feedback to the model without leaving the bounded run:

def validate_answer(answer: str):
    if not answer.startswith("count="):
        return "Answer must start with 'count='."
    return None

rlm = RLM(
    model="gpt-5-mini",
    final_answer_validator=validate_answer,
)

The validator returns None to accept an answer or a non-empty error string to reject it. Rejected answers and feedback are represented by lengths in redacted trajectories and by content only when capture_trajectory_content=True. Validator exceptions abort the run as application errors.

max_depth is an explicit constructor option so that runs remain reproducible. It is not read from an environment variable by the library. Applications may map their own configuration or environment variables to this