3. Examples, Context, & Schemas (Standalone)

• Where: examples/, context/, and schemas/ directories.
• What it is: examples/ contains "golden" input/output pairs for few-shot prompting. context/ contains general reference materials (API docs, glossaries). schemas/ houses standard JSON schemas for validating structured model outputs.
• Requirements: None.
• Compatibility: Supplement prompts and skills to secure structured output formats and enable programmatically validated check assertions.

4. Automated Evaluation Suite (Tooling)

• Where: tests/ and scripts/ directories.
• What it is: A Python-based runner that takes your prompts, injects variables, sends them to major LLMs via litellm, and asserts that the output matches expectations.
• Requirements: Python, litellm, PyYAML, and an API Key (e.g., LLM_API_KEY).
• Compatibility: This feature actively tests the items in prompts/ and examples/. It is strictly optional for users who just want to borrow the markdown files.

Setup

The toolkit relies on Python and uses litellm to dynamically switch between major LLM providers (OpenAI, Anthropic, Google Gemini, etc.) without requiring specific SDK lock-in.

1. Create and Activate a Virtual Environment: Due to PEP-668 protections on system Python environments, create a local virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate
   

2. Install Dependencies:

   pip install -r requirements.txt
   

3. Configure the Environment: Set the API key for your chosen model provider. For instance:

   export LLM_API_KEY="your-api-key-here"
   # Depending on the model, litellm will natively pick up provider-specific keys like OPENAI_API_KEY as well.
   

Usage

1. Scaffolding a New Prompt

Use the included python script to quickly scaffold a new prompt template, configuration file, and placeholder examples:

python scripts/new_prompt.py my-new-prompt

This generates prompts/my-new-prompt.md, prompts/my-new-prompt.config.yaml, and initial examples.

2. Rendering Templates

You can dynamically render a prompt for manual inspection or injection into other workflows:

python scripts/render_template.py prompts/refactor-logic.md --LANGUAGE Python --FRAMEWORK FastAPI --GOAL "execution speed"

Unknown arguments are treated as template variables mapping directly to {{VARIABLES}}.

3. Evaluating Prompts

To test for regressions or changes in model outputs, run the evaluation suite against a defined test case file:

python tests/runner.py --case tests/cases/refactor-logic.cases.yaml

The runner will dynamically call the LLM and run assertions against the outputs using the logic defined in tests/assertions.py.

Conventions & Rules

Every prompt must have two sidecar files:

1. prompts/<name>.md, the actual template, using {{VARIABLE}} placeholders.
2. prompts/<name>.config.yaml, the exact engine settings (temperature, target model, allowed variables), and links to its example pairs and eval cases.

See docs/CONVENTIONS.md for naming and versioning rules, and docs/MODEL_COMPAT.md for verification details against model versions.

Why this structure?

• Parameter Configs: Attaches exact engine settings (temperature, target model) directly to the prompt.
• Few-shot Pairs: Examples provide the fastest way to re-anchor a prompt when an upstream model silently changes behavior.
• Templating: Turns single-use prompts into scalable, reusable assets.
• Tests & CI: Migrates prompt regressions from "bugs found in production" to "failing tests caught in CI".
• Context Kitchen: Keeps large reference materials isolated, ensuring core prompt instructions remain short and legible.

Contributing

Open a PR using the template, it will ask you to confirm you've added or updated an example pair and run the eval suite locally. See CODEOWNERS for review routing.