Contributing

Setup, PR guidelines, adding test cases and providers.

Scope

PRs that add backend features, storage logic, or dashboard code to this repo will be closed. If unsure, open an issue first.

Setup

git clone https://github.com/syd-ppt/pramana
cd pramana
uv pip install -e ".[dev]"

Development

# Run tests
pytest tests/

# Lint
ruff check .

# Format
ruff format .

PR guidelines

• One feature per PR
• Include tests for new functionality
• Update documentation if applicable
• Follow existing code style
• No breaking changes without discussion
• pytest tests/ must pass
• ruff check . must pass

Adding test cases

Guidelines

• Categories: reasoning, factual, instruction_following, coding, safety, creative
• Format: one JSON object per line in JSONL files
• Immutability: once published, suites never change (create new versions)
• Cost awareness: estimate tokens accurately

Steps

1. Choose tier: cheap / moderate / comprehensive
2. Append to the corresponding JSONL file
3. Update manifest.toml with new count and cost estimate
4. Compute new suite hash:

   python -c "from pramana.hash import hash_suite; print(hash_suite('src/pramana/suites/v1.0/cheap.jsonl'))"

5. Submit PR with test case + rationale

Example test case

{
  "id": "cheap-011",
  "category": "reasoning",
  "input": "If 5 cats catch 5 mice in 5 minutes, how long for 100 cats to catch 100 mice?",
  "ideal": ["5 minutes", "5 mins"],
  "assertion": {
    "type": "contains_any",
    "case_sensitive": false
  },
  "metadata": {
    "difficulty": "medium",
    "tokens_est": 40,
    "tags": ["math", "logic"]
  }
}

Adding providers

Providers auto-register via the @register() decorator. No manual __init__.py edit needed.

1. Create src/pramana/providers/yourprovider.py
2. Subclass BaseProvider, decorate with @register()
3. Implement complete() and estimate_tokens()
4. Add model prefix to FALLBACK_MODELS in models.py
5. Add tests

See Providers for the full interface and example.

Issue reporting

• Bug reports: include Python version, OS, command, error output
• Feature requests: explain use case, not just solution
• Test case suggestions: provide example with expected behavior

Code of conduct

See CODE_OF_CONDUCT.md. Be respectful, focus on the idea, assume good intent, scientific rigor over opinions.