pramana docs

Crowdsourced drift detection for LLM APIs.

LLM providers update, fine-tune, and swap models behind stable identifiers. Same name, different behavior. Pramana measures that.

Install

pip install pramana-ai

30-second demo

$ export OPENAI_API_KEY=sk-...
$ pramana run --tier cheap --model gpt-5.2
Running cheap suite against gpt-5.2...
✓ 10/10 passed
Pass rate: 100.0%

$ pramana submit results.json
✓ Submitted 10 results

What it does

1. Runs fixed prompts against provider APIs with deterministic parameters (temperature=0.0, seed=42)
2. Hashes results (SHA-256 of model + prompt + output) for deduplication
3. Submits to a public dataset where hash-based consistency tracking detects changes across users and time

Pages

Scope

Pramana targets API providers (OpenAI, Anthropic, Google) where silent updates occur. Local models (Ollama, vLLM) are out of scope — you control the weights, so there's no drift to detect.