Methodology

How Pramana detects LLM drift through output consistency tracking.

Core Principle

Each submission stores output_hash = sha256(model_id | prompt_id | output). Same prompt + same model + different hash = the model's behavior changed. That's drift. No scoring needed.

Output Hash

Every submission computes a deterministic hash from three fields:

output_hash = SHA-256(model_id + "|" + prompt_id + "|" + output)

This hash is stored with each CSV record. It captures the exact output for a given (model, prompt) pair without storing raw outputs in the aggregated data.

Drift Detection

During daily aggregation, Pramana tracks the last-seen hash for each (model, prompt) pair. When a prompt's hash changes from one day to the next, it counts as a "drifted prompt."

Per-model, per-day statistics

MetricDefinition
submissionsTotal submission records for this model on this day
prompts_testedUnique prompt IDs submitted
unique_outputsUnique output hashes observed
drifted_promptsPrompts whose hash differs from the previous day's hash

Consistency Rate

consistency = (prompts_tested − drifted_prompts) / prompts_tested

A consistency rate of 1.0 means all tested prompts returned the same output as the previous day. A rate below 1.0 indicates drift.

ConsistencyStatusInterpretation
≥ 95%StableModel behavior is consistent
80% – 95%WatchSome outputs have changed
< 80%DriftingSignificant behavioral change detected

Aggregation Pipeline

  1. Parse all archive CSVs into flat record list
  2. Sort records by date
  3. Track prevHash: Map<model|prompt, hash> — last-seen hash per prompt
  4. Group records by (date, model)
  5. For each (date, model) group: compute submissions, prompts_tested, unique_outputs, drifted_prompts
  6. Store as chart_data.json
Privacy property: Only aggregate counts and hashes are stored in chart data. Individual outputs cannot be reconstructed from the aggregated statistics.

Badge Endpoint

The /api/badge/:model endpoint computes consistency over the last 7 days and returns an SVG badge: