pramana-api docs
Backend infrastructure + dashboard for LLM drift detection.
This is the server-side half of Pramana. It receives eval submissions from the CLI, stores them in R2, aggregates statistics, and serves a drift detection dashboard.
The Problem
LLM behavior changes silently. Provider updates, quantization changes, and infrastructure migrations alter model outputs without notice. Chen et al. (2023) documented significant accuracy drops in GPT-4 over a three-month period. How Is ChatGPT's Behavior Changing over Time? (Stanford/Berkeley).
Architecture
┌──────────┐ POST /api/submit ┌──────────────┐ R2 ┌─────────────┐
│ CLI │───────────────────────▸│ Hono API │────────▸│ buffer.csv │
│ (pramana)│ │ (CF Workers) │ │ user/*.json │
└──────────┘ └──────┬───────┘ └──────┬──────┘
│ cron │
▼ ▼
┌──────────────┐ ┌─────────────┐
│ compact │────────▸│ chart.json │
│ + aggregate │ │ archive/*.gz│
└──────────────┘ └──────┬──────┘
│
┌──────────┐ GET /api/data/chart │
│ Dashboard│◂───────────────────────────────────────────────────────┘
│ (Vite) │
└──────────┘
Data Flow
POST /api/submit— CLI sends eval results- Append to gzip-compressed CSV buffer in R2
- Update per-user summary (real-time)
- Daily cron compacts buffer → archive, rebuilds aggregate chart data
- Dashboard reads pre-aggregated JSON (single R2 GET, <50ms)
Design Decisions
- Stateless API — no database, no connection pools. CSV + JSON on R2.
- Append-only — immutable results. No edits, no deletes of submissions.
- Hash-based drift detection — output consistency tracking via SHA-256 hashes.
- Zero cost — Cloudflare Pages free tier + R2 free tier = $0/month.
- Privacy-preserving — individual submissions cannot be reconstructed from aggregates.
Stack
| Layer | Technology |
|---|---|
| API | Hono (Cloudflare Workers runtime) |
| Dashboard | Vite SPA (React + react-router) |
| Storage | Cloudflare R2 (native bindings, no S3 SDK) |
| Auth | jose JWT + OAuth (GitHub, Google) |
| Drift Detection | SHA-256 output hashing, per-prompt consistency tracking |
Pages
| Page | Content |
|---|---|
| Getting Started | Install CLI, run evals, submit, authenticate |
| API Reference | Endpoint schemas and response shapes |
| Methodology | Hash-based drift detection methodology |