Tribunal
System-prompt grading instrument
“The model judges. The rubric decides.”
An instrument that scores LLM system prompts on six dimensions. LLM judges find the evidence, and a deterministic rubric turns that evidence into the score.
- Dimensions
- 6
- Judge calls
- 1 / dim
- Numbers from LLM
- 0
CLASSIFIED · built for a client. Shown here as architecture and outcomes only: no names, no code.
The problem
“Is this system prompt safe?” usually gets a vibe-based answer. I wanted a repeatable scorecard.
The pipeline
regex rules → LLM judge (one call per dimension) → omission checks
→ quote verification → dedupe → gates → scorecardThe six dimensions are truth, steering, identity, privacy, safety and fairness.
The core rule
The model never picks a number.
LLM judges find evidence and must quote it. Every quote is checked against the source. The score comes only from a YAML rubric applied to verified findings. That makes the results repeatable and explainable, and hard to game.
Keeping it honest
A gold set and a corpus of real prompts are kept as fixtures. Each change to the instrument is measured against them.