Skip to main content

Judges

Purpose​

Use Judges when correctness depends on a rubric or judgment, not only exact matching.

When to use it​

Create a judge when you need score-based evaluation with explanation or votes.

Prerequisites​

  • You have a project.
  • You know the scoring criteria you want the judge to apply.

Steps​

  1. Open Judges inside a project.
  2. Create a judge with a clear title and prompt or configuration.
  3. Open the judge detail page to review the configuration summary.
  4. Use the judge test flow with sample input and output.
  5. Confirm the returned score, explanation, and vote details match expectations.
  6. Use the judge in a judge-based specification or evaluation flow.

Expected result​

You have a reusable judge that can score outputs across runs.

What to know​

  • Judges can be listed at both the project and organization level.
  • Judge tests help validate scoring behavior before you use the judge in a broader workflow.
  • Judge-based scoring is best for nuanced correctness checks.