Skip to main content

Specifications

Purpose​

Use Specifications to define the reusable test surface for an evaluation.

When to use it​

Create a specification when you want a named, repeatable setup that can be attached to one or more evals.

Prerequisites​

  • You have a project.
  • You have at least one primary dataset.
  • You know whether you want attack methods, adversarial datasets, or judge-based scoring.

Steps​

  1. Open Specs inside a project.
  2. Create a new specification.
  3. Choose the primary dataset you want to test.
  4. Add any attack methods you want Spec27 to use.
  5. If relevant, include adversarial dataset selections.
  6. If the workflow is judge-based, choose the judge configuration that should score outputs.
  7. Save the specification and open the detail page.
  8. Review the status, datasets, linked eval usage, and results summary.

Expected result​

You have a reusable specification that describes what should be tested and how adversarial coverage should be applied.

What to know​

  • A specification can be reused across multiple evals.
  • A specification can include multiple attack methods for the same primary dataset.
  • Plain-language definition: a specification is the evaluation recipe, not the run itself.