Skip to main content

Configuring Evaluators

Creating evaluators​

If you don't have evaluators yet, click Create new in the Evaluator section.

Each evaluator has:

  • Name - What you're measuring (e.g., "correctness")
  • Description - What the evaluator does
  • Feedback types - How evaluators will score responses

For example, a "correctness" evaluator might have:

  • is_correct - A yes/no question about accuracy
  • error_type - A multiple-choice field for categorizing mistakes

Available feedback types​

  • Boolean - Yes/no questions
  • Integer - Whole number ratings
  • Decimal - Precise numerical scores
  • Single-choice - Pick one option
  • Multi-choice - Pick multiple options
  • String - Free-text comments or notes
tip

Evaluators can include multiple related feedback types. For example:

Correctness evaluator:

  • is_correct - Yes/no question about accuracy
  • error_type - Multiple-choice field to categorize mistakes (only if incorrect)

Style adherence evaluator:

  • is_adherent - Yes/no question about style compliance
  • comment - Text field explaining why the style doesn't match (if needed)

This grouping helps you evaluate different aspects of your LLM's performance in an organized way.

Selecting evaluators​

After creating evaluators:

  1. Select the evaluators you want to use
  2. You can use multiple evaluators in a single evaluation
  3. Each evaluator will appear in the annotation interface

Next steps​