Skip to main content

Create Test Sets from Traces

Overview​

One of the most valuable sources of test cases is your production data. Traces captured in the Observability view represent real user interactions with your LLM application.


Adding a Single Trace​

To add a single trace to a test set:

  1. Navigate to the Observability view in Agenta
  2. Find a trace you want to add to a test set
  3. Click the Add to test set button at the top of the trace
  4. Choose to create a new test set or select an existing one
  5. Review the mapping between trace data and test set columns
    • Agenta will automatically map the inputs and outputs to appropriate columns
    • You can edit the expected answer if you don't agree with the output
  6. Click Save to add the trace to your test set

Adding Multiple Traces at Once​

To efficiently add multiple traces:

  1. In the Observability view, use the search function to filter traces
    • For example, search for specific response patterns like "I don't have enough information"
  2. Select all relevant traces by checking the boxes next to them
  3. Click Add to test set
  4. Choose an existing test set or create a new one
  5. Review the mapping for the traces
  6. Click Save to add all selected traces to your test set

Use cases​

Creating test sets from traces is particularly useful for:

  • Edge Cases: Capture unusual or problematic user interactions
  • Regression Testing: Save examples of correct behavior to prevent future regressions
  • Error Analysis: Collect failed cases to understand and fix issues

Next steps​