Roadmap

Use Google Cloud's Vertex AI models including Gemini and partner models in the playground, Model Hub, and through Gateway endpoints.

Filter and search for traces based on their annotations. Find traces with low scores or feedback quickly using the rebuilt filtering system.

Completely redesigned evaluation results dashboard with performance plots, side-by-side comparison, improved test cases view, focused detail view, configuration visibility, and run naming.

URLs now include workspace context, making them shareable between team members. Fixed workspace bugs with page refresh and workspace selection.

We improved the speed of the playground (creation of prompts, navigation, etc.) especially with hundreds of revisions.

You can view prompt and messages in markdown both in the playground and in the observability drawer.

You can now upload images to the playground and use them in your prompts.

Use structured output formats and generate multiple output fields (explanation, confidence, suggestions, issue categories) in LLM-as-a-judge evaluators.

Add Jinja2 template support to enable conditional logic, filters, and template blocks in prompts. The prompt type will be stored in the schema, and the SDK will handle rendering.

Add PDF support for models that support it (OpenAI, Gemini, etc.) through base64 encoding, URLs, or file IDs. Support extends to human evaluation for reviewing model responses on PDF inputs.

Create reusable prompt snippets that can be referenced across multiple prompts. Reference specific versions or always use the latest version to maintain consistency across prompt variants.

Adding the ability to configure evaluators (llm-as-a-judge or custom) and run them automatically on new traces.

Until now evaluations were only available as managed by Agenta. We are now adding the ability to run evaluations programmatically through the SDK.

We are adding the ability to filter traces by date range in the metrics dashboard.

Create folders and subfolders to organize prompts in the playground. Move prompts between folders and search within specific folders to structure prompt libraries.

Improve organization structure by adding projects and workspaces. Create projects for different products, set up workspaces for different environments or teams, and scope resources to specific workspaces.

Analyze prompts and suggest improvements based on best practices. Identify issues, propose refined versions, and allow users to accept, modify, or reject suggestions.

Add a button in observability to open any chat span directly in the playground. Creates a stateless playground session pre-filled with the exact prompt, configuration, and inputs for immediate iteration.

We are making it easy to use and navigate in the playground with large test sets .

Using test cases from different test sets is not possible right now in the Playground. We are adding the ability to append a single test case to a test set.

We are reworking the test set view to make it easier to visualize and edit test sets.

We are adding the ability to cache prompts in the SDK.

We are adding the ability to version test sets. This is useful for correctly comparing evaluation results.

We are adding the ability to tag traces, test sets, evaluations and prompts. This is useful for organizing and filtering your data.

We are adding the ability to use built-in LLM tools (e.g. web search) in the playground.