Australia Patch 5, Zurich Patch 12
Standalone Application
AI Agent Eval Studio is a ServiceNow application for systematically testing, evaluating, and improving Now Assist AI agents and use cases. It automatically builds evaluation context from agent definitions, tools, documentation, and execution history; generates meaningful test cases; supports human review; executes approved evaluations; and scores results using deterministic validators and AI-powered judging.
The app gives teams a repeatable way to measure agent quality, identify regressions, validate business outcomes, and continuously improve AI experiences before they reach production.
- Context-aware evaluation generation Builds evaluation context from agent definitions, instructions, tools, documentation, and historical execution data.
- Automated test-case generation Generates business-level, agent-level, and tool-level evaluation suites tailored to the selected AI agent or use case.
- Sufficiency and clarification workflow Detects missing context and guides users through targeted clarification questions before generating evaluations.
- Human review and approval Review, edit, approve, or reject generated test cases before execution.
- Validator-based testing Supports deterministic checks for expected outcomes, tool usage, data changes, response quality, and multi-step behavior.
- AI-powered judging Uses an LLM judge to evaluate nuanced responses and business outcomes that cannot be validated through deterministic rules alone.
- Real-agent execution Runs approved evaluation cases against the configured ServiceNow AI agent or use case.
- Execution tracking and scoring Tracks evaluation progress, captures results, and combines deterministic and AI-judged scores.
- Regression detection Compares evaluation results across runs to identify performance changes and recurring failures.
- Comprehensive reporting Provides pass rates, failed cases, regression insights, validator results, and executive summaries.
- Reusable validator authoring Enables teams to create, review, and reuse custom validators for organization-specific behaviors.
- ServiceNow-native integration Built on ServiceNow scoped application architecture and integrated with Now Assist,
sn_aia, and GenAI capabilities.
- Initial release
- New
- Introduces AI Agent Eval Studio for creating, executing, and analyzing evaluations for ServiceNow Now Assist AI agents and use cases.
- Generates evaluation cases across business, agent, and tool levels using agent instructions, tool definitions, documentation, and execution history.
- Provides sufficiency checks and clarification questions to identify missing evaluation context before test generation.
- Enables users to review, edit, approve, and reject generated evaluation cases before execution.
- Supports deterministic validators for expected outcomes, data changes, tool usage, response content, and multi-step behavior.
- Provides AI-powered judging for evaluating nuanced agent responses and business outcomes.
- Executes approved evaluation cases against the selected AI agent or use case and tracks execution status.
- Combines validator results and AI judge scores to produce evaluation results and pass rates.
- Provides regression insights and reports with failed cases, score trends, and executive summaries.
- Supports reusable validator authoring for organization-specific agent behaviors and business requirements.
Not applicable for this application version.
- A ServiceNow instance with support for scoped applications and the Now Assist AI Agent framework.
- The Now Assist Skill and AI Agent Studio apps are required Now Assist components enabled and configured.
- Access to the ServiceNow tables and services used by the selected AI agents, use cases, tools, and execution plans.
- A configured AI agent or AI use case or Now Assist Skill with available instructions, tools, and execution capabilities.
- Appropriate user permissions to create and manage evaluation runs, review generated test cases, execute evaluations, author validators, and view reports.
- Access to the AI models and GenAI configurations required for evaluation generation and AI-powered judging.
- Sufficient instance capacity to execute evaluation cases and retain evaluation context, execution results, validator outcomes, and reports.
- Any external integrations or agent tools used during evaluation must be configured and accessible from the ServiceNow instance.
- Evaluation executions may invoke the selected agent's real tools. Administrators should review tool behavior and data-impact considerations before enabling execution in a production environment.