The AI Specialist Onboarding Tool collapses AI Specialist readiness from weeks of manual scoping into days of guided, validated setup. Point it at your closed-incident history and the tool identifies the specific issue types your L1 AI Specialist can resolve, validates each one against your own KB and catalog content, runs simulated resolutions scored by an LLM judge, and generates the routing criteria that send the right incidents — and only the right incidents — to the AI Specialist in production.
As a by-product, it surfaces where content gaps exist, pointing to the KBs and catalog items that would unlock the next wave of automation.
Deployment guidance: Run this tool on a sub-production instance (a production clone with recent incident, KB, and catalog data). The simulation stage generates evaluation incidents against the AI Specialist, which is not appropriate for production environments. Generated routing criteria can then be promoted to production through your standard update set process.
Here is the documentation on how to use this tool - 2026-Jul-2-Onboarding_tool.docx
1. Ability to identify content coverage and gaps
2. Run the ZTSD evals and followed by LLM judge evaluation of ZTSD response
3. Recommended routing criteria on what types of incidents can be routed to L1 AIS
- Direct evaluation flow — Clustering and coverage analysis are now optional steps before an evaluation. From the data set selection page, users can skip clustering and go straight to the Simulate page, which lists the selected incidents with quick and advanced filtering plus check-box selection so specific incidents can be chosen and evaluated. Judge review of each response and the evaluation run history behave as before; Generate Routing Criteria is unavailable when clustering has been skipped.
- Cancel an evaluation run — Evaluation runs can now be cancelled from the evaluation screen. Cancelling stops any incidents still being processed in the background, and cancelled runs cannot be reopened for review. Users no longer need to wait for a long-running evaluation to finish before starting again.
- Configurable evaluation batch size — The number of incidents evaluated concurrently in a run is now controlled by a configurable property rather than a fixed value, with a default of 25. Administrators can raise or lower it to match the throughput of their instance.
- Partially resolved incidents out of scope — Incidents judged as partially resolved are now treated as out of scope during routing criteria generation, instead of being grouped with fully resolved incidents. Only fully resolved incidents remain in scope, and partially resolved incidents are expected to route to a human. The generated in-scope and out-of-scope criteria text reflects this tightened definition.
NA