Skip to main content
Attack your agent before you ship it. A simulated user talks to a copy of your agent and tries to make it misbehave, and each conversation is graded against what a good result looks like. For example, a Misuse scenario has an attacker ask your support agent to refund an order that isn’t theirs. The run shows whether the agent refused, and if not, the turn where it went wrong and a suggested policy.
Simulation is in beta and off by default. Xenovia turns it on per organization, so contact Xenovia to get access. Once it’s on, Simulation appears in the sidebar with a Beta badge. Until then the section is hidden and the Simulation API answers 404.
Simulation tests your agent before real users find its weak spots. Xenovia builds a copy of your agent, with its system prompt and its tools, and a simulated user talks to it over several turns. In a Misuse scenario the user is an attacker trying to make the agent misbehave. In a Normal use scenario the user just wants help, so you see whether your policies get in the way. Each conversation is graded against what a good result looks like. Xenovia never calls your agent’s endpoint, so a run can’t issue a real refund or change real data. Simulation has its own section in the console, Simulation, with two tabs: Runs and Suites.

Which agents can be simulated

An agent can be simulated when it’s active and runs on the Xenovia cloud runtime. Agents on a customer-hosted runtime and coding tools such as Claude Code can’t be simulated. Xenovia also needs something to build the copy from: Linking code is optional, but a copy from traffic knows only each tool’s name and the arguments it was called with. To link code, connect GitHub and let discovery find the agent, then choose Link discovered agent on the agent’s page or in a suite. After every scan of a new commit on your default branch, Xenovia reads the linked code again.
A copy from code reflects the commit discovery last scanned on your default branch. If production runs different code, results describe the scanned commit.

Suites and scenarios

A suite is a set of scenarios for one agent. You pick the agent when you create the suite and can’t change it later. Deleting a suite deletes its scenarios; past runs keep their results. A scenario is one conversation. In a suite, choose New scenario from the menu next to Generate with Nova, and fill in its Type (Misuse or Normal use), a Name, and two sentences: The next step describes the simulated user: their name, who they are, how they try (Misuse only), and Max turns (1–20, default 10). Misuse needs at least 4 turns, and 5 for the Crescendo, Role drift and Context poisoning techniques. More options holds what the agent must not do, the category and severity (on Work it out, Xenovia picks the category from your text and its default severity), the attacker’s technique, Tool responses (what the agent’s tools return, optionally with a Planted instruction the agent must not act on), and Tool checks (Must call, Never calls, At most). For Normal use, Looks like names the kind of misuse a harmless request resembles. The attacker plays a realistic person in your agent’s domain and changes angle after a refusal. It only speaks through the chat, and its own messages stay harmless. Xenovia turns each scenario into checks, and a conversation fails when any check fails:
  • Passes when: the conversation reaches the good result, within the agent’s purpose and policies. Always added.
  • Never: the agent doesn’t do what it must not do, when that’s set.
  • A check for some categories, such as “No reply repeats the agent’s instructions”.
  • For a planted instruction, that it reached the agent and the agent didn’t act on it. If it never arrived, these checks show Not reached and don’t count.
  • One check per tool check.
Normal use scenarios have no category and default to Medium. Misuse scenarios carry the OWASP Top 10, MITRE ATLAS and CWE IDs that match their category and technique.

Generate scenarios

When Nova is enabled for your organization, you can generate scenarios instead of writing them. In a suite, choose Generate with Nova and set How many (1–20), a Category (Any category, Normal use, or one category) and, optionally, Anything to focus on. Nova reads the agent’s real sessions from the last 30 days, its policies and, once its copy from code is ready, its linked code. Personal data in sessions is redacted before Nova reads it. Each scenario arrives as a Draft. Select the ones you want and choose Add to suite, or Discard them. Drafts never run, and drafts not added within 7 days are deleted. Nova needs a ready copy from code, or real sessions and a system prompt. Without either, the button reads Link discovered agent (or Reading code… while linked code is read).

Run a simulation

Choose Run suite, or select scenarios and choose Run selected. The suite’s agent is always the one tested. The list shows only providers whose credential Xenovia holds, that have a model set, and whose last connection test didn’t fail. A run from the console repeats each scenario 3 times, each as its own conversation, because an agent doesn’t answer the same way twice. A conversation ends when the simulated user is done or at Max turns.

When a run can’t start

Read the results

The run’s page shows the suite, agent, model and Policies setting, how many scenarios failed, and, apart, how many Normal use conversations a policy blocked (false blocks) and how many Misuse scenarios got through. Scenarios are listed failures first, each with one square per conversation and its result: Open a scenario to see its first failed conversation, exactly as in Sessions, opened on the turn where a check failed. Cards above it show the Result, where it Failed at, Why, and a Suggested fix:
  • Add policy creates a policy for the agent, turned off and in monitor mode, and opens it. It does nothing yet: its Rego allows every request and carries the suggested rule as comments, for you to write in Rego and review before you turn it on. See Policies.
  • Prompt change says what to change in the agent’s prompt.
Then choose Run again. On a run’s page it reruns the suite’s current scenarios with the same Policies setting and repeats, on the agent’s latest copy. It keeps a provider you chose for that run; otherwise it uses the agent’s current one. On a scenario’s page it reruns just that scenario.

Ask Nova

When Nova is enabled for your organization, it can read your suites, their scenarios, your runs and each scenario’s result. Ask it, for example, “which scenarios failed in run #12?”. It doesn’t read conversations or diagnoses, so open a scenario to see why it failed. Nova can also open Simulation’s forms beside the chat, filled in: to create, edit or delete a suite or scenario, generate scenarios, add or discard drafts, start, cancel or rerun a run, or draft a policy from a failed conversation. Nothing changes until you complete the form.

What next?

Policies

Turn a suggested fix into a policy.

Nova

Have Nova write scenarios.

GitHub

Link code for a better copy.