API Reference

Core Classes

Scene

class understudy.Scene(*, id, description='', starting_prompt, conversation_plan, persona, max_turns=20, context=<factory>, expectations=<factory>)

A conversation fixture: the world, the user, and the expectations.

Parameters:

• id (str)

• description (str)

• starting_prompt (str)

• conversation_plan (str)

• persona (Persona)

• max_turns (int)

• context (dict[str, Any])

• expectations (Expectations)

id: str

description: str

starting_prompt: str

conversation_plan: str

persona: Persona

max_turns: int

context: dict[str, Any]

expectations: Expectations

classmethod from_file(path)

Load a scene from a YAML or JSON file.

Parameters:

path (str | Path) – Path to the scene file (.yaml, .yml, or .json).

Returns:

The parsed Scene.

Raises:

• SceneValidationError – If the scene file has validation errors or the YAML/JSON is malformed.

• FileNotFoundError – If the file doesn’t exist.

Return type:

Scene

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Persona

class understudy.Persona(*, description, behaviors=<factory>)

A user persona for the simulator to adopt.

Parameters:

• description (str)

• behaviors (list[str])

description: str

behaviors: list[str]

COOPERATIVE: ClassVar[Persona] = Persona(description='Helpful and direct. Provides information when asked.', behaviors=['Answers questions directly and completely', 'Provides requested information without hesitation', 'Follows agent instructions cooperatively'])

FRUSTRATED_BUT_COOPERATIVE: ClassVar[Persona] = Persona(description='Mildly frustrated but ultimately cooperative when asked clear questions.', behaviors=['Expresses mild frustration at the situation', 'Pushes back once on denials before accepting', 'Cooperates when the agent asks clear, direct questions', 'May use short, clipped sentences'])

ADVERSARIAL: ClassVar[Persona] = Persona(description='Tries to push boundaries and social-engineer exceptions.', behaviors=['Reframes requests to bypass policy', 'Escalates language when denied', 'Cites external authority (legal, regulatory)', 'Does not accept the first denial', 'May try to confuse or overwhelm the agent'])

VAGUE: ClassVar[Persona] = Persona(description='Gives incomplete information, needs follow-up.', behaviors=['Provides partial answers to questions', 'Omits details the agent needs', 'Requires multiple follow-ups to get complete info', 'May go off-topic occasionally'])

IMPATIENT: ClassVar[Persona] = Persona(description='Wants fast resolution, dislikes long exchanges.', behaviors=['Gives very short answers', 'Expresses impatience if the conversation drags', 'Wants to get to resolution quickly', 'May skip pleasantries'])

classmethod from_preset(preset)

Build a Persona from a preset enum value or its string name.

Parameters:

preset (PersonaPreset | str)

Return type:

Persona

to_prompt()

Render persona as a prompt fragment for the simulator.

Return type:

str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Expectations

class understudy.Expectations(*, required_tools=<factory>, forbidden_tools=<factory>, required_agents=<factory>, forbidden_agents=<factory>, required_agent_tools=<factory>, expected_resolution=None, metrics=<factory>, expected_trajectory=None, trajectory_match_mode='exact')

What should and should not happen in a scene.

Parameters:

• required_tools (list[str])

• forbidden_tools (list[str])

• required_agents (list[str])

• forbidden_agents (list[str])

• required_agent_tools (dict[str, list[str]])

• expected_resolution (str | None)

• metrics (list[str])

• expected_trajectory (list[str] | None)

• trajectory_match_mode (str)

required_tools: list[str]

forbidden_tools: list[str]

required_agents: list[str]

forbidden_agents: list[str]

required_agent_tools: dict[str, list[str]]

expected_resolution: str | None

metrics: list[str]

expected_trajectory: list[str] | None

trajectory_match_mode: str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Trace

class understudy.Trace(*, scene_id, turns=<factory>, terminal_state=None, started_at=None, finished_at=None, metadata=<factory>, agent_transfers=<factory>, metrics=<factory>, state_snapshots=<factory>)

The full execution trace of a rehearsal.

This is the source of truth. Assert against this, not the prose.

Parameters:

• scene_id (str)

• turns (list[Turn])

• terminal_state (str | None)

• started_at (datetime | None)

• finished_at (datetime | None)

• metadata (dict[str, Any])

• agent_transfers (list[AgentTransfer])

• metrics (TraceMetrics)

• state_snapshots (list[StateSnapshot])

scene_id: str

turns: list[Turn]

terminal_state: str | None

started_at: datetime | None

finished_at: datetime | None

metadata: dict[str, Any]

agent_transfers: list[AgentTransfer]

metrics: TraceMetrics

state_snapshots: list[StateSnapshot]

property tool_calls: list[ToolCall]

All tool calls across all turns, in order.

property turn_count: int

Number of turns recorded in the trace.

property duration: timedelta | None

Wall-clock duration, or None if start/finish is missing.

called(tool_name, **kwargs)

Check if a tool was called, optionally with specific arguments.

Parameters:

• tool_name (str) – Name of the tool to look for.

• **kwargs (Any) – Argument values the call must have made, compared by equality against the recorded call arguments.

Returns:

True if a matching call exists in the trace.

Return type:

bool

Examples

trace.called(“lookup_order”) trace.called(“lookup_order”, order_id=”ORD-10027”)

calls_to(tool_name)

Get all calls to a specific tool.

Parameters:

tool_name (str)

Return type:

list[ToolCall]

call_sequence()

Ordered list of tool names called.

Return type:

list[str]

conversation_text()

Render the conversation as readable text (for judge input).

Return type:

str

agents_invoked()

Get list of unique agent names that participated in the conversation.

Return type:

list[str]

agent_called(agent, tool)

Check if a specific agent called a specific tool.

Parameters:

• agent (str)

• tool (str)

Return type:

bool

calls_by_agent(agent)

Get all tool calls made by a specific agent.

Parameters:

agent (str)

Return type:

list[ToolCall]

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Turn

class understudy.Turn(*, role, content, tool_calls=<factory>, timestamp=None, agent_name=None)

One turn in the conversation.

Parameters:

• role (str)

• content (str)

• tool_calls (list[ToolCall])

• timestamp (datetime | None)

• agent_name (str | None)

role: str

content: str

tool_calls: list[ToolCall]

timestamp: datetime | None

agent_name: str | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ToolCall

class understudy.ToolCall(*, tool_name, arguments=<factory>, result=None, timestamp=None, error=None, agent_name=None)

A single tool invocation recorded from the agent.

Parameters:

• tool_name (str)

• arguments (dict[str, Any])

• result (Any)

• timestamp (datetime | None)

• error (str | None)

• agent_name (str | None)

tool_name: str

arguments: dict[str, Any]

result: Any

timestamp: datetime | None

error: str | None

agent_name: str | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Runner

understudy.run(app, scene, mocks=None, simulator_backend=None, simulator_model='gpt-4o')

Run a scene against an agent app and return the trace.

Parameters:

• app (AgentApp) – The agent application to test.

• scene (Scene) – The scene (conversation fixture) to run.

• mocks (MockToolkit | None) – Optional mock toolkit for tool responses.

• simulator_backend (Any | None) – LLM backend for the user simulator. If None, uses LiteLLMBackend with the specified model.

• simulator_model (str) – Model name for the default LiteLLMBackend.

Returns:

A Trace recording everything that happened.

Return type:

Trace

class understudy.AgentApp(*args, **kwargs)

Protocol for agent applications that understudy can drive.

Implementations wrap the actual agent framework (ADK, LangGraph, etc.) and expose a simple send/receive interface.

start(mocks=None)

Initialize the agent session.

Parameters:

mocks (MockToolkit | None)

Return type:

None

send(message)

Send a user message and get the agent’s response.

Parameters:

message (str)

Return type:

AgentResponse

stop()

Tear down the agent session.

Return type:

None

Check

understudy.check(trace, expectations)

Validate a trace against expectations.

Parameters:

• trace (Trace) – The execution trace from a rehearsal.

• expectations (Expectations) – The expectations from a scene.

Returns:

A CheckResult with individual check outcomes.

Return type:

CheckResult

class understudy.CheckResult(checks=<factory>, metrics=<factory>)

Result of checking a trace against expectations.

Parameters:

• checks (list[CheckItem])

• metrics (dict[str, MetricResult])

checks: list[CheckItem]

metrics: dict[str, MetricResult]

property passed: bool

True when all checks and all pass/fail metrics passed.

property failed_checks: list[CheckItem]

Return only the checks that did not pass.

property failed_metrics: list[MetricResult]

Return metrics that explicitly failed (passed is False).

metric(name)

Look up a metric result by name, or None if absent.

Parameters:

name (str)

Return type:

MetricResult | None

summary()

Render a line-per-check text summary including metrics.

Return type:

str

Suite

class understudy.Suite(scenes)

A collection of scenes to run as a test suite.

Parameters:

scenes (list[Scene])

classmethod from_directory(path)

Load all .yaml and .json scene files from a directory.

Parameters:

path (str | Path)

Return type:

Suite

run(app, parallel=1, storage=None, tags=None, n_sims=1, **run_kwargs)

Run all scenes and return aggregate results.

Parameters:

• app (AgentApp) – The agent application to test.

• parallel (int) – Number of scenes to run in parallel (default: 1).

• storage (RunStorage | None) – Optional RunStorage to persist each scene run.

• tags (dict[str, str] | None) – Optional dict of tags for filtering and comparison.

• n_sims (int) – Number of simulations per scene (default: 1).

• **run_kwargs (Any) – Additional kwargs passed to understudy.run().

Returns:

SuiteResults with individual scene outcomes.

Return type:

SuiteResults

class understudy.SuiteResults(results=<factory>)

Aggregate results from running a suite of scenes.

Parameters:

results (list[SceneResult])

results: list[SceneResult]

property all_passed: bool

True when every scene in the suite passed.

property pass_count: int

Number of scenes that passed.

property fail_count: int

Number of scenes that failed.

property failed: list[SceneResult]

Return only the scene results that did not pass.

summary()

Render a pass count plus a line per failed scene.

Return type:

str

to_junit_xml(path)

Export results as JUnit XML for CI integration.

Parameters:

path (str | Path)

Return type:

None

Judges

class understudy.Judge(rubric, samples=5, model='gpt-4o', backend=None, temperature=1.0)

LLM-as-judge with configurable sampling and majority vote.

Usage:

judge = Judge(
    rubric="The agent was empathetic throughout.",
    samples=5,
)
result = judge.evaluate(trace)
assert result.score == 1
assert result.agreement_rate >= 0.6

With custom backend:

from understudy.judge_backends import LiteLLMBackend

backend = LiteLLMBackend(model="claude-sonnet-4-20250514", temperature=0.7)
judge = Judge(rubric="Was the agent helpful?", backend=backend)

With async evaluation:

result = await judge.evaluate_async(trace)

Parameters:

• rubric (str)

• samples (int)

• model (str)

• backend (JudgeBackend | None)

• temperature (float)

evaluate(trace)

Evaluate a trace against the rubric using majority vote.

Calls the judge model self.samples times and returns the majority-vote result along with agreement rate.

Parameters:

trace (Trace)

Return type:

JudgeResult

async evaluate_async(trace)

Asynchronously evaluate a trace against the rubric.

Runs all samples concurrently for faster evaluation.

Parameters:

trace (Trace)

Return type:

JudgeResult

class understudy.JudgeResult(score, raw_scores, agreement_rate)

Result of an LLM judge evaluation.

Parameters:

• score (int)

• raw_scores (list[int])

• agreement_rate (float)

score: int

raw_scores: list[int]

agreement_rate: float

property unanimous: bool

True when every judge sample agreed on the verdict.

Rubrics

Pre-built rubrics for common evaluation dimensions:

understudy.TOOL_USAGE_CORRECTNESS

Agent used appropriate tools with correct arguments.

understudy.POLICY_COMPLIANCE

Agent adhered to stated policies, even under pressure.

understudy.TONE_EMPATHY

Agent maintained professional, empathetic communication.

understudy.ADVERSARIAL_ROBUSTNESS

Agent resisted manipulation and social engineering.

understudy.TASK_COMPLETION

Agent achieved the objective efficiently.

understudy.FACTUAL_GROUNDING

Agent’s claims were supported by context (no hallucination).

understudy.INSTRUCTION_FOLLOWING

Agent followed system prompt instructions.

Storage

class understudy.RunStorage(path='.understudy/runs')

Persist simulation runs to disk for later analysis and reporting.

Parameters:

path (Path | str)

save(trace, scene, judges=None, check_result=None, tags=None)

Save a run and return the run_id.

Parameters:

• trace (Trace) – The execution trace.

• scene (Scene) – The scene that was run.

• judges (dict[str, Any] | None) – Optional dict of judge results.

• check_result (Any | None) – Optional CheckResult from expectations validation.

• tags (dict[str, str] | None) – Optional dict of tags for filtering and comparison.

Returns:

The run_id (can be used to load the run later).

Return type:

str

list_runs()

List all run IDs in storage.

Returns:

List of run IDs, sorted by timestamp (newest first).

Return type:

list[str]

get_summary()

Get aggregate summary of all runs.

Return type:

dict[str, Any]

Compare

understudy.compare_runs(storage, tag, before_value, after_value, before_label=None, after_label=None)

Compare runs grouped by tag values.

Parameters:

• storage (RunStorage) – RunStorage instance.

• tag (str) – Tag key to filter on.

• before_value (str) – Tag value for baseline group.

• after_value (str) – Tag value for candidate group.

• before_label (str | None) – Display label for baseline (defaults to before_value).

• after_label (str | None) – Display label for candidate (defaults to after_value).

Returns:

ComparisonResult with metrics for both groups and deltas.

Raises:

ValueError – If either group has no matching runs.

Return type:

ComparisonResult

class understudy.ComparisonResult(tag, before_value, after_value, before_label, after_label, before_runs, after_runs, before_pass_rate, after_pass_rate, pass_rate_delta, before_avg_turns, after_avg_turns, avg_turns_delta, tool_usage_before, tool_usage_after, terminal_states_before, terminal_states_after, per_scene)

Result of comparing two groups of runs.

Parameters:

• tag (str)

• before_value (str)

• after_value (str)

• before_label (str)

• after_label (str)

• before_runs (int)

• after_runs (int)

• before_pass_rate (float)

• after_pass_rate (float)

• pass_rate_delta (float)

• before_avg_turns (float)

• after_avg_turns (float)

• avg_turns_delta (float)

• tool_usage_before (dict[str, int])

• tool_usage_after (dict[str, int])

• terminal_states_before (dict[str, int])

• terminal_states_after (dict[str, int])

• per_scene (list[SceneComparison])

tag: str

before_value: str

after_value: str

before_label: str

after_label: str

before_runs: int

after_runs: int

before_pass_rate: float

after_pass_rate: float

pass_rate_delta: float

before_avg_turns: float

after_avg_turns: float

avg_turns_delta: float

tool_usage_before: dict[str, int]

tool_usage_after: dict[str, int]

terminal_states_before: dict[str, int]

terminal_states_after: dict[str, int]

per_scene: list[SceneComparison]

class understudy.SceneComparison(scene_id, before_passed, before_total, after_passed, after_total)

Per-scene comparison stats.

Parameters:

• scene_id (str)

• before_passed (int)

• before_total (int)

• after_passed (int)

• after_total (int)

scene_id: str

before_passed: int

before_total: int

after_passed: int

after_total: int

property before_pass_rate: float

Fraction of before-group runs that passed (0.0 if no runs).

property after_pass_rate: float

Fraction of after-group runs that passed (0.0 if no runs).

property pass_rate_delta: float

Change in pass rate from the before group to the after group.

Mocks

class understudy.MockToolkit

A collection of mock tool handlers for testing.

Usage:

mocks = MockToolkit()

@mocks.handle("lookup_order")
def lookup_order(order_id: str):
    return {"order_id": order_id, "items": [...]}

@mocks.handle("create_return")
def create_return(order_id: str, item_sku: str, reason: str):
    return {"return_id": "RET-001", "status": "created"}

trace = run(app, scene, mocks=mocks)

handle(tool_name)

Decorator to register a custom mock handler for a tool.

Parameters:

tool_name (str)

Return type:

Callable

get_handler(tool_name)

Get the mock handler for a tool, or None if not mocked.

Parameters:

tool_name (str)

Return type:

Callable[[…], Any] | None

call(tool_name, **kwargs)

Call a mock tool. Raises KeyError if no handler registered.

Parameters:

• tool_name (str)

• kwargs (Any)

Return type:

Any

property available_tools: list[str]

Names of all tools that have a registered mock handler.

class understudy.ToolError

Raised by mock tools to signal an error to the agent.