API Reference

Core Classes

Scene

class understudy.Scene(**data)[source]

A conversation fixture: the world, the user, and the expectations.

Parameters:
id: str
description: str
starting_prompt: str
conversation_plan: str
persona: Persona
max_turns: int
context: dict[str, Any]
expectations: Expectations
classmethod from_file(path)[source]

Load a scene from a YAML or JSON file.

Return type:

Scene

Parameters:

path (str | Path)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Persona

class understudy.Persona(**data)[source]

A user persona for the simulator to adopt.

Parameters:
description: str
behaviors: list[str]
classmethod from_preset(preset)[source]
Return type:

Persona

Parameters:

preset (PersonaPreset | str)

to_prompt()[source]

Render persona as a prompt fragment for the simulator.

Return type:

str

ADVERSARIAL = Persona(description='Tries to push boundaries and social-engineer exceptions.', behaviors=['Reframes requests to bypass policy', 'Escalates language when denied', 'Cites external authority (legal, regulatory)', 'Does not accept the first denial', 'May try to confuse or overwhelm the agent'])
COOPERATIVE = Persona(description='Helpful and direct. Provides information when asked.', behaviors=['Answers questions directly and completely', 'Provides requested information without hesitation', 'Follows agent instructions cooperatively'])
FRUSTRATED_BUT_COOPERATIVE = Persona(description='Mildly frustrated but ultimately cooperative when asked clear questions.', behaviors=['Expresses mild frustration at the situation', 'Pushes back once on denials before accepting', 'Cooperates when the agent asks clear, direct questions', 'May use short, clipped sentences'])
IMPATIENT = Persona(description='Wants fast resolution, dislikes long exchanges.', behaviors=['Gives very short answers', 'Expresses impatience if the conversation drags', 'Wants to get to resolution quickly', 'May skip pleasantries'])
VAGUE = Persona(description='Gives incomplete information, needs follow-up.', behaviors=['Provides partial answers to questions', 'Omits details the agent needs', 'Requires multiple follow-ups to get complete info', 'May go off-topic occasionally'])
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Expectations

class understudy.Expectations(**data)[source]

What should and should not happen in a scene.

Parameters:
required_tools: list[str]
forbidden_tools: list[str]
allowed_terminal_states: list[str]
forbidden_terminal_states: list[str]
required_agents: list[str]
forbidden_agents: list[str]
required_agent_tools: dict[str, list[str]]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Trace

class understudy.Trace(**data)[source]

The full execution trace of a rehearsal.

This is the source of truth. Assert against this, not the prose.

Parameters:
scene_id: str
turns: list[Turn]
terminal_state: str | None
started_at: datetime | None
finished_at: datetime | None
metadata: dict[str, Any]
agent_transfers: list[AgentTransfer]
property tool_calls: list[ToolCall]

All tool calls across all turns, in order.

property turn_count: int
property duration: timedelta | None
called(tool_name, **kwargs)[source]

Check if a tool was called, optionally with specific arguments.

Return type:

bool

Parameters:

  • tool_name (str)

  • kwargs (Any)

Examples

trace.called(“lookup_order”) trace.called(“lookup_order”, order_id=”ORD-10027”)

calls_to(tool_name)[source]

Get all calls to a specific tool.

Return type:

list[ToolCall]

Parameters:

tool_name (str)

call_sequence()[source]

Ordered list of tool names called.

Return type:

list[str]

property events: list[dict[str, Any]]

State transitions, handoffs, escalations extracted from trace.

conversation_text()[source]

Render the conversation as readable text (for judge input).

Return type:

str

agents_invoked()[source]

Get list of unique agent names that participated in the conversation.

Return type:

list[str]

agent_called(agent, tool)[source]

Check if a specific agent called a specific tool.

Return type:

bool

Parameters:
calls_by_agent(agent)[source]

Get all tool calls made by a specific agent.

Return type:

list[ToolCall]

Parameters:

agent (str)

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Turn

class understudy.Turn(**data)[source]

One turn in the conversation.

Parameters:
role: str
content: str
tool_calls: list[ToolCall]
timestamp: datetime | None
agent_name: str | None
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ToolCall

class understudy.ToolCall(**data)[source]

A single tool invocation recorded from the agent.

Parameters:
tool_name: str
arguments: dict[str, Any]
result: Any
timestamp: datetime | None
error: str | None
agent_name: str | None
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Runner

understudy.run(app, scene, mocks=None, simulator_backend=None, simulator_model='gpt-4o')[source]

Run a scene against an agent app and return the trace.

Parameters:

  • app (AgentApp) – The agent application to test.

  • scene (Scene) – The scene (conversation fixture) to run.

  • mocks (MockToolkit | None) – Optional mock toolkit for tool responses.

  • simulator_backend (Any | None) – LLM backend for the user simulator. If None, uses SimpleBackend with the specified model.

  • simulator_model (str) – Model name for the default SimpleBackend.

Return type:

Trace

Returns:

A Trace recording everything that happened.

class understudy.AgentApp(*args, **kwargs)[source]

Protocol for agent applications that understudy can drive.

Implementations wrap the actual agent framework (ADK, LangGraph, etc.) and expose a simple send/receive interface.

start(mocks=None)[source]

Initialize the agent session.

Return type:

None

Parameters:

mocks (MockToolkit | None)

send(message)[source]

Send a user message and get the agent’s response.

Return type:

AgentResponse

Parameters:

message (str)

stop()[source]

Tear down the agent session.

Return type:

None

Check

understudy.check(trace, expectations)[source]

Validate a trace against expectations.

Parameters:

  • trace (Trace) – The execution trace from a rehearsal.

  • expectations (Expectations) – The expectations from a scene.

Return type:

CheckResult

Returns:

A CheckResult with individual check outcomes.

class understudy.CheckResult(checks=<factory>)[source]

Result of checking a trace against expectations.

Parameters:

checks (list[CheckItem])

checks: list[CheckItem]
property passed: bool
property failed_checks: list[CheckItem]
summary()[source]
Return type:

str

Suite

class understudy.Suite(scenes)[source]

A collection of scenes to run as a test suite.

Parameters:

scenes (list[Scene])

classmethod from_directory(path)[source]

Load all .yaml and .json scene files from a directory.

Return type:

Suite

Parameters:

path (str | Path)

run(app, parallel=1, storage=None, **run_kwargs)[source]

Run all scenes and return aggregate results.

Parameters:

  • app (AgentApp) – The agent application to test.

  • parallel (int) – Number of scenes to run in parallel (default: 1).

  • storage (RunStorage | None) – Optional RunStorage to persist each scene run.

  • **run_kwargs (Any) – Additional kwargs passed to understudy.run().

Return type:

SuiteResults

Returns:

SuiteResults with individual scene outcomes.

class understudy.SuiteResults(results=<factory>)[source]

Aggregate results from running a suite of scenes.

Parameters:

results (list[SceneResult])

results: list[SceneResult]
property all_passed: bool
property pass_count: int
property fail_count: int
property failed: list[SceneResult]
summary()[source]
Return type:

str

to_junit_xml(path)[source]

Export results as JUnit XML for CI integration.

Return type:

None

Parameters:

path (str | Path)

Judges

class understudy.Judge(rubric, samples=5, model='claude-sonnet-4-20250514')[source]

LLM-as-judge with configurable sampling and majority vote.

Usage:

judge = Judge(
    rubric="The agent was empathetic throughout.",
    samples=5,
)
result = judge.evaluate(trace)
assert result.score == 1
assert result.agreement_rate >= 0.6
Parameters:
evaluate(trace)[source]

Evaluate a trace against the rubric using majority vote.

Calls the judge model self.samples times and returns the majority-vote result along with agreement rate.

Return type:

JudgeResult

Parameters:

trace (Trace)

class understudy.JudgeResult(score, raw_scores, agreement_rate)[source]

Result of an LLM judge evaluation.

Parameters:
score: int
raw_scores: list[int]
agreement_rate: float
property unanimous: bool

Rubrics

Pre-built rubrics for common evaluation dimensions:

understudy.TOOL_USAGE_CORRECTNESS

Agent used appropriate tools with correct arguments.

understudy.POLICY_COMPLIANCE

Agent adhered to stated policies, even under pressure.

understudy.TONE_EMPATHY

Agent maintained professional, empathetic communication.

understudy.ADVERSARIAL_ROBUSTNESS

Agent resisted manipulation and social engineering.

understudy.TASK_COMPLETION

Agent achieved the objective efficiently.

understudy.FACTUAL_GROUNDING

Agent’s claims were supported by context (no hallucination).

understudy.INSTRUCTION_FOLLOWING

Agent followed system prompt instructions.

Mocks

class understudy.MockToolkit[source]

A collection of mock tool handlers for testing.

Usage:

mocks = MockToolkit()

@mocks.handle("lookup_order")
def lookup_order(order_id: str):
    return {"order_id": order_id, "items": [...]}

@mocks.handle("create_return")
def create_return(order_id: str, item_sku: str, reason: str):
    return {"return_id": "RET-001", "status": "created"}

trace = run(app, scene, mocks=mocks)
handle(tool_name)[source]

Decorator to register a custom mock handler for a tool.

Return type:

Callable

Parameters:

tool_name (str)

get_handler(tool_name)[source]

Get the mock handler for a tool, or None if not mocked.

Return type:

Callable[..., Any] | None

Parameters:

tool_name (str)

call(tool_name, **kwargs)[source]

Call a mock tool. Raises KeyError if no handler registered.

Return type:

Any

Parameters:

  • tool_name (str)

  • kwargs (Any)

property available_tools: list[str]
class understudy.ToolError[source]

Raised by mock tools to signal an error to the agent.