Judge API

The faithful judge interface for external evaluation harnesses: the prompt is sent VERBATIM (no persona, no scaffolding), nothing is cached, and every result carries its model id and prompt hash for auditability.

async LayoutLens.judge(image_path, prompt, *, max_tokens=AUTO, timeout=120.0)[source]

Send prompt VERBATIM with an image and return a parsed verdict.

This is the faithful judge interface for external evaluation harnesses (e.g. UIJudgeBench). Unlike analyze(), LayoutLens adds NOTHING to the prompt: no system persona, no scaffolding, no appended JSON-format instruction. The caller owns the entire prompt, including any response contract. The call always hits the model (no caching) and honors the per-model parameter policy (Claude 4.6+/5 omit temperature).

Parameters:

  • image_path (str | Path) – Path to an existing image file (mime inferred from the extension: .jpg/.jpeg -> JPEG, otherwise PNG).

  • prompt (str) – The exact text to send as the sole text block.

  • max_tokens (int | _Auto) – Maximum tokens to generate. Defaults to AUTO, which resolves to 8000 for reasoning/thinking models (they spend thinking tokens inside this budget) and 300 otherwise. Pass an explicit integer to override.

  • timeout (float) – Per-call timeout in seconds (default 120 — reasoning models can take well over 30s on a single judgment).

Returns:

JudgeResult with the parsed answer/confidence/rationale, the raw text, a refusal flag, per-model usage split, and the parse mode.

Raises:
Return type:

JudgeResult

class layoutlens.JudgeResult(answer, confidence, rationale, raw, refused, usage, model, parse_mode, truncated=False, prompt_sha256='')[source]

Bases: object

Structured outcome of a single judge() call.

Parameters:
answer

Parsed answer field, or “unknown” if unparseable.

Type:

str

confidence

Parsed confidence in [0, 1], else 0.0.

Type:

float

rationale

Parsed rationale/reasoning field, else “”.

Type:

str

raw

Full raw model text (always populated, even on refusal).

Type:

str

refused

True if the response matched a refusal pattern.

Type:

bool

usage

Token counts with keys prompt_tokens/completion_tokens/total_tokens; reasoning-capable backends may also report thought_tokens.

Type:

dict[str, int]

model

The model that produced the response.

Type:

str

parse_mode

“json”, “fallback”, or “none”.

Type:

str

truncated

True if the model stopped because it hit the token budget (finish_reason == "length") — the verdict may be incomplete.

Type:

bool

prompt_sha256

SHA-256 of the exact prompt sent (judge-contract pinning: model + prompt hash make a result auditable).

Type:

str

Batch Judging

async LayoutLens.judge_batch(requests, *, max_tokens=AUTO, resume=True, manifest_path=None, poll_interval=10.0, poll_timeout=86400.0, reasoning_effort=None, image_detail='auto')[source]

Judge many image+prompt requests over a provider batch transport.

The batched counterpart to judge(): each request sends its prompt VERBATIM with the same image bytes as judge(), honors the reasoning-aware max_tokens default and the per-model parameter policy, and is parsed into a JudgeResult. Batch APIs are ~50% cheaper and the right transport for bulk offline evaluation (e.g. UIJudgeBench). LayoutLens is thus the reference batched judge.

The backend is chosen from an explicit provider/model combination: provider="gemini" with gemini/* uses the google-genai inline batch (optional extra layoutlens[gemini]); native OpenAI uses the official Responses Batch API, and supported non-Gemini/non-OpenAI providers use the litellm file-based batch. All backends are resumable via a manifest.

Parameters:

  • requests (list[BatchRequest]) – The batch items. Each id must be unique and keys its result. A request whose image is missing yields an "unknown" result rather than aborting the batch.

  • max_tokens (int | _Auto) – Per-request token budget. Defaults to AUTO (8000 for reasoning models, else 300); an explicit integer overrides.

  • resume (bool) – When True (default), collect any prior jobs from the manifest first and submit only uncovered ids. When False, the selected manifest path must not already exist.

  • manifest_path (str | Path | None) – Where submitted job ids persist for resume. Defaults to a content-addressed path under output_dir/batch keyed by the backend, endpoint, model, token budget, reasoning effort, image detail, exact prompts, image MIME types, and image bytes.

  • poll_interval (float) – Seconds between batch-status polls.

  • poll_timeout (float) – Max seconds to wait for a single batch job.

  • reasoning_effort (str | None) – Native OpenAI reasoning effort. None retains the model default; otherwise one of none, low, medium, high, xhigh, or max. Rejected for other providers so a requested setting is never silently ignored.

  • image_detail (str) – Native OpenAI image-detail setting: auto, low, high, or original. Non-default values are rejected for other providers.

Returns:

{request_id: JudgeResult} for every request.

Raises:

  • AuthenticationError – If no API key is configured for a mapped provider.

  • ImportError – If a gemini/* model is used without google-genai.

  • ValidationError – If request ids repeat or an existing resume manifest does not match the exact request.

Return type:

dict[str, JudgeResult]

class layoutlens.BatchRequest(id, image_path, prompt)[source]

Bases: object

One item in a batch judge call.

Parameters:
id

Caller-owned unique id; results are keyed by it.

Type:

str

image_path

Path to the image to judge.

Type:

str | pathlib.Path

prompt

The exact prompt to send VERBATIM (as in judge()).

Type:

str

layoutlens.batch_usage_summary(results)[source]

Aggregate token usage (and estimated cost) across judge_batch results.

Parameters:

results (dict[str, JudgeResult]) – Judge results keyed by the caller-owned batch request ids.

Returns:

Dict with request counts, per-field token totals, and estimated_cost_usd (None when the model is unknown to litellm).

Return type:

dict[str, Any]