Each synthetic task starts with a defined environment and a known target state. An agent reads documents and uses tools to create the required clinical resources. A deterministic verifier scores the final state.
What is measured
The benchmark measures completion of synthetic tool-use tasks in three families: Document → FHIR, Clinical coding, and Claims. It evaluates the resulting state, rather than a multiple-choice answer.
Scoring
Each episode receives a deterministic score between 0 and 1. Family-specific checks cover the requested resources and task constraints. Integrity checks cover patient identity and preservation of existing resources. No LLM judge is used.
Standard results
The leaderboard reports overall and family scores for 12 models, with three attempts per model and 95% confidence intervals. Results are dated October 2026.
Reproducibility
Submissions must identify the evaluation date and runner version so that results can be reproduced.
Submission policy
Only publicly available models, through an API or open weights, are ranked. Submit one result per model version. Best-of-N selection is not allowed. Each published submission must identify its date and submitter; results must reproduce with the stated runner version.
Private evaluations
Private evaluation reports return aggregate scores. Task contents, per-task outputs, and trajectories from held-out evaluations are not returned to submitters.
Public examples
The examples on the landing page come from the public task split and use synthetic inputs.