Metric library by intent
The seven intent groups from our paper, each with its prototypical metrics. The summary definition and alternate names on each card are our synthesis across frameworks. Expand any metric to see each framework's original definition and source.
All 116 metrics
Every metric extracted from the 38 sources, with its definition, a reference, and how it is evaluated. These are the same metrics summarized in the library, shown in full. CHAI’s use-case metrics are summarized here at the construct level; the full set of 196 CHAI metrics by use case is in the downloadable dataset.
Sources reviewed
The 38 sources in the review (oldest first): 34 peer-reviewed papers, 3 preprints, and 1 white paper.
Methodology
Full detail on how we conducted the review.
Search strategy
We searched Web of Science, Scopus, and Google Scholar with the terms "LLM" or "Large Language Models", "metrics" or "evaluation", and "healthcare" or "clinical", supplemented by a Google search for white papers and references in included papers. We included sources in English, published from January 2023 to August 2026.
Inclusion criteria
- Proposes, validates, or reviews an evaluation framework, benchmark, instrument, checklist, or reporting guideline for LLM applications in healthcare
- Covers human evaluation, automated evaluation, or LLM-as-a-judge evaluation
- Published in English as a peer-reviewed paper, preprint, or white paper (peer-review status was recorded)
Exclusion criteria
- Evaluations of a single application that did not propose or adapt a framework
- Papers on LLM evaluation outside healthcare
Data extraction and coding
The second and third author screened titles and abstracts and read full texts; the rest of the team checked inclusion decisions. We included 38 sources (34 peer-reviewed, 3 preprints, and 1 white paper). For each source, we recorded the framework, its stated intent, every metric defined, rating scale, who performed the rating, the clinical domain, and any reported reliability or validity statistics. For each metric, we recorded its stated definition, related concepts, and the original instrument where cited. We then grouped metrics with overlapping definitions by intent, using the stated definition rather than the paper's heading, and within each group selected as prototypical the metric whose definition most directly named the construct, listing other names as alternatives. The seven intent groups emerged inductively from the extracted metrics and were validated by two authors.
Suggest an addition
If you know of a framework, benchmark, or metric set we should include, let us know.