Clinical Meaning Integrity · From consultation to note
Clinical AI Evaluation
An independent written evaluation of whether AI-generated clinical notes preserve the meaning of the consultations they document.
The question is not simply whether the system captured the right words. It is whether the note accurately records what was said and established: whose condition was discussed, whether a finding was present or denied, how certain the clinician was, and what was planned or actually done.
A note may contain the correct clinical terms and still record incorrect clinical information.
For example, a system may remove a negation, record a family member's condition as the patient's own diagnosis, turn a suspected condition into a confirmed diagnosis, or record a planned action as something already done. It may also change the meaning of a colloquial expression or fill in a template field that was never discussed.
Once signed, these errors can carry into the problem list, coding, billing, medication records, clinical decision support, and the next clinician's understanding of the patient.
Who it is for
Healthcare organisations evaluating these systems before deployment can also commission this service.
The evaluation is particularly relevant to systems working across languages and dialects, including Arabic and mixed-language consultations.
What you send
All consultations must be simulated. No patient records or patient data are needed or accepted
How the evaluation works
This includes what the patient reported, what the examination showed, and what the clinician concluded, planned, or instructed.
First, the clinical information in each consultation is identified independently, before reviewing the generated note.
This ensures that the note itself does not determine what is assessed.
Each item is recorded with the context needed to preserve its meaning, including:
• Whether it was present or absent.
• Who it referred to.
• When it occurred.
• How certain the statement was.
• Any relevant qualifiers.
• Whether an action was planned, conditional, or actually carried out.
Each item is then checked against the generated note.
The four classifications
• Preserved: The note records the information accurately, with its original meaning intact.
• Omitted: The information from the consultation is missing from the note.
• Altered: The information appears in the note, but its meaning has changed. For example, a negation is lost, the person it refers to changes, uncertainty is removed, the timing changes, or a condition is dropped.
• Invented: The note includes information that was never stated in the consultation, whether introduced by the AI or added through a default template.
What you receive
For every element, the report shows:
• The original clinical information.
• What the generated note records.
• Its classification.
• The nature of any change or error.
• How the error could affect subsequent clinical or administrative processes.
The report also identifies patterns across the evaluation set, showing which types of clinical information the system most often omits, alters, or invents.
The engagement follows a simple process:
1. Submit your request.
2. Receive the fee and terms in writing.
3. Work begins on written acceptance of the terms and full advance payment.
4. Share the simulated consultations and generated notes.
The evaluation covers up to ten simulated consultations in one language setting and one note type, and is delivered within ten business days of receiving the complete materials and payment.
All work is conducted asynchronously and delivered in writing.No meetings are required.
More than ten consultations, or additional languages or note types, are evaluated as separate evaluations, on the same terms.
What it does not cover
• A test of the system on consultations outside the supplied evaluation set. • A clinical validation study or certification of clinical safety, effectiveness, or regulatory compliance. • An assessment of coding or billing accuracy. • Technical work to correct, retrain, or modify the system. • An assessment of whether the clinician's medical reasoning was correct. • An endorsement or recommendation of the product.
The findings apply only to the consultations and notes included in the evaluation. They must not be presented as a measure of the system's overall performance.
Independance & use of the report
• The reviewer's name may not be used to endorse, promote, or validate the product.
• The report may not be presented as a certification or approval.
• Findings may not be used to make performance claims beyond the scope of the evaluation.
• Unless agreed in writing, the report may not be published or distributed, and its findings may not be quoted in marketing materials.
See a sample
A full sample evaluation is available in Writing. It examines a simulated paediatric consultation in Levantine Arabic and the ambient AI-generated note based on that consultation. Read the sample evaluation →
If your organisation needs an independent evaluation of AI-generated clinical notes, submit a request.
You will receive the fee, scope and delivery terms in writing.
Clinical AI Evaluation · Up to 10 simulated consultations · Ten business days · Element-level assessment · Written report · No patient data · No meetings required