Balsam Semantica Computable clinical meaning

Clinical Meaning Integrity · Sample

Sample Clinical AI Evaluation — Simulated Arabic Consultation and Ambient Scribe Note

About this sample

The consultation and the note are both simulated. The consultation was written for this sample, in Levantine Arabic with the clinician's English mixed in, as it is spoken in many clinics. The note was constructed to show failures typical of ambient scribes. It is not the output of any product.

In a client evaluation, the client runs simulated consultations through its own system and sends the scripts with the notes the system produced. No patient data is involved.

How the note is evaluated

The unit of evaluation is the clinical element, not the word: something the patient said, something the examination showed, something the clinician concluded or planned.

The elements are listed from the consultation first, before the note is read, so that the note cannot shape what is looked for. Each element is then traced into the note and given one of four results:

  • Preserved: the note carries the element with its meaning intact.
  • Omitted: the element is absent from the note.
  • Altered: the element appears, but its meaning has changed.
  • Invented: the note contains something the consultation never said.

For every element not preserved, the register records where the error travels once the note is signed: the problem list, coding, billing, decision support, or the next clinician's reading.

The consultation

A mother brings her four-year-old son to a paediatric clinic.

الطبيبة: أهلين، شو صاير معه؟

الأم: صرلو تلت تيام سخن، أعلى شي وصلت الحرارة ٣٩.

الطبيبة: في سعلة؟

الأم: إي، وصدره عم يصفّر بالليل.

الطبيبة: في إسهال أو استفراغ؟

الأم: لا، ما في إسهال ولا استفراغ. بس ما عم ياكل منيح.

الطبيبة: في حدا بالبيت عنده ربو؟

الأم: أخوه الكبير عنده ربو، هو لأ.

الطبيبة: Okay, let me listen to his chest… في crackles بالقاعدة اليمين.

الطبيبة: شاكّة إنها ذات رئة. رح نعمل صورة صدر، وإذا تأكدت منبلّش أموكسيسيلين.

الطبيبة: وإذا ما نزلت الحرارة خلال يومين، ارجعوا.

The note

HPI: 3-day history of fever, maximum 39°C. Cough. Chest tightness at night. Diarrhoea.

PMH: Asthma.

Immunisations: Up to date.

Examination: Crackles, right base.

Assessment: Pneumonia.

Plan: Chest X-ray. Amoxicillin started.

Element register

Eleven clinical elements were identified from the consultation before the note was read. The note preserved three, altered five, omitted two, and invented one.

# In the consultation In the note Result Where it can travel
1 Fever for three days, maximum 39 Same Preserved —
2 Cough Same Preserved —
3 صدره عم يصفّر — his chest whistles at night: wheeze "Chest tightness at night" Altered The dialect phrase for wheeze became a different symptom. The record no longer supports the finding the history pointed to.
4 No diarrhoea, no vomiting "Diarrhoea" Altered The negation was dropped and the negated vomiting omitted. The record asserts a symptom the mother denied.
5 Poor appetite — Omitted Lost from the record of the illness.
6 The brother has asthma; the child does not "PMH: Asthma" Altered Family history became the child's own diagnosis. It can reach the problem list, coding, and every future encounter.
7 Crackles, right base Same Preserved —
8 Suspected pneumonia "Assessment: Pneumonia" Altered The suspicion became a diagnosis. It can reach diagnosis coding, billing and quality reporting.
9 Amoxicillin only if the X-ray confirms "Amoxicillin started" Altered A conditional plan became an action taken. It can reach the medication record and antimicrobial reporting.
10 Return if the fever has not settled within two days — Omitted The record no longer shows that the family was told when to come back.
11 Never discussed "Immunisations: Up to date" Invented A template field was filled with a finding no one stated. It can reach the immunisation record.

The patterns behind the errors

Four of the five alterations are failures of context. A negation, a family history, a suspicion and a condition were each stripped, and what remained was written as fact. The fifth is a failure of dialect: a common Levantine phrase for wheeze was rendered as a different symptom.

The invented element has a different cause. The note follows a template, and the template had a field the consultation never filled.

A word-level measure would rate this note highly. Almost every clinical term in it appears in the consultation: fever, cough, diarrhoea, asthma, pneumonia, amoxicillin. What changed is not the words. It is whose condition, whether it was present, how certain, and whether it happened. A note can contain the right words and still record the wrong clinical facts.

Take one element. The mother said: «أخوه الكبير عنده ربو، هو لأ», his older brother has asthma, he does not. The statement carries more than a condition. It names a subject, a relationship, and an explicit negation about the child. The note kept the condition and lost the rest. "PMH: Asthma" is not a shorter version of the same fact. It is a different fact.

What this evaluation does not cover

It evaluates the notes supplied, against the consultations supplied. It does not test the system on other consultations, establish product-wide performance, certify clinical or regulatory compliance, assess coding or billing accuracy, or assess the clinical safety of any product. It does not judge whether the clinician's reasoning was right; it asks whether the note still says what the consultation said.

Sample Clinical AI Evaluation · Simulated consultation · Arabic and mixed-language clinical setting · Element-level meaning preservation

This sample illustrates the Clinical AI Evaluation. See the service →