Balsam Semantica Computable clinical meaning

Method · CIRI™ - the Clinical Intelligence Readiness Index

Method and Limitations

Instruments of this kind should explain how they were developed, what they measure, and what has not yet been established. This page provides that information.

Institutions considering a CIRI™ assessment should understand the basis of the instrument, how its scores are determined, and the limitations that still apply..

← Back to the assessment

The instrument's scoring tables and limitation notes, open on a desk
What the instrument does not yet rest on is published here, so that no institution has to discover it later.

How the instrument was constructed

Three conditions, eighteen metrics, one underlying principle

CIRI™ assesses three conditions:
* Semantic Sovereignty* Clinical Authorship* Infrastructural Readiness
These conditions are measured through eighteen metrics, each scored from 0 to 5.
The instrument is based on one central principle: an AI system can produce fluent clinical text without accurately representing clinical meaning.
Whether an institution can identify and prevent this problem depends on conditions that must be established before an AI model is deployed.
This principle is explained in the CIRI™ Guide. The assessment instrument was developed from that principle.
Each metric examines a specific stage in the process from a clinical encounter to the use of clinical information by a computer system.
Every metric has a defined purpose and a clear explanation of what it does not measure. This prevents the same institutional strength from being counted across multiple metrics.
The scoring rules were established before the first institutional assessment and are applied consistently to every institution.
The scoring logic is also implemented in code. If the written scoring tables differ from the code, the code takes precedence. This ensures that scoring rules are applied consistently rather than interpreted differently from one assessment to another.

What has not been established

Six limitations

CIRI™ has limitations that must be clearly understood. These are not hidden assumptions.
 They are open questions that future institutional assessments will help address.

01 · Weights

The weighting reflects a reasoned judgement, not a statistical calculation.

Semantic Sovereignty accounts for 40% of the overall score. Clinical Authorship and Infrastructural Readiness each account for 30%.This weighting reflects the position that an institution must have control over clinical meaning. Strong performance in other areas cannot compensate for a fundamental weakness in this condition.However, these percentages are based on a reasoned position, not statistical analysis. No dataset was used to calculate them when the weights were established.
Future assessment data may provide grounds for reviewing them.

02 · Divergence bands

The Divergence Index thresholds were established before institutional data was availablet

The thresholds used to classify results as Aligned, Drift, Divided and Governed on Paper were selected as reasonable starting points.
Their accuracy cannot be confirmed until institutions have been assessed and the results can be examined.
For example, if most institutions fall within the same band, this may indicate that the thresholds need to be reviewed.

03 · Validation

The instrument has not been empirically validated

CIRI™ has not been tested against clinical outcomes, compared with reported incidents, or evaluated against another assessment instrument.
Its current claim is limited to how it is designed to measure readiness: using defined criteria, rules established in advance and evidence provided by the institution.
CIRI™ does not claim to predict whether an AI system will perform safely or successfully.
Its scores should not be interpreted as predictions of future performance.

04 · Scale anchors

The scoring levels have not yet been calibrated using data from real institutions.

Each metric has scoring levels from 0 to 5. The descriptions defining each level, known as scale anchors, were established during the development of the instrument.
However, it is not yet known whether the boundaries between levels are set at the right points.
For example, further assessments are needed to determine whether the difference between a score of 3 and a score of 4 is appropriate across all metrics.

05 · What cannot be documented

The instrument cannot measure what a clinician knew or noticed but did not record.

CIRI™ evaluates documented evidence, such as technical specifications, query results, contract clauses and clinical records.
Some clinical information never enters the record. A clinician may notice something during a consultation, or make a decision based on information that is not documented.
Because this information leaves no evidence in the record, the instrument cannot directly assess it.
Instead, CIRI™ examines whether the systems and processes are capable of preserving clinical meaning. This includes the available fields, the distinctions that can be recorded and the governance processes that support accurate documentation.
The instrument also includes a question answered by clinicians alone. It asks whether clinical decisions are being made without their reasons being adequately recorded.
This response provides an account of the problem, not a direct measurement of undocumented clinical reasoning. It is treated accordingly. 

06 · Discrimination

Some metrics may not distinguish between institutions

A metric that gives every institution the same score provides little information about differences in readiness.
It is not yet known whether any of the eighteen metrics will show this pattern.
This can only be established after enough institutions have been assessed and their results compared..

How these limitations are being addressed

Through institutional assessments and accumulated evidence

Each of the six limitations is recorded as an open question for future versions of CIRI™.
Anonymised, aggregated assessment scores are retained to support this work.
As more institutions are assessed, the weighting, Divergence Index thresholds and scoring levels will be reviewed against the accumulated data.
If changes are made, they will be published together with the reasons behind them.
This is important: changes to an assessment instrument should be supported by evidence and explained, rather than introduced without justification.
The instrument remains unchanged throughout each annual version. All institutions assessed within the same year are therefore evaluated using the same criteria.
Every assessment report identifies the version used to calculate its scores.

What the instrument does claim

Three defined  claims

  • The conditions it assesses come before the safe deployment of clinical AI, and can be examined through institutional evidence.
  • Comparing three independent institutional accounts reveals differences between what an institution has established and what happens in practice. A self-assessment cannot provide the same independent comparison.
  • A score based on documented evidence and rules established in advance provides a stronger basis for assessment than a rating an institution assigns to itself. This remains true even while the scoring weights and thresholds are being further examined.

Those three claims are what the assessment rests on. Everything above is what it does not yet rest on, and it is published here so that no institution has to discover it later.

Clinical meaning is not generated by AI. It is captured, governed, and made computable.