Plausible is the risk
Most of the risk in clinical AI is not a model saying something absurd. It is a model saying something plausible that the record does not support: a "possible tear" becomes a "tear", a symptom the patient denied turns up as present, or a citation looks real and points nowhere. Wording like that survives a quick read, and a busy clinician gives most drafts a quick read.
This autumn we built a set of checks aimed at exactly that kind of error, and measured them before putting them in front of anyone. This post describes what we built, how we tested it, and what is still to do.
Checks that look for meaning, not just matching words
ArkCloud's AI features already withhold any text that contains a number, name, code or quotation not found in the record the model was given. That stops invented figures. It does not stop a sentence whose every word is in the record but whose meaning is not. The new checks look for that:
- A dropped hedge. The record says "?ACL tear" or "possible meniscal injury"; the draft states it as fact.
- An asserted denial. The record says "denies numbness"; the draft says numbness is present.
- An unsupported diagnosis. The draft says "diagnosed with" something the record does not affirm and the problem list does not carry.
- A fabricated citation. Any reference the model was not given, whether a PubMed ID, DOI, link, bracketed number or "Smith et al.", is treated as invented.
- Instructions that direct care, such as "stop taking", "increase the dose" or "no need to go to the emergency room", where the source does not say so.
- Instruction-like text in a patient's message or a retrieved document ("ignore previous instructions"), which is treated as data and never followed.
When a check fails, the AI text is withheld and the screen shows the rule-built version instead. We also applied the checks to clinical decision support, which until now returned the model's answer next to its citations without checking that the citations supported it.
Every item next to its exact words
The first feature built on these checks is an encounter review for therapy notes. It lists what a saved note says, such as symptoms, history, medications, allergies, findings and the patient's questions, and every item is shown beside the exact words it came from. An item whose words cannot be found in the note is discarded and counted, never shown.
Alongside the AI list, rule-based checks with no AI involved flag:
- a missing medication or allergy review;
- two statements in the note that contradict each other, with both passages quoted;
- a current medication that names a recorded allergy.
The medication check matches names only. It does not check drug classes or interactions, and the screen says so.
The review writes nothing to the note or the patient's lists. If a clinician wants an item, they type it.
How we tested it
We used a public synthetic dataset of 2,194 generated doctor-patient conversations as test material, and only as test material. It is kept in a separate synthetic-only store that the application cannot load, and an automated test fails if any product code tries.
We did not use the dataset's answers as a grading key. Each test case's correct result was known by construction: a faithful copy of a clinician's sentence must pass, and the same sentence with one meaning-changing edit must be caught. On those cases:
- 987 of 1,090 dropped hedges and 1,405 of 1,500 asserted denials were caught, with no false alarms on the faithful copies (0 of 1,090 and 0 of 1,500);
- 2,189 of 2,194 PubMed IDs with one digit changed were caught, and none of the 2,194 real ones were rejected;
- every inserted prompt injection was detected (2,194 of 2,194), with no false alarms across 70,983 ordinary conversation turns.
Checking the dataset also told us what not to use it for. Compared against the official US ICD-10-CM release, 28 of its diagnosis codes are not US codes, and 332 are category headings that cannot be billed. Its drug identifiers are Canadian. So it plays no part in coding, drug checks or anything a patient sees.
What is not done yet
These results measure our checks, not clinical correctness. Whether an extracted item is the right clinical reading can only be judged against cases reviewed by licensed clinicians, and that reference set does not exist yet. Until it does, and until clinical, model-provider and regulatory reviews are signed, the encounter review stays off by default and is not available to customers. Our contradiction check is deliberately a warning, never a block: in a hand review of a sample, about a third of its flags were a patient contradicting their own history.
We have also added an AI quality screen to ArkCloud Insights. It shows how often AI text was shown or withheld, how long it took, and what clinicians decided, all counted from records the platform already keeps. Cost appears as "not metered" because the model runs on the organization's own server and reports no usage to bill. We would rather show a blank than a guess.
See every AI feature and its status on AI Across ArkCloud Health, or read how ArkCloud uses AI across the healthcare journey.
Test material: Doctor-Patient Conversations - All Human Diseases (Opus 5.5), nisten, 2026, on [Hugging Face](https://huggingface.co/datasets/nisten/opus5-5-doctor-patient-conversations-all-human-diseases), revision 9277244e. Synthetic data; not medical advice.
