We took health AI into the field. Now we’re sharing what came back.
With support from the Gates Foundation’s maternal-health team, Crane AI Labs is sharing AfriMed-PII privacy models and evaluation resources, alongside findings from our EaseHealth field evaluation in Luweero District, Uganda.
The resources give other teams something to inspect and test. The findings show where the clinical workflow and device configurations need further work. Together, they move the conversation beyond whether a model can run on a phone to what it takes for that system to be useful.
EaseHealth provides offline, guideline-grounded decision support for frontline health workers. The health worker remains responsible for investigation, interpretation and the clinical decision. The privacy models serve a different purpose: helping identify personal information in text.
Models. Benchmarks. Field findings.
This is what we learned from examining them separately.
Privacy belongs in the language of the record
A clinical note can describe an illness and identify a person at the same time. It may contain a patient’s name, a relative’s name, a location, a date or a contact number. Before that text is reused, there is a privacy question to address.
AfriMed-PII helps identify and redact specified personal-information spans in Luganda, Kiswahili, Kinyarwanda and Dholuo. The release includes resources based on AfriBERTa, XLM-R and AfroXLMR, with evaluation materials and model cards describing their performance and limitations.
For another team, that means there is something concrete to examine: a model to test, an evaluation to inspect and documentation explaining which kinds of information were represented.
The benchmark work includes translated clinical proxy text, tests with unseen names and out-of-domain news material. These ask different questions: does detection carry beyond familiar names, and how does it behave when the context changes?
They do not yet establish performance on independently annotated real clinical records in each language. Nor does a successful redaction example guarantee that an unseen record is anonymous.
That boundary is part of the resource. The next team should be able to see where the evidence ends before deciding where to use the model.
A plausible answer is not the same as acceptable clinical support
The EaseHealth evaluation asked a separate question: how did physicians assess the guidance generated during field use?
Two physicians independently reviewed 250 saved outputs. Before seeing each model response, they assessed the clinical information entered by the health worker. They then reviewed the output. A third physician adjudicated the 60 cases that met the review’s trigger rules.
Seventy-four of the 250 outputs—29.6%—met the final clinical-acceptability rule.
That rule combined assessments of label-and-guidance accuracy, completeness, safety concerns and harmful omissions. The percentage describes reviewed outputs, not diagnostic accuracy, patient benefit or the proportion of patients who received appropriate care.
It is a finding we have to build from, not explain away.
The study assessed recorded information within a feasibility and acceptability evaluation, rather than independently examining patients or measuring clinical effectiveness. Within that scope, it identified weaknesses that a successful demonstration alone could not resolve.
The practical conclusion is clear: completing an assessment is not enough. The output also has to meet a clinical standard, and the evaluation did not support broad expansion of the unchanged configuration.
The findings should change the next build
One priority is the information available when the model is asked to respond. The evaluation found an association between input sufficiency and output acceptability. It also found consequential errors in outputs generated from records reviewers considered sufficient.
Better inputs therefore matter, but they are not the whole remedy.
Our next design priorities include more structured clinical history, clearer handling of missing information and uncertainty, and safeguards around consequential output errors. These are improvements to develop and evaluate—not features we are announcing as finished.
The evaluation also needs to extend beyond the generated answer. What did the health worker investigate? Was the guidance set aside? Was a referral made, and was it completed? The existing study did not establish most of those subsequent actions.
Those questions move the next phase closer to the actual purpose of decision support: helping a health worker make an informed decision, rather than merely producing text.
The device is part of the workflow
Device testing exposed another limit.
In the July compatibility tests, none of the four tested 4 GB-class devices completed the protocol with the tested configuration. Three of four testable 8 GB-class devices did. One additional 8 GB-class device could not be tested.
These are results for particular units and a particular build, not compatibility rates for every phone with the same memory label. Even within the higher-memory group, the label alone did not predict success.
That matters when moving from a demonstration to a deployment decision. The application, model and device have to be qualified together. The phone must do more than hold the model file: it must complete the intended workflow.
We also tested cloud-assisted configurations and different connectivity conditions. Those results belong to their own test setups; cloud-client timings are not handset measurements, and faster inference does not establish better clinical guidance.
Offline and cloud delivery place different demands on a system. The choice should follow the user’s conditions and the evidence—not a preference for one architecture before testing it.
A release to build from
For developers, AfriMed-PII offers models and evaluation resources to inspect and test.
For health-system and research teams, the EaseHealth findings identify questions the next workflow and evaluation need to answer.
For our own team, the release marks a change in what progress means. Getting the model to run was an engineering milestone. Understanding where the system falls short—and turning those findings into the next test—is how we move the work forward.
Thank you to the Crane AI Labs team, our clinical collaborators and language reviewers for the work behind these resources and findings.
The next build should be shaped by what the last evaluation taught us.
Explore the AfriMed-PII collection and read the model cards:
This material is based on research funded by the Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Gates Foundation.
