A joint research evaluation from Jacaranda Health and Crane AI Labs
Jacaranda Health’s PROMPTS answers maternal and newborn health questions for mothers across Kenya using an AI model that runs in the cloud. We wanted to know what would happen if a maternal-health model ran on a phone instead, with no internet connection.
Between January and October 2026, Jacaranda Health and Crane AI Labs investigated that question. We evaluated one-billion-parameter (1B) and four-billion-parameter (4B) models using the same 950 English questions from Jacaranda’s test set. Each model was tested in Jacaranda’s cloud environment and as an 8-bit version running offline on an Android device. Jacaranda’s automated evaluator scored the answers against its reference answers.
The 4B model’s average medical-accuracy score was 3.82 out of 5 on-device and 3.78 in the cloud. The 1B model scored 2.02 on-device and 3.10 in the cloud. The larger model’s encouraging average came with substantial speed and memory constraints, and differences between individual answers that need further investigation.
This is a research checkpoint, not an offline product launch. No mother or health worker received the answers generated in this evaluation, and no rollout is being announced.
1. How the work was divided
Jacaranda fine-tuned the models, supplied the questions and reference answers, defined the quality metrics and ran the automated evaluator. Crane converted the models for Android, ran them on physical devices, and collected the available performance telemetry.
The collaboration’s starting point was to evaluate an edge model before discussing a pilot. The work therefore examined both the generated answers and the practical constraints of running the models without connectivity.
2. What was tested and how it was scored
The 1B device configuration ran an 8-bit model on a Samsung tablet using LiteRT; its cloud reference ran at bfloat16 precision on an A100 GPU. The 4B device configuration ran an 8-bit model on an 8 GB Infinix phone using LiteRT; its cloud reference also ran at bfloat16 on an A100.
The cloud reference in each pair was Jacaranda’s own model, not a frontier AI service. The configurations answered the same English question set using Jacaranda’s prompt format. Because precision, runtime and hardware changed together, this comparison does not isolate the effect of quantization.
Jacaranda used an LLM-as-a-judge approach: an AI model evaluated the generated responses against reference answers and a scoring rubric. Medical accuracy was scored out of 5, agent behaviour out of 5, and grammar out of 2, for a maximum total of 12 points.
Medical accuracy covered clinical explanations, danger signs and when to seek care. The rubric capped the medical score at 2.5 when a response missed one of several problems in a question. Agent behaviour covered qualities such as empathy, clear and concise language, and matching the language of the question. Grammar covered punctuation, capitalization, spelling and spacing.
These are rubric scores, not percentages of clinically correct answers. The evaluation also compared each response with its reference answer using three similarity measures. ROUGE-L looks at shared words in the same order. METEOR also allows for related word forms and synonyms. BERTScore uses an AI model to compare meaning across differently worded answers. These measures describe similarity to a reference; they do not establish clinical correctness or safety.
Conversion required more than checking that the model would load. An early export omitted a language token used by the fine-tuned model’s prompt format. The team corrected the export and reran the question set. The practical lesson is that a converted model must preserve the inputs expected by the trained model, not just its weights.
3. The quality result
The reported mean scores across the 950 questions in each configuration were:
The 4B’s average medical-accuracy scores were close: a difference of 0.04 points on the five-point scale. Its average agent-behaviour and grammar scores were also close. That supports a narrow observation about these test results—not a conclusion that the phone outperformed the cloud or that the configurations are clinically equivalent.
The tested 1B device configuration showed a much larger gap in medical-accuracy score: 2.02 versus 3.10. It would be misleading to translate that difference into a percentage of clinical accuracy lost, or to attribute it to precision alone.
The 4B device run generated answers for all 950 questions, with no failed answers reported. Completing a run is an execution result; it does not mean every answer was correct or safe.
4. The averages held. Individual answers moved.
Close averages do not mean the two configurations answered every question equally well. The paired results included answers that met the evaluator’s acceptance threshold in the cloud but fell below it on-device, as well as answers that moved in the opposite direction.
Those changes matter, particularly for danger-sign questions. Improvements on some questions do not cancel potentially consequential deterioration on others. A favourable average cannot establish that the same important clinical information is preserved in every response.
This evaluation used one generated answer per question in each configuration and one judge pass per answer. That design cannot separate all variation in generation from variation in judging. Repeated generation, repeated scoring of saved answers and clinician review would help determine which changes are reproducible and clinically consequential.
The result is an encouraging aggregate finding with an unresolved answer-level reliability question—not evidence that the tested offline configuration is ready for clinical use.
5. What did not work
The 4B device configuration also exceeded the speed and memory targets recorded in the evaluation report.
Time to first token was reported at 11.6 seconds at the 90th percentile, against a target below 3 seconds. Generation speed was reported at about 1.5 tokens per second, against a target of at least 3.
Full-answer time was estimated at 117 seconds at the 90th percentile, against a target below 15 seconds. The report describes this figure as derived from gaps between logged answers rather than a direct per-answer timing measurement. It is indicative, not a measured consultation duration, and should not be confused with time to the first word.
Peak memory was reported at approximately 5.2 GB on an 8 GB phone, exceeding the report’s stated 1.5 GB memory target. The tested configuration therefore did not meet the intended lower-memory deployment requirement. This result concerns that model, runtime and device—not every 4B model or every 8 GB handset.
The tested device path did not use GPU or NPU acceleration, prompt caching or compression below 8-bit. Those are possible optimization routes, not demonstrated improvements. Any change would need both performance measurement and a fresh check of answer quality.
The 1B ran on a tablet and the 4B on a phone. Their operating speeds cannot be read as an isolated comparison of model size. The practical gap remains: the tested smaller model had a weaker medical-accuracy score, while the larger model required more memory and longer waits.
6. The question the benchmark could not answer
When would an offline model be more useful than a connected service?
A benchmark cannot establish that by itself. One setting worth investigating is provider use where unreliable connectivity interrupts access to guidance. Whether the model is useful there depends on the task, the device, the quality of its answers and the time a health worker can reasonably wait.
That is a research question, not a product commitment by either organisation. This evaluation did not establish demand from mothers or providers, a preferred deployment route, or a plan to introduce offline models into PROMPTS.
7. What this evaluation did not establish
This was a retrospective evaluation on 950 English questions, not a clinical trial. It did not measure patient outcomes, changes in health-worker decisions, referral completion or the effect of using the model during care.
The medical-accuracy results came from an automated judge. They are not independent clinician assessments, and the evaluation did not establish clinical equivalence between the device and cloud configurations. It also did not compare either model with a frontier service.
The results do not establish Kiswahili or code-switched performance. They also do not separate model capability from conversion, precision, runtime and hardware effects.
Energy and cost estimates in the report are not full service costs. In particular, the 4B energy figure was estimated rather than directly measured because the device was charging during the run. Those estimates cannot establish the cost of a real-world deployment, including devices, connectivity, distribution and support.
The timing derivation, single-pass scoring and answer-level changes are therefore part of the interpretation, not footnotes to be set aside. No pilot, clinical adoption or offline rollout is being announced.
8. What comes next
The next research should resolve these uncertainties rather than assume that a faster implementation will be sufficient.
First, separate the causes of performance differences. Where technically feasible, test the same checkpoint and numerical precision on a common platform, then vary conversion, runtime or hardware. Verify tokenization, prompt templates and generation settings before interpreting a quality difference.
Second, investigate answer-level reliability. Repeat generation and judge scoring separately, and ask clinicians to review consequential disagreements, especially danger-sign responses. Similar average scores should not be the sole acceptance criterion.
Third, measure the complete user experience. Instrument the time from submission to displayed guidance on the device, including failures and retries. Test potential runtime improvements alongside answer quality, sustained use, temperature and battery behaviour. Define useful waiting times with the people who would use the system rather than treating a benchmark target as a clinically established limit.
Fourth, test the language and workflow that a proposed use case requires. Kiswahili and code-switched questions need their own evaluation. Provider input is necessary to establish whether offline access addresses a real problem and what information or safeguards the workflow needs.
A longer technical paper is under discussion between the teams. Its scope, authorship and timeline remain to be agreed; it is not a completed or peer-reviewed output.
The useful conclusion is not that the phone has beaten the cloud. It is that the 4B configuration produced similar average scores in this evaluation, while leaving important reliability and deployment questions unanswered. The next task is to establish when—and for whom—running the model locally is worth those trade-offs.
Credits and source
On-device conversion, evaluation and device metrics:
Bronson Bakunga, Crane AI Labs.
Model fine-tuning, quality metrics and judging:
Stephen Obonyo, Jacaranda Health.
Built on open Gemma-family models; converted with LiteRT.
Figures are reported from Jacaranda × Crane — Edge Evaluation Close-out v2, 10 October 2026, and the accompanying joint-review draft. They have not been independently recomputed for this article.
