AI Use Case: Electronic Lab Review
Oscar continues to experiment and iterate on clinical-AI-human use cases through Oscar Medical Group (OMG). OMG is a team of 120+ providers who offer virtual urgent and primary care for our members. It operates on top of Oscar’s in-house technology stack, including our internally-built Electronic Health Record (EHR) system.
A few months ago, we built an AI integration that analyzes an electronic lab result through our EHR system — surfacing an initial draft of review documentation for a virtual care provider. The goal is to enable providers to practice at the top of their license and simplify the more tedious parts of their work.
Here are some of the results.
The screenshot below shows the view of lab results in our EHR*, including the summary fields and follow-up notes. Based on the lab results, we’re able to leverage AI to provide suggested assessments and patient instructions that providers can use, amend, or discard.
Oscar has built several safeguards to ensure providers are properly reviewing and editing the outputs as needed based on their clinical discretion.**
The chart below shows the percent of cases in which providers accept or edit the AI-produced lab assessments. Providers had a binary response: some providers made minimal changes to the AI-provided output, while a majority discarded it or significantly edited the output based on personal preferences.
We found a greater degree of nuance in how providers approached patient next steps and instructions. The majority of providers made a lot more changes here — using some of the output and coupling it with their own language or editing portions out completely. We also noticed provider-specific behavior (i.e., the same provider tended to erase).
Below is what this looks like in practice. Provider changes to the AI output fell in two categories: 1) language style, and 2) patient context.
With regard to language, providers removed deterministic, unnecessary, or wordy language — and added personalization. For example:
Providers added either a greeting, patient name, or sign-off as their only change in 8% of cases.
They also gave the AI model feedback to “cut to the chase” and “better use real estate” in member notifications and alerts.
With regard to patient context edits, lab results alone were not enough to produce high-quality or complete outputs. For example:
If the model was missing the exact reason for which a lab was ordered, providers edited the output to put instructions into that context.
If the lab was part of a longer care journey for a patient in a virtual primary care relationship, providers had to edit that context back in.
As a next step, we know we need to provide the AI model with more context in future experiments in order to provide better quality output. Oscar has that information as a member’s insurer — including claims, external clinical charts, and concierge conversations. This is fertile ground for producing more refined outputs in service of our members.
*All data represented here is dummy data
**For example, providers are required to go through a review process prior to submission as a safeguard and providers must attest to the completeness and correctness of the review before it can be submitted. The note itself must be edited by the provider, as the output uses specific words like “suggested” and “suspected” in its suggested assessment and care plan. In addition, the EHR includes detailed disclaimers explaining how the AI feature works and its limitations. Finally, providers were trained on how the feature works and its limitations before the feature was launched.