Building a Local AI Physician Assistant Using RAG

How Personal Health Experiences Inspired an AI Clinical Decision Support Tool

Introduction

Large Language Models (LLMs) are transforming many industries, but healthcare adoption remains cautious due to concerns about privacy, explainability, and reliability. While cloud-based AI solutions dominate the conversation, local AI systems using Retrieval Augmented Generation (RAG) offer a compelling alternative: secure, interpretable, and deployable within controlled environments.

Recently, I built a local AI Physician Assistant prototype that analyzes patient records and clinical guidelines to generate clinical recommendation summaries. The system integrates structured patient data (EMR, lab results, radiology reports) with guideline knowledge to provide physicians with context-aware decision support.

What motivated this project, however, was not purely academic or technical.

It was personal.

A Personal Story Behind the Project

I have not shared this outside my family before, but for the past six years I have experienced severe unilateral headache episodes.

The first major episode occurred in early 2019 following a flu infection, severe enough to require an ER visit. Since then, these headaches have recurred intermittently, sometimes lasting weeks or months.

Over the years I tried several approaches for relief:

• Pure oxygen therapy, assuming the headaches might be cluster headaches

• Rizatriptan, commonly prescribed for migraine

• Over-the-counter medications such as:

  • Ibuprofen

  • Acetaminophen

  • Migraine formulations

I also underwent diagnostic imaging:

• 1 CT scan

• 2 MRI scans

Fortunately, all imaging results were normal, but the frustrating part remained: no definitive explanation of the cause.

My most recent episode lasted several months, with pain levels fluctuating between 3 and 9 on a 10-point pain scale.

During that period:

• Off the counter medications had no meaningful effect

• Migraine medications provided minimal relief

Eventually my neurologist prescribed a one-week steroid course, which significantly improved symptoms. Most days now the headaches are absent, though it remains unclear how the condition will evolve.

The Idea: Could AI Assist Clinical Reasoning?

During this process I wondered:

Could a small AI system analyze my medical records and provide a clinical recommendation similar to what physicians suggest?

So I experimented.

I fed the following into a local RAG-based Physician Assistant model:

• EMR data

• Lab results

• Radiology reports

• Clinical guideline documents for:

  • migraine

  • cluster headache

  • tension headache

The result surprised me.

The AI-generated recommendation was very close to the diagnostic reasoning and treatment direction suggested by my family physician.

This does not replace doctors, nor should it.

But it demonstrates something powerful:

A relatively simple local AI system can assist physicians by rapidly synthesizing patient data and medical guidelines.

System Overview

The prototype implements a Retrieval Augmented Generation architecture to ground LLM reasoning in clinical knowledge.

The system combines:

• Patient data retrieval

• Medical guideline retrieval

• LLM reasoning

Architecture

EMR / Labs / Radiology

↓

Document ingestion & embedding

↓

Vector database retrieval

↓

Clinical guideline retrieval

↓

LLM reasoning

↓

Clinical recommendation summary

This allows the system to generate evidence-grounded suggestions rather than hallucinated answers.

Key Components

1. Patient Data Layer

The application loads patient records including:

• Electronic Medical Records

• Laboratory results

• Radiology reports (CT, MRI, X-ray)

Each document is converted into embeddings and indexed for retrieval.

2. Clinical Knowledge Base

Guideline documents are stored in text files including:

• Migraine clinical guidance

• Cluster headache diagnostic criteria

• Tension headache treatment guidelines

These documents form the medical knowledge layer for the RAG pipeline.

3. Retrieval Augmented Generation

When a physician selects a patient:

1. Relevant patient records are retrieved

2. Related clinical guideline sections are retrieved

3. The LLM synthesizes both sources

This produces a structured response containing:

• Possible diagnoses

• Supporting evidence

• Recommended next clinical steps

Example AI Recommendation Output

For a patient with unilateral headaches and normal imaging, the system generated recommendations similar to:

Potential diagnoses

• Cluster headache

• Chronic migraine

• Trigeminal autonomic cephalalgia variants

Clinical considerations

• Evaluate headache timing and circadian patterns

• Assess response to oxygen therapy

• Consider triptan effectiveness

• Evaluate steroid responsiveness

Possible management suggestions

• Neurology follow-up

• Preventive therapies if episodes recur

• Further diagnostic classification

The alignment between AI suggestions and physician recommendations was striking.

Why Local AI Matters in Healthcare

Many healthcare systems hesitate to adopt AI because of:

• Patient privacy concerns

• Regulatory compliance

• Data security

• Cloud dependence

A local RAG-based AI assistant addresses many of these issues:

Advantages include:

• Patient data never leaves the local environment

• Transparent guideline grounding

• Low infrastructure requirements

• Rapid deployment within clinics

This approach could serve as a clinical decision support layer rather than a diagnostic authority.

What This Project Demonstrates

This experiment shows that a simple AI architecture can already provide useful clinical insight.

Important takeaways:

1. RAG dramatically improves reliability compared to standalone LLMs

2. Local models can still produce meaningful clinical summaries

3. AI can reduce cognitive load for physicians

4. Structured guideline retrieval improves explainability

The system is not meant to replace medical expertise.

Instead, it acts as a rapid knowledge assistant, surfacing relevant information when clinicians need it.

Future Improvements

The next steps for this project include:

• Integrating larger clinical guideline datasets

• Adding structured diagnostic scoring systems

• Expanding radiology interpretation pipelines

• Improving UI for physician workflows

• Integrating secure hospital databases

Ultimately, the goal is to evolve this prototype into a transparent clinical reasoning assistant.

Final Thoughts

My experience with recurring headaches highlighted how complex diagnosis can be, even with extensive testing.

What surprised me most was seeing a small AI prototype produce reasoning similar to what experienced physicians suggested.

It reinforced a key idea:

AI should not replace physicians — but it can absolutely assist them & patients.

Even a small local application, built with open tools and a few hundred lines of code, can provide meaningful clinical insights.

And that opens the door to a future where AI becomes a trusted clinical companion rather than a black box.

Code Repository

You can explore the full implementation here:

GitHub:

AI-Physician-Assistant-with-RAG

The repository includes:

• RAG ingestion scripts

• Patient data loader

• Clinical guideline embeddings

• Streamlit UI interface

If you're interested in building AI systems for healthcare, experimenting with local RAG pipelines is an excellent place to start.