Robotics
MediBot: A Continuous AI Patient Monitoring System for Nigerian Hospitals
# MediBot: A Continuous AI Patient Monitoring System for Nigerian Hospitals
## Introduction
Every day in Nigerian hospital wards, doctors and nurses make impossible choices.
One nurse. Twenty patients. Eight hours. There's no way to watch everyone's health state at once.
MediBot was built to change that, not by replacing nurses, but by giving them a system that never looks away.
It is a continuous AI-powered patient monitoring system that combines embedded sensors, computer vision, edge machine learning, and a real-time web dashboard to detect when a patient needs attention before a crisis becomes irreversible.
The entire system runs on affordable hardware that can be purchased locally in Nigeria, communicates through Supabase as its cloud backbone, and is accessible from any phone or laptop browser without any installation, making it deployable in hospitals that could never afford traditional ICU-grade monitoring equipment.
---
## The Problem
### The Monitoring Gap
In most Nigerian hospital general wards, continuous bedside monitoring simply does not exist.
A patient's oxygen saturation can fall from healthy to life-threatening in under two minutes. A patient losing consciousness may go unnoticed for twenty minutes or more if the nurse is responding to another emergency across the ward. Pain in sedated, post-operative, or non-verbal patients often goes completely undetected. Not because the staff does not care, but because no system is watching continuously enough to catch it.
This is not an assumption. As of 2019, Nigeria had only 15 nurses and midwives per 10,000 patients [1], making continuous individual patient observation practically impossible in most general wards. A mixed-methods study conducted across three hospitals in Kenya found that during the 24 hours before a patient's death, respiratory rate was documented in only 1.2% of patient records, with insufficient monitoring equipment and high workload identified as the primary reasons nurses could not detect deterioration in time [2]. The WHO estimates that up to 134 million adverse events occur annually in hospitals across low and middle-income countries, with 83% of those events being preventable [3].
The problem is not the dedication of the people working in these hospitals. The problem is that the tools designed to close this gap were never built for these environments.
### Why Existing Solutions Do Not Fit
ICU grade monitoring equipment exists. It works well. It costs millions of naira per unit, requires stable electricity, demands regular calibration by biomedical engineers, and was designed entirely around assumptions that do not match the realities of Nigerian general wards.
The problem is not only funding. It is the design. Most medical monitoring technology was built for environments with completely different constraints.
### Why This Matters
The consequences of the monitoring gap are measurable. Delayed response to cardiac events. Undetected respiratory decline. Unmanaged pain in post-operative patients. Missed early signs of fever escalation. Preventable deterioration during low-staffing periods.
MediBot was built as a direct engineering response to that gap. Affordable, deployable, and designed around the realities that Nigerian healthcare workers actually operate in.
---
## What MediBot Does
MediBot continuously monitors patients using a combination of medical sensors and AI vision. It classifies the patient's current condition in real-time and alerts medical staff when something requires attention.
The system determines whether the patient is awake and comfortable, sleeping normally, in pain, showing signs of distress, potentially unconscious, or whether the bed is empty altogether.
It achieves this by combining signals from multiple sources at the same time.
| Signal | Source |
|--------|--------|
| Heart rate | MAX30102 sensor |
| Blood oxygen saturation (SpO₂) | MAX30102 sensor |
| Body temperature | MLX90614 infrared sensor |
| Facial expression | AI vision |
| Eye state (open or closed) | AI vision |
| Body motion | Frame differencing |
| Bed occupancy | ML object detection model |
No single signal decides on its own. All signals are combined. The system is designed so that if one source fails, the others continue providing useful information.
---
## Components Used
### Hardware
| Component | Purpose |
|-----------|---------|
| ESP32 Dev Module | Sensor reading, decision logic, alert management |
| ESP32-CAM AI Thinker | Camera stream and ML vision inference |
| MAX30102 | Heart rate and blood oxygen |
| MLX90614 | Contactless infrared body temperature |
| Active buzzer | Audible patient alerts |
| Status LED | Visual connection indicator |
### Software and Platforms
| Tool | Role |
|------|------|
| Arduino IDE and C++ | ESP32 firmware development |
| Edge Impulse | ML model training and deployment |
| TensorFlow Lite | On-device ML inference on ESP32-CAM |
| face-api.js | Browser-based facial expression detection |
| Supabase | Real-time database and authentication backend |
| Netlify | Dashboard deployment |
| Vanilla HTML, CSS, and JavaScript | Dashboard frontend |
---
## System Architecture
MediBot uses a two-board architecture. The responsibilities are deliberately split between two ESP32 devices because of memory constraints that will be explained shortly.
```
┌─────────────────────────────────────────────────────────┐
│ ESP32-CAM │
│ │
│ Bed Occupancy → Face Detection → Expression Model │
│ → Eye State Model │
│ → Motion Detection │
│ │
│ Sends JSON result via UART │
└──────────────────────────┬──────────────────────────────┘
│ GPIO16
┌──────────────────────────▼──────────────────────────────┐
│ ESP32 Dev Module │
│ │
│ MAX30102 + MLX90614 → Patient State Engine │
│ → Posts to Supabase every 3s │
│ → Buzzer and Status LED │
└──────────────────────────┬──────────────────────────────┘
│
Supabase DB
│
┌──────────────────────────▼──────────────────────────────┐
│ Doctor Dashboard (Netlify PWA) │
│ Vitals · Patient State · Emotion · Alerts · Camera │
│ Works on phone, tablet, and laptop. No installation. │
└─────────────────────────────────────────────────────────┘
```
The reason for two boards comes down to memory. The standard ESP32 has 520KB of RAM. Running WiFi, sensor reads, decision logic, and TFLite models simultaneously would exceed that. The ESP32-CAM adds 4MB of PSRAM, which makes it capable of running multiple vision models sequentially. Splitting responsibilities keeps each board well within its limits while the overall system stays powerful.
---
## How It Works
### Step 1: Bed Occupancy Check
The ESP32-CAM runs a bed occupancy detection model on every frame. If the bed is confirmed empty and no vital signs have been detected for 60 seconds, all monitoring is suspended. This is what prevents false alerts in an empty ward.
### Step 2: Vital Signs Collection
The ESP32 Dev reads heart rate and SpO₂ from the MAX30102, and body temperature from the MLX90614 every two seconds. A rolling average smooths out sensor noise. Temperature alerts only fire after a reading has been abnormal for ten continuous seconds, which prevents single-spike false alarms from triggering the system unnecessarily.
### Step 3: Vision Analysis
The ESP32-CAM processes the camera feed through a sequential model pipeline. Each model only runs if the previous one confirms its precondition. If no face is detected, the expression model is skipped entirely. This saves compute and prevents false classifications.
```
Frame → Occupancy model → Face detection → Expression classifier → Eye state
```
### Step 4: Patient State Decision
The ESP32 Dev combines all signals using a layered decision engine. Emergency logic always runs first, before anything else.
```
Layer 1 — Emergency (runs first, every cycle)
Critical HR or SpO₂ or temperature → EMERGENCY
Layer 2 — Face visible
Eyes open + normal expression + normal vitals → AWAKE, comfortable
Eyes open + pain expression + abnormal vitals → IN PAIN (confirmed)
Eyes open + pain expression + normal vitals → IN PAIN (confirm manually)
Eyes closed + normal vitals + low motion → SLEEPING, well
Eyes closed + abnormal vitals → SLEEPING, possible pain
Layer 3 — Face not visible
HR variance low + HR under 70 + normal vitals → LIKELY SLEEPING
High motion + abnormal vitals → DISTRESS suspected
Any critical vital → ALERT regardless
```
When the patient is facing away from the camera and no face can be detected, the system does not give up. It uses heart rate variance to estimate whether the patient is sleeping. During sleep, the heart rate drops and becomes more regular. Low variance combined with a low resting heart rate is a clinically established indicator of sleep, and MediBot uses it as a fallback when vision fails.
### Step 5: Alerts
The buzzer provides immediate local feedback that works even when the doctor is not looking at the dashboard.
| Patient State | Buzzer Behaviour |
|---------------|-----------------|
| System alive, patient stable | Beep every 2 seconds |
| Patient confirmed sleeping | Silent, watching quietly |
| Alert confirmed | Beep every 500ms |
| Emergency | Beep every 150ms |
| WiFi not connected | Silent |
The dashboard simultaneously receives all data and displays timestamped alerts.
### Step 6: Dashboard
The doctor views the dashboard from their phone anywhere on the same network. It shows live vitals, patient state, expression, and eye state from the camera, pain score, PPG waveform, alert history, and the camera feed.
---
## The Dashboard
The dashboard is a Progressive Web App. It runs on any mobile or desktop browser with no installation needed. It was designed mobile-first because phones are the most accessible computing devices in Nigerian clinical settings.
### Camera Mode Switching
One of the things I am most proud of in MediBot is the ability to switch the vision source at runtime directly from the dashboard.
| Mode | How it works |
|------|-------------|
| Phone | face-api.js runs in the browser. Any smartphone becomes the AI vision system. |
| ESP32-CAM | The hardware camera runs all ML models on-device via TensorFlow Lite. |
| None | Vitals-only monitoring. Emergencies are still detected from sensor data alone. |
When the mode is changed on the dashboard, the ESP32 reads the update from the database within fifteen seconds and automatically adjusts its data source. No reflashing. No reconfiguring. Just a button tap.
### Security
Login credentials are stored in the database. There are no passwords anywhere in the source code. Row-level security is enabled on all database tables. Sessions clear automatically when the browser tab closes. Accounts are role-based for doctors, nurses, and administrators.
---
## The Machine Learning Pipeline
All vision models run on the ESP32-CAM using Edge Impulse and TensorFlow Lite.
| Model | Type | Input | Output | Status |
|-------|------|-------|--------|--------|
| Bed occupancy | FOMO object detection | Full frame 96×96 | occupied or empty | In training |
| Face detection | FOMO pretrained | Full frame 96×96 | Face bounding box | Ready |
| Expression classifier | MobileNetV1 0.25 | Face crop 96×96 | pain, neutral, happy, sad | In training |
| Eye state classifier | MobileNetV1 0.1 | Eye crop 48×48 | open or closed | Planned |
| Motion detection | Frame differencing | Frame delta | high, low, or none | Complete |
FOMO was chosen specifically because it was designed for microcontrollers. Standard object detectors like YOLO take three to five seconds per inference on the ESP32-CAM. FOMO runs in approximately 200ms on the same hardware. For real-time patient monitoring, that difference matters enormously.
### Expression Dataset
Training data for the expression classifier comes from FER2013 with over 35,000 facial expression images, the MRL Eye Dataset with over 84,000 open and closed eye images, and custom photos taken specifically for this project.
The custom photos were the most important part. Most public datasets use front-facing photos taken in controlled environments. In a hospital bed, the camera might be overhead, mounted on a wall, in a corner, or close to the patient at bed level. A model trained only on face-level photos performs poorly on overhead views of someone lying down. The custom dataset was collected from all angles that match real deployment.
---
## Code Samples
### False alarm prevention using a confirmation streak
A single abnormal reading is not clinically significant. This logic requires three consecutive abnormal cycles before confirming any alert.
```cpp
#define CONFIRM_CYCLES 3
int abnormalStreak = 0;
bool alertActive = false;
void updateAlertConfirmation(bool abnormal) {
if (abnormal) {
abnormalStreak++;
} else {
abnormalStreak = 0;
alertActive = false;
}
if (abnormalStreak >= CONFIRM_CYCLES) {
alertActive = true;
Serial.println("Alert confirmed");
}
}
```
### Temperature stability window
Temperature only flags after ten continuous seconds of abnormal readings.
```cpp
#define TEMP_VALID_MIN 35.0f
#define TEMP_STABLE_MS 10000
unsigned long tempLowSince = 0;
bool tempLowFlag = false;
void checkTempStability(float avgTemp) {
unsigned long now = millis();
if (avgTemp < TEMP_VALID_MIN) {
if (tempLowSince == 0) tempLowSince = now;
if (now - tempLowSince >= TEMP_STABLE_MS) {
tempLowFlag = true;
}
} else {
tempLowSince = 0;
tempLowFlag = false;
}
}
```
### Sleep detection from heart rate variance
When the patient faces away from the camera, the system uses heart rate variance to infer sleep state.
```cpp
float hrVariance = 0.0f;
int hrHistory[10];
int hrCount = 0;
void updateHRVariance(int hr) {
hrHistory[hrCount % 10] = hr;
hrCount++;
float mean = 0;
int n = min(hrCount, 10);
for (int i = 0; i < n; i++) mean += hrHistory[i];
mean /= n;
float variance = 0;
for (int i = 0; i < n; i++)
variance += pow(hrHistory[i] - mean, 2);
hrVariance = variance / n;
}
bool isLikelySleeping(int hr, int spo2) {
bool lowHR = hr > 0 && hr < 70;
bool stableHR = hrVariance < 25.0f;
bool normalSpO2 = spo2 == 0 || spo2 >= 94;
return lowHR && stableHR && normalSpO2;
}
```
### Non-blocking buzzer state machine
The buzzer never blocks the main loop. It goes silent during confirmed sleep but immediately resumes if distress is detected.
```cpp
#define BUZZER_PIN 18
unsigned long lastBuzzTime = 0;
bool buzzerOn = false;
void updateBuzzer(bool wifiOK, bool sleeping) {
unsigned long now = millis();
if (!wifiOK) {
digitalWrite(BUZZER_PIN, LOW);
return;
}
int interval, duration;
if (emergencyConfirmed) {
interval = 150; duration = 100;
} else if (alertActive) {
interval = 500; duration = 80;
} else if (sleeping) {
digitalWrite(BUZZER_PIN, LOW);
return;
} else {
interval = 2000; duration = 60;
}
if (!buzzerOn && now - lastBuzzTime >= (unsigned long)interval) {
digitalWrite(BUZZER_PIN, HIGH);
buzzerOn = true; lastBuzzTime = now;
}
if (buzzerOn && now - lastBuzzTime >= (unsigned long)duration) {
digitalWrite(BUZZER_PIN, LOW);
buzzerOn = false;
}
}
```
### Clinical expression mapping in the dashboard
Seven raw emotion outputs from face-api.js are simplified to four labels that are actually meaningful in a clinical context.
```javascript
const CLINICAL_EXPR = {
pain: { icon: '😣', label: 'Pain', color: '#ef4444' },
disgusted: { icon: '😣', label: 'Pain', color: '#ef4444' },
fearful: { icon: '😣', label: 'Pain', color: '#ef4444' },
angry: { icon: '😠', label: 'Pain', color: '#ef4444' },
happy: { icon: '😊', label: 'Happy', color: '#10b981' },
neutral: { icon: '😐', label: 'Neutral', color: '#4a6480' },
sad: { icon: '😢', label: 'Sad', color: '#f59e0b' },
};
```
### Merging two independent data sources in the dashboard
The dashboard reads sensor vitals from the ESP32 and expression data from the phone camera simultaneously, then merges them correctly.
```javascript
async function fetchLatestData() {
const [espRes, phoneRes] = await Promise.all([
fetch(VITALS_URL + '?source=eq.esp32&order=created_at.desc&limit=1',
{ headers }),
fetch(VITALS_URL + '?source=eq.phone_camera&order=created_at.desc&limit=1',
{ headers })
]);
const esp = (await espRes.json())[0];
const phone = phoneRes.ok ? (await phoneRes.json())[0] : null;
const phoneAge = phone
? (Date.now() - new Date(phone.created_at).getTime()) / 1000
: 999;
const phoneFresh = phoneAge < 30 && phone?.emotion !== 'unknown';
return {
...esp,
emotion: phoneFresh ? phone.emotion : esp.emotion,
eyes: phoneFresh ? phone.eyes : esp.eyes,
pain_score: phoneFresh ? phone.pain_score : esp.pain_score,
};
}
```
---
## Challenges Faced
### Sensor Instability
The MAX30102 is sensitive to finger placement, pressure, and ambient light. Small movements produce large reading fluctuations. The MLX90614 reads closer to room temperature when the patient is under a blanket or the sensor is not pointing directly at bare skin. The solution for both was moving average smoothing combined with the confirmation streak system. Sustained readings matter. Single spikes do not.
### Memory on the ESP32
Running WiFi, HTTPS, sensor reads, JSON serialisation, and multiple HTTP clients simultaneously on a device with 520KB of RAM required careful engineering. Making HTTP clients static so they are allocated once globally rather than on the stack for every call, and staggering requests so only one HTTP call fires per loop cycle, were what eventually made the system stable.
### Camera Angle vs Training Data
This was one of the most practically important problems in the project. Public expression datasets are captured face level with the subject looking directly at the camera. A camera mounted above a hospital bed sees something completely different. A model trained only on face level photos performs poorly on overhead views of a lying patient. Building a custom dataset from the actual deployment angles was not optional. It was necessary.
### Programming the ESP32-CAM
The ESP32-CAM has no USB-to-serial chip. Programming it required using the main ESP32 Dev Module as a programmer by bridging its EN pin to GND, crossing TX and RX lines, and holding IO0 to GND on the CAM during upload. This is not clearly documented anywhere and required significant troubleshooting to figure out.
### False Alarms in Early Testing
In early testing, the buzzer fired constantly from single-spike readings. One abnormal HR reading was enough to trigger the entire alert system. The three-cycle confirmation streak was the fix. A reading must be abnormal for three consecutive cycles before any alert fires. This eliminated almost all false alarms without meaningfully slowing down response to real emergencies.
### Merging Two Independent Data Sources
The dashboard reads from two independent sources simultaneously, such as the ESP32 posting sensor vitals and the phone camera posting vision results. Getting the merge logic right required timestamp comparison, freshness validation, and careful field prioritisation to make sure the correct source overwrites the correct fields without corrupting either dataset.
---
## Project Gallery
Vitals Overview
The Camera Connection page
The Vital Evaluation Log page
The Current Training Progress
Medibot Device
---
## Current Status
| Component | Status |
|-----------|--------|
| ESP32 vitals firmware | ✅ Complete |
| MAX30102 HR and SpO₂ | ✅ Working |
| MLX90614 temperature | ✅ Working |
| Buzzer alert system | ✅ Complete |
| Status LED | ✅ Complete |
| Dashboard PWA | ✅ Deployed |
| Login and role security | ✅ Complete |
| Camera mode switching | ✅ Complete |
| Real-time alert feed | ✅ Complete |
| ESP32-CAM flashing | 🔄 In progress |
| Bed occupancy model | 🔄 Training |
| Expression classifier | 🔄 Dataset collection |
| Eye state model | 📋 Planned |
| Offline local mode | 📋 Planned |
| Clinical validation | 📋 Planned |
---
## What Comes Next
The immediate focus is on completing the ESP32-CAM vision pipeline, deploying the trained ML models to hardware, and beginning clinical validation in real patient environments.
Long term, the goal is an offline-first version where the ESP32 hosts the dashboard locally so the entire system works without any internet connection. Hospitals with unreliable network access should not have to choose between monitoring and connectivity.
---
## Closing
The monitoring gap in Nigerian hospitals is not only a funding problem. It is also an engineering problem, and engineering problems have engineering solutions.
MediBot is one attempt at building that solution around the realities that actually exist here. Affordable hardware. Local components. Open-source software. Infrastructure hospitals already have.
The talent to build these systems already exists in Nigeria. What has been missing is infrastructure, research continuity, and systems designed specifically for the environments where the need is real.
---
## References
References
[1] Runcie CW Chidebe et al. (2023). Brain Drain in Cancer Care: The Shrinking Clinical Oncology Workforce in Nigeria. Journal of Clinical Oncology. Available at pmc.ncbi.nlm.nih.gov/articles/PMC10752460
[2] Nickcy M. et al. (2024). General ward nurses detection and response to clinical deterioration in three hospitals at the Kenyan coast. BMC Nursing. Available at pmc.ncbi.nlm.nih.gov/articles/PMC10905788
[3] Peter M. Nthumba et al. cited in: Patient safety in a rural sub-Saharan Africa hospital. PLOS Global Public Health (2024). doi.org/10.1371/journal.pgph.0003919
ESP32
TensorFlow Lite
Edge Impulse
face-api.js
Supabase
Arduino C++
Netlify
