Andrew J. McDonald

Professional project 4 min read

AR Telehealth with HoloLens 2

A proof of concept that let a remote doctor see, hear and monitor a patient through a nurse's HoloLens 2.

Painted collage of a nurse in a mixed reality headset holding a stethoscope to an elderly patient in bed, while floating panels show a remote doctor and the patient's vital signs

From 2022 into 2023 I worked as lead developer on an augmented reality telehealth prototype, a collaboration between La Trobe University, Cisco Innovation Central Melbourne and Northern Health’s Victorian Virtual Emergency Department. I was employed by La Trobe, and my role was funded by Cisco. The aim was to bring a doctor’s eyes and ears to a patient in another place.

Northern Health and La Trobe set up the Virtual ED in 2020, during the COVID-19 pandemic, to give non-urgent patients emergency care by video from anywhere in Victoria. This helps places where a doctor is hard to get in person, such as remote clinics and aged care homes. In aged care, almost any event leads to an ambulance trip, even when the visit is only for a check-up and back. The trip is hard on an elderly resident, and it ties up an ambulance that could go to a more urgent case.

A video consultation can avoid that trip, but it still needs someone at the patient’s side to hold the camera. In an aged care home that often means two staff members: a nurse with the resident, and a second person filming with a phone or tablet. Staff are scarce, so the Virtual ED’s challenge to us was to free the nurse’s hands.

Our answer was to put a Microsoft HoloLens 2 headset on the nurse. Through it, the doctor could see and hear what the nurse did, including the sound of a digital stethoscope, and watch the patient’s vital signs arrive live from Bluetooth medical devices. La Trobe described the aim as letting aged care residents be “diagnosed, treated and monitored in situ without the cost and distress of having to be transported to hospital”.

How it worked

A man in scrubs wearing a HoloLens 2 stands beside a seated woman who holds a stethoscope to her chest; behind them a monitor shows two people on a video call beside a panel of readings
The prototype in use. Photo: La Trobe University.

The nurse wore the HoloLens 2, which opened our web portal in its browser and joined a video call through it. The doctor opened the same portal on an ordinary computer. The portal showed the call on the left and the patient’s live readings on the right: blood pressure, blood oxygen, heart rate, temperature and a graph of the pulse wave. We ran the call on Jitsi, an open-source video-conferencing server, hosted on Google Cloud.

The stethoscope was central to the project. Northern Health told us that if a remote doctor could hear a patient’s breathing clearly through a stethoscope over the video link, it would give their doctors a tool they did not have. Heart sounds were useful, but breath sounds mattered more. We used a Thinklabs digital stethoscope and fed its sound into the call. Jitsi normally filters call audio to favour voices, so we turned that filtering off to carry the stethoscope’s sound at full fidelity.

The readings came from three iHealth Bluetooth devices: a blood pressure monitor, a pulse oximeter and a thermometer. An Android phone read them through iHealth’s software development kit, and we added an MQTT client to the phone’s app so that it published each reading to a RabbitMQ message broker. The portal subscribed to the broker and updated as each reading arrived.

We hosted every part of the system ourselves. It carried medical data, so we wanted to control the whole pipeline, from the devices to the doctor’s screen, and keep that data private.

Outcome

The prototype became the flagship demonstration of La Trobe’s innovation space at the Bundoora campus, and it was in constant use to show what the team and the space could do. We demonstrated it to strategic partners and at technology and medical events, at La Trobe and at other sites.

A doctor wearing a HoloLens 2 sits beside a stand-in patient in a hospital bed, while a group of people watch from around the bed
A demonstration at La Trobe. Photo: Innovation Central Melbourne.

In 2022, Cisco made the project the subject of a short film, A university, a network, and three brilliant students, and featured it on its education blog. In May 2023, ABC News reported on the project, and La Trobe published a media release about it.

Cisco's film about the project, A university, a network, and three brilliant students, 2 minutes 32 seconds
Cisco's film about the project, 2022.

Coverage