To be able to literally “see inside” a patient’s skull and identify the location of the tumour deep in the brain, right on the operating table, may no longer be science fiction with the help of a Mixed Reality (MR) headset.

The National University Health System (NUHS) has embarked on a research and development programme within the academic healthcare cluster to explore the use of MR technology in clinical care. The research programme hopes to support the development of next-generation clinical applications and improve patient safety. This would augment clinical processes, and enhance both undergraduate and postgraduate education. While the use of holographic technology in operating theatres is still nascent, NUHS hopes to apply it in multiple fields of surgery.

“Holographic technology may radically transform the way we practice medicine. Early adoption will place NUHS at the forefront of medical MR research and position us as a pioneer in the clinical use of this technology,” said Associate Professor Ngiam Kee Yuan, NUHS’s Group Chief Technology Officer, who is overseeing the research and development of the holomedicine programme in NUHS.

While holomedicine is new in the field of medicine, it has become increasingly prominent in the past year. “It leverages the concept of MR not only to augment our physical environment, but permits interaction with virtual objects superimposed onto the real world. The virtual objects can also be manipulated relative to the real world using natural hand gestures,” explained Dr Gao Yujia, Associate Consultant with the Division of Hepatobiliary & Pancreatic Surgery, National University Hospital (NUH), and the programme lead for holomedicine at NUHS.

Potential of holographic technology in brain surgery

A team of neurosurgeons at NUH has initiated a study to assess the feasibility of using holographic technology to spatially locate brain tumours when operating on patients.

Using holographic visors^[1], a three-dimensional (3D) hologram of a patient’s brain scan is projected into space and superimposed onto the patient’s head during surgery. This hologram is generated from the patient’s own brain CT scan using a 3D medical software created by apoQlar, called Virtual Surgery Intelligence (VSI). When the hologram is set in place on a patient’s head, surgeons are able to view the 3D holographic images of the brain from different angles.

They can also pull up information and alter the images in the visor by gesture and speech recognition, allowing them to interact with, and control the holographic image that is superimposed onto the patient.

Clinical Associate Professor Yeo Tseng Tsai, Head and Senior Consultant with the Division of Neurosurgery, NUH said, “With this holographic technology, you are able to see inside the brain. You will be able to see the blood vessels and most importantly identify the tumour quickly and precisely, as well as to know which angle and the exact location to make the incision. For over thirty years now, we have been using a handheld navigation system to navigate and identify the location of the tumour. In comparison, this new mixed reality system is more intuitive as we can now see inside the patient’s head without the need to look up and refer to a computer screen while performing a procedure.”

With the potential of higher precision and quicker localisation, patient care and safety can be improved. Furthermore, with the MR headset weighing only around 500 grams, it may one day replace existing bulky operation theatre equipment and even reduce the exposure to radiation in procedures such as spine operations, where X-rays are currently used to guide the insertion of metallic implants.

NUHS has completed the first phase of the research programme involving a proof of concept study, where various disciplines such as neurosurgery, plastics surgery and ophthalmology at NUH have evaluated the MR devices and brainstormed on future areas of development. The headsets were used for evaluation purposes to identify use cases, understanding of clinicians’ acceptance level, system evaluation, as well as assessing its long-term deployment sustainability.

NUHS has developed a holistic holomedicine roadmap, including short- and long-term research projects, integration with existing hospital systems, and procedures to onboard users and enhance the hospital’s infrastructure to support the system. Some potential use cases include streaming of live data from image acquisition machines, and the employment of artificial intelligence and machine learning for advanced image processing and predictive analytics. In addition, the HoloLens 2 device may also be used by patients to help them better understand the procedures they are to undergo. This is achieved by projecting a 3D rendering of a patient’s scan to better illustrate the steps of their procedure.

Clinical validation studies and trials will be necessary prior to the adoption of this holomedicine solution as a primary clinical method. This involves comparing MR technology to current gold standards of clinical practice and outcomes measured, including data on accuracy, stability, and potential risks and limitations of this capability. Registration with the necessary governing bodies including the Health Science Authority (HSA) will also be needed before this solution can be used in a direct patient intervention role.

Collaboration with industry

The NUHS team, in collaboration with the Engineering Design & Innovation Centre under the National University of Singapore’s (NUS) Faculty of Engineering, has also been awarded a grant of $100,000 from the Engineering in Medicine Grant under NUHS and NUS in March 2021. The grant will enable the team to embark on a project on real-time volumetric rendering and positioning of ultrasound scans.

Industry collaboration is an integral part of the research and development of holomedicine in NUHS. NUHS has been collaborating with Microsoft and apoQlar in its holomedicine efforts. Moving forward, NUHS will continue to build on its existing partnerships and explore new industry collaborations to push the boundaries of MR technology in the healthcare environment.

NUHS is also privileged to be a founding member of The Holomedicine Association, an international organisation comprising clinicians, scientists and industry partners who are engaged in active research and development of holomedicine. With 12 members on the founding committee including the executive board of the association, NUHS is well poised to participate in collaborations with other hospitals, research institutes, and industry partners around the world.

“We are not just an end-user of holomedicine; we hope to be an active developer and to validate its use in medicine. We will work with our partners to make holomedicine customisable, more user-friendly and clinically applicable,” said A/Prof Ngiam Kee Yuan.

“The experience of the past year shows us that technology can empower healthcare workers and assist them to help protect and save the lives of patients. NUHS’s holomedicine study bears further testament to how innovative use of technology, such as Mixed Reality solutions and Microsoft HoloLens 2, may have a truly transformational impact in healthcare. Technology-enabled neurosurgeons may be able to perform safer procedures, enable improved outcomes and ultimately provide better patient care. We are proud to work with NUHS and other clinicians in the region to explore the full potential of digital technology in holomedicine across multiple therapy areas in the years to come,” said Dr. Keren Priyadarshini, Regional Business Lead, Worldwide Health, Microsoft Asia.

“Holomedicine has the potential to sustainably revolutionise medical standards in most – if not all – medical fields. We are very excited to join forces with NUHS, a world-renowned academic health system, to achieve our mission of further improving patient outcomes and safety,” said Mr Sirko Pelzl, CEO, apoQlar.

The software by apoQlar has successfully obtained HSA’s Class A License and the Singapore Standard SS620 certification in April 2021. NUHS hopes to promulgate holographic technology to all its hospitals and institutes within the healthcare cluster in the near future.

“The advancements in technology, especially in the field of MR, has opened new horizons to the medical profession. By merging other technologies such as artificial intelligence, real-time image recognition, and predictive modelling, these MR devices offer clinicians capabilities that were once thought impossible,” said A/Prof Ngiam Kee Yuan.