Malignant tumors, a complex and often deadly disease, have seen an upsetting rise in younger populations in recent years, making them the leading cause of death globally. With advancements in medical standards and heightened public awareness, cancer screening and diagnosis have captured more attention. This, however, has led to mounting workloads and increasing diagnostic pressures for doctors, particularly in specialized fields like pathology.

To address this challenge, Professor ZHANG Jing’s team from the First Affiliated Hospital of Zhejiang University School of Medicine (FAHZU) in collaboration with Professor SONG Mingli’s team from Zhejiang University’s College of Computer Science and Technology, set out to leverage AI technology to enhance clinical diagnostics in 2020, aiming to improve both the efficiency of healthcare providers and the overall quality of care.

This year, the team unveiled the OmniPT, an AI-powered pathology assistant that integrates visual and language models. This tool has been clinically validated at FAHZU, where it has successfully identified cancer lesions in pathology images in just 1 to 3 seconds, streamlining the diagnostic process for several high-incidence cancers.

Today, let’s take a closer look at how this innovative AI tool is transforming cancer diagnosis, bringing a new level of intelligence to the battle against cancer.

Teaching machines to “understand” doctors

For the team led by Professor SONG Mingli, with its foundation in computer science, the path to interdisciplinary research began with a steep learning curve. Grasping the basics of medicine and unraveling the intricate world of pathology presented their first major challenge.

“As compared to reading X-rays or CT scans, pathology diagnosis is far more complex,” explained ZHANG Xiuming, Deputy Chief Physician at the FAHZU Pathology Department. “It requires not only precise qualitative and quantitative analysis but also the ability to navigate the vast diversity of tissue structures.” He continued, “While a single cancer tissue slide might be no larger than a fingernail, the scanned images from multiple slices stretch into the billions of pixels, vastly larger than a phone photo, a challenge in its own right.”

A pathologist examines cellular morphology to diagnose tumors.

Within these billions of microscopic images lies an ocean of life, an intricate web of biological information and medical secrets. Pathologists spend hours combing through these slides, scrutinizing every detail. But the demands of this meticulous work can result in visual fatigue. Moreover, because early-stage cancerous cells are sparse, the risk of missed diagnoses becomes one of the most pressing concerns for medical professionals.

Digital pathology slide scanning system at FAHZU.

The AI assistant has undoubtedly offered new hope, breathing fresh life into the work of doctors and the fight against cancer. Drawing on a vast pool of medical literature and engaging in deep, ongoing conversations with ZHANG Jing’s team, SONG Mingli’s team began weaving together the threads of pathology knowledge.

But the question loomed large: how could they translate human understanding into a language that machines could “comprehend”?

The answer lay in a painstaking process of distilling the morphological patterns of cancerous tissue, annotating and classifying images with precision, and refining algorithms for tasks like cell detection and tissue segmentation. After four years of tireless development, the result was OmniPT — a specialized diagnostic tool for a range of cancers.

To improve reasoning efficiency, OmniPT employs a cross-layer fast-locking technique when processing ultra-large pathology images. It first identifies a broad swath of suspicious regions and then zooms in step-by-step to analyze finer details.

FENG Zunlei, an associate professor from the School of Software Technology, described it as a “key-point-focused” approach. “This method mirrors how pathologists think when diagnosing,” he said. “And it can significantly boost the efficiency of intelligent pathology analysis.” What once took a doctor 10 minutes to analyze, OmniPT can achieve in just 1 to 3 seconds, swiftly locking onto cancerous lesions with pinpoint accuracy.

After numerous trials and refinements, OmniPT overcame the daunting challenge of processing ultra-large pathology images with real-time, accurate analysis. It now boasts over 95% diagnostic accuracy for high-incidence cancers such as gastric, colorectal, and cervical cancer, providing invaluable support for pathologists and faster, more reliable results for patients.

Human-AI collaboration empowering research

“The design of OmniPT is fully aligned with the habits and needs of doctors,” said FENG Zunlei. “It blends text and visuals seamlessly, like a smart companion attuned to the nuances of the task at hand.”

By simply inputting text or drawing bounding boxes on images, doctors can quickly pinpoint areas of interest for further examination. The breakthrough comes from a cross-layer, high-efficiency feature-locking technology, which enables a seamless “dialogue” between the doctor and OmniPT. This collaboration allows doctors to guide the AI in focusing on the details that matter most, enhancing image diagnosis with more precision and intelligence.

Demonstration of an AI pathology assistant supporting pathologists in diagnostic analysis tasks.

As clinical data continues to grow and algorithm models evolve, OmniPT has gained invaluable insights from tens of thousands of cancer cases, particularly those involving high-incidence cancers. It has gradually transformed into a sophisticated senior consultant with the ability to analyze prognosis and recommend customized treatment plans. These insights are based on each patient’s specific condition and pathology, helping to predict the likely outcomes of various treatment options.

With such a senior consultant at their side, doctors can now develop more comprehensive and reliable treatment plans, grounded in the AI’s tailored recommendations.

From clinical work to research, OmniPT is honing in on the microscopic processes that drive the onset and development of a cancer. The AI’s ability to analyze tumor marker has opened up new avenues for understanding tumor biology and clinical prognosis. These markers are crucial for developing personalized treatment strategies.

Professor ZHANG Jing shares progress in the development and application of large pathology models at the annual pathology conference.

Through advanced computing, OmniPT quantitatively analyzes cell morphology and tissue structures from high-resolution pathology images, successfully identifying several novel tumor markers with significant potential for clinical applications. “These markers offer huge value in early cancer detection, classification, and prognosis,” said ZHANG Xiuming, a leading researcher on the project.

OmniPT has also uncovered potential links between these markers and the progression of tumors, offering fresh perspectives and methods for diagnosing, treating, and even designing drugs. This lays a solid foundation for future breakthroughs in cancer research.

“We aim to contribute to the refinement of clinical diagnostic guidelines, providing scientific evidence to inform their development,” said FENG Zunlei. “This will enhance the standardization and effectiveness of cancer diagnosis and treatment, ultimately leading to better patient outcomes.”

A pressing question surrounding the rise of AI in medicine is whether it could replace human doctors. The team behind OmniPT is clear: the answer is no. “AI’s role is to assist doctors, not replace them,” said ZHANG Jing. “While algorithms and data are invaluable, the final diagnosis and treatment decisions rest with the experience and judgment of medical professionals.”

Adapted and translated from the article by DAN Hanyu, WANG Yun, and ZHA Meng  
Translator: FANG Fumin
Photo: The research team 
Editor: TIAN Minjie