A Pathological Diagnosis Method Combining Image-Text Large Model and Rule-Based Reasoning

Biopsy is considered the “gold-standard” for disease diagnosis. In clinical practice, it is necessary to scan biopsy tissue slices into electronic image, which allows for cooperative reasoning from multiple perspectives of the induction of the morphology and structure of cells and tissues, as well as the deduction of the cross disciplinary knowledge of clinical and pathological. Multimodal image-text-pair based large models have improved the ability of generalization for computational pathological analysis, but lack the diagnostic reasoning ability. To address the above limitation, a pathology-base diagnosis method combining large models and rule-based reasoning is proposed in this paper to take both performance of generalization and reasoning into account. The reasoning rules, summarized by experts to reflect medical domain knowledge, serve as the core components for diagnostic decisions. Specifically, clinical practice guidelines are transformed into rules of Datalog which is a reasoning engine based on first-order predicate logic. Based on electronic images obtained from scanning biopsy tissue samples, features extracted from pathological analysis using pre-trained large image-text models are treated as facts of Datalog for rule- fact matching during reasoning. In order to verify the usability of our proposed method, a case study of a downstream task of liver fibrosis staging is demonstrated, and comparisons are conducted among different studies.