Advanced Deep Learning Algorithms for Medical Imaging Applications: Tumor and Organ at Risk Segmentation

Accurate segmentation of tumors and organs at risk (OAR) remains a fundamental challenge in medical image analysis, directly affecting diagnostic accuracy, treatment planning, and patient outcomes. Although automated segmentation methods have advanced, they still face difficulties in anatomically complex regions and in cases with low tissue contrast. A major contributor to this variability is intra- and interobserver disagreement, introducing uncertainty into annotations and model predictions. This dissertation investigates the role of segmentation uncertainty and its impact on model reliability, interpretability, and clinical integration, with a focus on lung, pancreatic, and head and neck cancers. The first component introduces uncertainty-aware segmentation models that incorporate probabilistic outputs and spatially guided loss functions. A novel Uncertainty-Guided Coarse-to-Fine framework enhances both conventional and transformer-based architectures, improving performance in ambiguous regions. Additionally, self-supervised pretraining is employed to improve feature representation and generalization, particularly in settings with limited labeled data. The second component examines how tumor-specific characteristics relate to segmentation performance. Radiomic features emerge as more reliable indicators of case-level difficulty than traditional descriptors. These insights inform the design of expert-in-the-loop systems for selective case review and model refinement. The final component extends uncertainty modeling to clinical outcome prediction. A hybrid strategy that combines deep neural embeddings with handcrafted radiomic features is applied to two tasks: predicting three-year survival for non-small cell lung cancer and stratifying the risk of pancreatic cysts using MRI. Notably, a correlation is observed between peritumoral uncertainty and prognostic value in lung tumors. This dissertation presents clinically grounded, uncertainty-aware frameworks for segmentation and outcome prediction. By integrating anatomical priors, spatial uncertainty, and radiomic interpretability, it advances robust solutions for improving segmentation accuracy and supporting precision oncology through predictive modeling.