SiCal-CytoNet: A Stain-Invariant, Calibrated Multi-Scale Network for Trustworthy Cytology on SIPaKMeD

Abstract

Background and Objectives: Cervical cancer screening through Pap-smear cytology remains a cornerstone of early diagnosis. However, its large-scale deployment is constrained by stain and illumination variability, morphological complexity across spatial scales, class imbalance, and unreliable confidence estimation in automated systems. Although deep learning models have achieved high classification accuracy on cytology benchmarks, many lack robustness to domain shifts and produce poorly calibrated probabilities, limiting their suitability for clinical triage. The objective of this research was to develop a stain-invariant, multi-scale, and well-calibrated deep learning framework that delivers both strong discrimination and trustworthy probability estimates for cervical cell classification, thereby enabling safe and effective decision support in screening workflows.

Methodology: This study proposes SiCal-CytoNet, a stain-aware and calibrated multi-scale network designed for Pap-smear cytology analysis. This framework employs optical-density–based stain deconvolution with controlled perturbations to address stain variability in combination with domain generalization strategies, including Fourier Domain Adaptation and MixStyle. Morphological features are learned at multiple resolutions and adaptively fused using a learnable resolution gate. A prototype-guided metric head improves class compactness under imbalance, while probability reliability is enhanced through a combination of Brier loss, evidential Dirichlet modeling, and post-hoc temperature scaling. A selective prediction mechanism based on calibrated confidence is integrated to support clinically meaningful abstention.

Main Results: On the SIPaKMeD five-class benchmark, SiCal-CytoNet achieves a test accuracy of 98.9% and a macro-F1 score of 98.7%, with excellent ranking performance reflected by an AUROC of 99.7% and an AUPRC of 99.4%. This model demonstrates strong probability calibration, attaining a low Brier score of 0.012 and an expected calibration error of 1.7%, while maintaining high discrimination under simulated stain, style, blur, and illumination shifts. Ablation experiments confirm the importance of multi-scale fusion, domain generalization, and calibration components. Using a utility-optimized selective prediction strategy, the system confidently automates 88.1% of cases while preserving high sensitivity, specificity, and predictive values for clinical triage.

Conclusions: The findings show that combining stain physics, adaptive multi-scale learning, prototype-guided classification, and explicit calibration produces a cytology model that is both highly accurate and clinically trustworthy. SiCal-CytoNet moves beyond accuracy-centric evaluation by delivering reliable probability estimates and robust performance under domain shifts, addressing critical requirements for real-world deployment in cervical cancer screening.

Practical Application: SiCal-CytoNet can be deployed as an automated triage system in cytology laboratories to assist pathologists by rapidly classifying routine Pap-smear samples while referring uncertain cases for expert review. Its calibrated outputs support transparent threshold selection, reduce screening workload, and enhance consistency across laboratories with heterogeneous staining conditions, offering tangible benefits to public health screening programs and AI-assisted diagnostic platforms.