Diabetic retinopathy detection system using hybrid convolutional neural network

Diabetic retinopathy (DR) screening systems often drop 15–20% of performance when moving different clinics because of the data differences. Pure CNNs often miss the big picture, while Transformers overlook tiny details. To tackle this, we ran a head to head test of hybrid CNN-Transformer models across datasets. We trained six top models—classic CNNs like InceptionV3, ResNet50, and DenseNet121; pure Transformer (ViTBase); and hybrids ConvNeXt and MaxViT—on 35,126 EyePACS images, using smart sampling and targeted augmentations to handle imbalance in the dataset. We then tested them cold on 3,662 APTOS and 1,748 Messidor images. MaxViT stood out with a 91.8% AUC (95% CI: 90.5–93.1%), beating CNNs by 1.5–2.9%, slashing cross-dataset performance dips by 38% (p < 0.001; just 2.5% drop vs. 4.2% for ResNet50), and cutting false positives by 15% thanks to its clever grid-window attention that nails both micro-lesions and the overall retinal pattern.