Vision Transformer with ECA and CBAM for Underrepresented Class Prediction in Diabetic Retinopathy

Classification of DR using Deep learning contributes to early intervention; however, class imbalance may have an impact on the performance. The paper presents the results of applying two attention mechanisms, such as Efficient Channel Attention (ECA) and Convolutional Block Attention Module (CBAM), to a pre-trained Vision Transformer (ViT) to perform classification of four degree levels of DR, and the paper refers to classes with disproportionate representation. Three models were used- baseline ViT, ViT+ECA, ViT+CBAM, ViT+CBAM+ECA. The evaluation was performed using metrics like accuracy, macro F1-score, confusion matrices and per-class recall. Although the baseline ViT performance was inferior in terms of accuracy (70%) and macro F1-score (68%), the ViT+CBAM+ECA model demonstrated better recall in the minority classes (75% Mild DR and 69% Proliferative DR), but had decreased overall and macro F1-scores (62% and 63% respectively). This change of performance was backed up by confusion matrices. This demonstrates a trade-off because the inclusion of the ECA and CBAM focus in the model can decrease the overall accuracy by improving sensitivity to underrepresented classes, which makes the technique suitable in targeted DR detection.