WeedNet-ViT: A Vision Transformer Approach for Robust Weed Classification in Smart Farming

Weeds continue to pose a serious challenge to agriculture, reducing both the productivity and quality of crops. In this paper, we explore how modern deep learning, specifically Vision Transformers (ViTs), can help address this issue through fast and accurate weed classification. We developed a transformer-based model trained on the DeepWeeds dataset, which contains images of nine different weed species collected under various environmental conditions, such as changes in lighting and weather. By leveraging the ViT architecture, the model is able to capture complex patterns and spatial details in high-resolution images, leading to improved prediction accuracy. We also examined the effects of model optimization techniques, including fine-tuning and the use of pre-trained weights, along with different strategies for handling class imbalance. While traditional oversampling actually hurt performance, dropping accuracy to 94%, using class weights alongside strong data augmentation boosted accuracy to 96.9%. Overall, our ViT model outperformed standard Convolutional Neural Networks, achieving 96.9% accuracy on the held-out test set. Attention-based saliency maps were inspected to confirm that predictions were driven by weed regions, and model consistency under location shift and capture perturbations was assessed using the diverse acquisition sites in DeepWeeds. These findings show that with the right combination of model architecture and training strategies, Vision Transformers can offer a powerful solution for smarter weed detection and more efficient farming practices.