A Lightweight Neural Network with Transformer to Predict Credit Default

Financial institutions desperately need accurate credit default prediction tools to minimize losses, optimize lending and foster responsible borrowing. Existing methods often rely on complex ensemble approaches which are computationally intensive. This paper proposes a relatively lightweight solution: a simple encoder-only transformer which extracts latent features directly from credit card transaction data, followed by a simple neural network for the credit default prediction. The proposed approach achieves comparable performance on the American Express (AMEX) Default Prediction Kaggle Competition dataset against state-of-the-art (SOTA) ensemble techniques such as 3-Ensemble Modules with Gradient Boosting Decision Tree (GBDT) and 3-LightGBM with Dropouts meet Multiple Additive Regression Trees (DART) and pseudo-labelling ensemble methods. It achieves better performance than LightGBM based method.