Comprehensive Evaluation of Transformer Models for Multilingual PII Detection

Personal Identifiable Information (PII) detection is a critical component of data governance and regulatory compliance across enterprises handling multilingual structured datasets. Traditional rule-based approaches, such as regular expressions (RegEx), are computationally efficient but perform poorly in multilingual contexts, particularly when working with structured data that lacks column headers. In such cases, entities must be inferred directly from values, and languages like Japanese and Korean further complicate detection due to multi-byte characters. This paper proposes utilization of transformer-based models to enhance PII detection in structured enterprise data. By fine-tuning multilingual models such as mBERT and language-specific models including KLUE-BERT (Korean), we demonstrate high detection accuracy while maintaining moderate model sizes and computational cost. Synthetic datasets were generated in Japanese, Korean, and English to simulate real-world PII entity diversity. Experimental results show that monolingual fine-tuned models significantly outperform both RegEx and multilingual models, achieving over 92 % accuracy across languages. The findings support the adoption of transformer-based models in modern enterprise data catalog systems for multilingual PII detection and data governance.