La-Math-ex: LaTeX Code Generation for Handwritten Mathematical Expressions

We present a novel approach to offline handwritten mathematical expression (HME) recognition and LaTeX code generation by fine-tuning Microsoft's TrOCR-Base vision-language Transformer on the MathWriting dataset. The TrOCR architecture couples a BEiT-initialized image Transformer encoder processing inputs as sequences of $16 \times 16$ pixel patches augmented with absolute positional embeddings with a RoBERTa initialized text Transformer decoder that autoregressively emits LaTeX tokens. Each sample comprises a handwritten formula paired with both raw and normalized LaTeX annotations to facilitate learning of structural conventions. Training was conducted using the Hugging Face Seq2Seq Trainer with cross-entropy loss and teacher-forcing. Our model achieves a character error rate (CER) of 4.72% and an exact-match formula accuracy of 75.0% on held-out test data, representing a significant improvement over traditional OCR engines (CER $\approx 12$%) and CTC-based Transformer baselines (exact-match $\approx 58 {\%}$). We analyze error patterns highlighting challenges in nested superscripting and ambiguous stroke segmentation and demonstrate that large pre-trained vision-language models can be effectively specialized for two-dimensional notation.