Compression and Fine-Tuning of Large Language Models

Large Language Models (LLMs) such as GPT, LLaMA, and BERT deliver state-of-the-art performance but require high memory and computation, limiting use on resource-constrained devices. This paper presents a unified compression and fine-tuning pipeline integrating ultra-low-bit quantization (3-bit/4-bit) using BitsAndBytes, Activation-Wise Quantization (AWQ), and Quantized Low-Rank Adaptation (QLoRA). A 14B-parameter LLM is compressed to 8–10 GB, fine-tuned on downstream tasks, and evaluated on GSM8K, OpenBookQA, and SQuAD. The system achieves up to 75% model size reduction, 2× faster inference, and 2% accuracy drop. Stability limits in 3-bit quantization and fairness effects on StereoSet are also analyzed. The results show that hybrid quantization–fine-tuning enables efficient, high-performance LLM deployment on edge devices with minimal accuracy loss.