Fine-Tuning of Large Language Models (LLMs)
Authors : Prof. Shankar Gadhve , Tapashya Pandit, Megha Ghodke, Sakshi Hudke, Vaishnavi Swami, Samuel Rodrigues
DOI : 10.46335/IJIES.2024.9.5.13
Abstract : By merging self-supervised Language Modeling with supervised Machine Translation aims, the paper investigates a unique method for pre-training Large Language Models (LLMs). By utilizing cross-lingual parallel data, this hybrid pre-training approach produces LLMs with enhanced in-context learning capabilities. In addition, the paper reports on the results of optimizing Mistral 7B, a general-purpose LLM, for translation adaptation. As compared to other LLMs, the results show competitive results and significant quality improvements in both zero- shot and one-shot translation scenarios, outperforming baseline performance. These results demonstrate the effectiveness of hybrid pre-training and fine-tuning in improving LLMs; translation quality and adaptability.
