Marru(convert): Structure Data Creation from Unstructured Text for Fine-tuning Large Language Models in Indian Languages

This research presents an innovative methodology to convert unstructured text into a structured question-and-answer format for fine-tuning Large Language Models (LLMs) in 11 Indian languages. Addressing the scarcity of high-quality, task-specific datasets in Indian languages, the study employs a ternary quantized model based on the LLaMA-2 architecture to achieve efficient data transformation. Key contributions include the development of a parent-child text chunking system, the use of cosine similarity for enhanced data relevance, and training on 47 million tokens formatted in Alpaca style. The research further evaluates the methodology using Grice’s Maxims and AI-assisted techniques to ensure the quality and contextual accuracy of the generated datasets. This work has significant implications for improving LLM applications across domains such as education, healthcare, legal, and financial services, fostering linguistic inclusivity and technological advancement in underrepresented languages.