A Study on Self-Supervised Pretraining of Large Language Models on Higher Education Text Data

With the rapid advancement of natural language processing (NLP), large language models (LLMs) have demonstrated remarkable performance across various domains. However, existing LLMs are primarily pre-trained on general-purpose corpora, limiting their effectiveness in higher education contexts that require domain-specific knowledge and complex textual structures. This study explores a self-supervised pretraining framework tailored for higher education text data, integrating masked language modeling (MLM) and contrastive learning to enhance academic text comprehension and generation. We construct a specialized corpus encompassing textbooks, research papers, and course syllabi, ensuring domain relevance and linguistic diversity. Experimental evaluations on academic summarization, automated grading, and scholarly question-answering tasks demonstrate that the domain-adaptive LLM outperforms general-purpose models in accuracy and coherence. Additionally, we analyze key factors influencing model performance, including data quality, training efficiency, and generalization across disciplines. The findings highlight the potential of self-supervised LLMs to advance intelligent educational applications while also identifying challenges such as computational costs and explainability.