Scientific Publication Indexing and Definition Extraction Resource

Defining relevant concepts from scientific texts is always challenging because of the complex and large amount of information, especially in fast-growing disciplines. We propose SPIDER (Scientific Publication Indexing and Definition Extraction Resource), a tool that can automatically index and define scientific papers in portable document format (PDF) and text format. The primary task of SPIDER is to address the issue of understanding massive scientific documents and further improves the accessibility of scientific information. By applying the state-of-the-art natural language processing (NLP) techniques and self-contained large-language models(LLM) such as Ollama, phi3, and mistral; SPIDER increases the readability of the scientific contents. The model uses the conversion of large PDF document to text, and then it applies NLP methods such as tokenization, stemming, and part-of-speech tagging to create an accurate glossary that is further fine-tuned with the help of self-hosted LLMs for detailed description. The SPIDER method uses NLP modules such as PyPDF2 for PDF handling, NLTK for NLP operations, and transformers for state-of-the-art language modeling to efficiently comprehend scientific articles. The paper describes the SPIDER approach to data cleaning, tokenizer, stemmer, PoS tagger, and the use of self-hosted LLMs. Potentially this model could benefit the enhancement of various fields such as speeding up a literature review, enhancement of educational materials, creating comprehensive glossaries, and supporting the development of knowledge-based systems in scientific and technological disciplines. The result shows SPIDER's effectiveness in generating accurate glossaries achieving optimal performance with a threshold range from 0.1 to 0.4. Thus, SPIDER can be considered a useful tool for researchers, professors, and others who aim to handle a large amount of scientific information.