Efficiënte kennisoverdracht en -injectie voor het begrijpen van documenten in geval van beperkte informatiebronnen

In today's digital age, people frequently interact with a vast array of documents to gather information, make decisions, and gain insights. Traditionally, this involves meticulous efforts of reading, extracting key elements, comprehending content, and summarizing. However, the advent of natural language processing (NLP) technologies is transforming this paradigm. These challenges are particularly evident in the clinical and financial domains, where extracting critical information can be time-consuming and labor-intensive. Tasks such as cohort selection for drug trials or research necessitate advanced natural language understanding and document comprehension. Despite significant progress in NLP, there remains a noticeable gap between the performance of NLP models in controlled research settings and their effectiveness in real-world applications. This thesis aims to bridge this gap, ensuring NLP techniques are robust and reliable in practical scenarios, especially when annotated data is scarce. Our research addresses the critical issues of deploying deep learning (DL) pipelines in real world applications where access to annotated data is limited. State-of-the-art models often underperform in such applications due to several factors: limited annotated data from the target domain, noisy annotations, data complexity, domain variability, the necessity for task-specific solutions, low-resource languages, lack of domain expertise, and privacy constraints. This thesis explores three key areas to mitigate these challenges: transfer learning, domain and task-aware adaptive pretraining, augmentations, and knowledge-injected learning. This thesis introduces systematic experiments on efficient task and domain-specific knowledge transfer and injection, exploring various transfer learning strategies and domain adaptive pretraining methods across financial, biomedical, and multilingual settings. Firstly, we leverage transfer learning to adapt pre-trained transformer-based models to new tasks and domains with limited labeled data. We integrate word and character-level representations with gated attention mechanisms together with Bidirectional Encoder Representations from Transformers (BERT) embeddings to evaluate their contextual understanding in the financial and biomedical domains. Secondly, we employ domain adaptive pretraining to train models on domain-specific data, utilizing methods such as selective masking and Gaussian embeddings to enhance token representations. Multilingual translations are also utilized as augmentations to improve representations in low-resource settings. We further investigated task-adaptive contrastive pretraining, which improves model robustness by distinguishing between similar and dissimilar examples, enhancing token category similarity in sentences. In the clinical domain, particularly for clinical electronic health records (EHRs), we developed a comprehensive pipeline for a named entity recognition (NER) incorporating domain-specific pretraining, a novel concept-aware masking incorporating contrastive learning, and an NER tool. The proposed approach provides significant improvement in precision for domain-specific information extraction tasks. We explore different augmentation techniques to enhance the diversity and richness of training data, improving the generalization capabilities of AI models. Consistency loss integration is also explored to improve sentence and token level representations. Thirdly, we integrate domain-specific knowledge into AI models through knowledge-injected learning, enhancing their performance in real-world applications. This involves using structured information and domain-specific context to guide the learning process. We explored multi-task approaches within biomedical NER by integrating categorical label information alongside syntactic contexts, such as part-of-speech (POS) labels. We developed context filtering modules for generative question answering (GQA) systems to mitigate hallucination in generated answers and utilize style tokens to direct the generation style of the answer. Lastly, to address the privacy constraints in biomedical and financial domains, this thesis makes use of federated learning (FL), analyzing the effects of data heterogeneity, label imbalance, and optimization on model performance. Our findings provide insights for setting up FL between collaborating institutions without needing to share the data outside the respective institutions. Overall, this thesis proposes several approaches to leverage the potential of advanced NLP techniques using transfer learning, domain adaptive pretraining, augmentations, and knowledge-injected learning. We present extensive experimental results to validate the proposed approaches in real-world applications with limited annotated data.