MLM4SynMed: Masked Language Modelling for Synthetic Free-text Medical Records Generation

In this paper, we systematically investigate the possibility, quality, and potential usage of generating synthetic free-text medical records, such as discharge summaries, admission notes, and doctor correspondences, using the Masked Language Modelling (MLM) strategy. Our system is designed to preserve the critical information of the records while introducing significant diversity and minimising re-identification risk. The system incorporates a de-identification component that uses Philter to mask Protected Health Information (PHI), followed by a Medical Entity Recognition (NER) model to retain key medical information. We explore various masking ratios and mask-filling techniques to balance the trade-off between diversity and fidelity in the synthetic outputs without affecting overall readability. Our results demonstrate that the system can produce high-quality synthetic data with significant diversity while achieving a HIPAA-compliant PHI recall rate of 0.96 and a low re-identification risk of 0.035. Furthermore, downstream evaluations using an NER task reveal that the synthetic data can be effectively used to train models with performance comparable to those trained on real data. The flexibility of the system allows it to be adapted for specific use cases, making it a valuable tool for privacy-preserving data generation in medical research and healthcare applications. We host our models and data publicly at https://github.com/HECTA-UoM/MLM4SynMed