Privacy Evaluation of Synthetic Clinical Text

Introduction: This thesis investigates privacy risks in synthetic medical text generation. While synthetic text offers a potential solution to privacy concerns in medical research, preventing unintentional leakage remains a challenge. This study examines whether semantic similarity between synthetic and training text heightens privacy risks. Research Question: The research question is: Does a higher semantic similarity between training text and synthetic text increase the risk of privacy leakage? Method: This research uses a correlational study and a mixed-methods approach to assess privacy risks in synthetic medical text generation. The fine-tuned LLaMA-3.1-8B model is evaluated by comparing its generated text to the de-identified Swedish medical records used in training. Privacy risks are analyzed using lexical (N-gram overlap) and semantic similarity metrics, alongside a qualitative manual review to identify potential privacy breaches. Results: The results show a low 8-gram overlap but high unigram overlap between generated and training text indicating vocabulary reuse but low direct copying. The average semantic similarity between synthetic and training text was moderate (0.57) this suggests contextual alignment. The correlational test revealed a weak but statistically significant (τb =−0.165, p < .001) negative correlation between semantic similarity and unigram overlap. The manual review showed that most synthetic text did not contain personally identifying health information. The review also revealed that synthetically generated text contained fewer quasi-identifiers. Discussion: The correlational test together with the manual review indicates that higher semantic similarity does not correlate with an increased risk of personal health information leaking. The manual review also suggested that text with higher semantic similarity had better coherence and was more informative in the generated text.