Beyond Standard Metrics: Assessing NLP for Anonymisation of Clinical Trial Narratives

<b>Introduction:<br/></b>The anonymisation of unstructured free text data is important for sharing large amounts of healthcare information generated by electronic health records, publications and clinical trials. To automate this process, natural language processing (NLP), information extraction (IE) and named entity recognition (NER) are essential tools. However, evaluating NLP performance for downstream anonymisation requires understanding the categories of entities involved and their significance for re-identification.<br/><br/><b>Methods:<br/></b>This paper looks in-depth at patient safety narratives from clinical trials from F. Hoffmann-La Roche as a case study of free text data that would undergo anonymisation for public release. The study employs a mixed-methods approach including quantitative text analysis and qualitative content analysis to identify challenges in the use of dictionary-based NLP systems to structure data for downstream anonymisation.<br/><br/><b>Results:<br/></b>The findings show that clinical trial narratives are dense in patient-level clinical information which may be qualified as quasi-identifiers depending on an intruder’s background knowledge. We discuss the proposal of assigning different weights to entities when evaluating NLP performance for anonymisation. The current NLP evaluation metrics don’t translate well to anonymisation tasks as some entities are more crucial to remove than others. Qualitative insights suggest that different therapeutic areas may require different anonymisation approaches due to the nature of the disclosive information. Notably, psychiatric poses significant challenges for dictionary-based NLP due to the complex semantic characteristics of the free-text data.<br/><br/><b>Conclusions:<br/></b>Current NLP evaluation metrics are not well-suited for anonymisation tasks. Certain therapeutic areas, such as psychiatry, push the limits of NLP capabilities more than others. Further research is needed to develop robust mathematical equations for evaluating NLP in the context of anonymisation.