The Effectiveness of Uncased Tokeniziaion for Clinical Notes

The impact of case-sensitive tokenization on clinical notes is not well understood.While clinical notes share similarities with biomedical text in terminology, they often lack the proper casing found in polished publications.Language models, unlike humans, require a fixed vocabulary and case sensitivity is a tradeoff that must be considered carefully.Improper casing can lead to sub-optimal tokenization and increased sequence length, degrading downstream performance and increasing computational costs.While most recent opendomain encoder language models use uncased tokenization for all tasks, there is no clear trend in biomedical and clinical models.In this work we (1) show that uncased models exceed the performance of cased models on clinical notes, even on traditionally case-sensitive tasks such as named entity recognition and (2) introduce independent case encoding to better balance model performance on case-sensitive and improperly-cased tasks.