Tracking sources of biased decision-making in medical language models: a case study on gender bias

Though recent advances in medical large language models (LLMs) indicate the great potential of language models (LMs) in healthcare, other studies showed these AI doctors were biases as human doctors did. The current study tracked three sources of bias in LMs: bias in LMs, bias in clinical data, and the interaction between the former two. Through word embedding, we found that, compared to BERT base, Bio+Clinical BERT was gender-biased for certain symptoms, indicating potential bias in its training data. We further examined how biases affected the decision-making of LMs in a downstream task of gender prediction. Leveraging two masking strategies, we found that models were affected not only by biases in model and training data but also by an interaction between the two. Altogether, our results captured how different sources of bias affected the decision-making in language models, informing further understanding on the underlying mechanisms of biased medical LMs.