Bias in Language Models: Interplay of Architecture and Data?

Pre-trained language models (PLMs), despite showing strong performance, can carry and increase biases, which can limit the development of fair NLP and IR systems. This research investigates the foundational origins of bias within PLMs, moving beyond detection to a detailed analysis of its formation and propagation across diverse architectures. Through a novel attention weight analysis, we reveal distinct attention patterns for biased versus neutral content, offering insights into the internal representations learned by PLMs. Our findings demonstrate a complex interplay between training data and model architecture, revealing that while the transformer's self-attention mechanism amplifies existing biases, the training data plays a crucial role in the initial encoding of bias within the model's representations.