Evidence of what exactly? Questioning the use of neural networks to create evidence within research narratives

In this paper, we aim to disentangle the dataset lens from the model architecture lens. We gather three multimodal datasets of the Japanese prints and paintings collections from the Tokyo National Museum, the Victoria and Albert Museum, and the Rijksmuseum. These datasets – similar but created with different implicit cultural frameworks – are used to highlight the ways in which inherent cultural biases are exaggerated and warped by different model architectures. We explore what can or cannot be projected in the model’s internal representation of the data (latent space), highlighting what is amplified, and what is underrepresented. We unpick the internal representations of different models to highlight how the relationships between data change depending on choices made during training and visualisation and how this consequently shapes our understanding of the data.