Investigating the representational transformations underlying the learning of exceptions in visual categories

Categories in the real world often contain perceptually incoherent objects, such as structural exceptions that resemble members of competing categories and oddball exceptions that encompass features distinct from any known categories. Prominent theories suggest that learning exceptions depends on flexible transformations of category representations, yet evidence of such representational dynamics in the brain is limited. Here, we had participants learn competing visual categories that included both structural and oddball exceptions while fMRI data was collected before, in the middle of, and after category learning. Representational similarity analysis of neural patterns for category stimuli at each learning stage revealed that exception learning induced unique transformations in object representations in distinct brain regions. Specifically, the introduction of exceptions led to an increase in feature-specific information in visual cortex representations. In contrast, representations in the prefrontal cortex exhibited an increase in prototype information consistent with coding for category regularities. Notably, subfields of the hippocampal formation also showed distinct transformations—feature-specific information increased in dentate gyrus representations and decreased in CA1 representations. These results align with the dentate gyrus’ theorized role in constructing item-specific representations and CA1’s role in generalization across related experiences. Moreover, we found that exception learning induced distinct representational transformations for structural and oddball exceptions. Particularly, within the representational spaces of the prefrontal and temporal cortices, structural exceptions became uniquely more differentiated from regular category members through learning. This finding aligns with the expectation that learning structural exceptions relies on distinguishing them from perceptually confusable items in the competing category via differentiation. Altogether, our results demonstrate that object representations can be flexibly and selectively transformed across the brain to support the learning of category regularities and their exceptions.