Unpacking similarity effects in visual memory search: categorical, semantic, and visual contributions

• Regression analysis was used to predict trial-wise memory search performance. • Visual target-distractor similarity predicted RTs for short- and long-term memory. • Semantic target-distractor similarity predicted RTs for long-term memory. • Results revealed similarities and differences between search in STM and LTM. Visual memory search involves comparing a visual probe item against multiple memorized items. Previous work has shown that distractor probes from a different object category than the objects in the memory set are rejected more quickly than distractor probes from the same category. Because objects belonging to the same superordinate category usually share both visual and semantic features compared with objects of different categories, it is unclear whether the category effects reported in previous studies reflected visual and/or semantic target-distractor similarity. Here, we employed old/new recognition tasks to examine the role of categorical, semantic, and visual similarity in short- and long-term memory search. Participants (N = 64) performed visual long-term memory (LTM) or short-term memory (STM) search tasks involving animate and inanimate objects. Trial-wise RT variability to distractor probes in LTM and STM search was modelled using regression analyses that included predictors capturing categorical target-distractor similarity (same or different category), semantic target-distractor similarity (from a distributional semantic model), and visual target-distractor similarity (from a deep neural network). We found that categorical, semantic, and visual similarity all explained unique variance in trial-wise memory search performance. However, their respective contributions varied with memory set size and task, with STM performance being relatively more strongly influenced by visual and categorical similarity and LTM performance being relatively more strongly influenced by semantic similarity. These results clarify the nature of the representations used in memory search and reveal both similarities and differences between search in STM and LTM.