Similarity ratings for basic-level categories from the Nosofsky et al. (2018) database of rock images

The concept of similarity is crucial to our exploration and understanding of cognitive processes.For example, by examining how visual attention is differentially distributed to targets and distractors (varying in similarity of appearance), researchers can gain insight into cognitive resource management, template matching, and object recognition. In such experiments, researchers often couple manipulation of similarity among their stimuli with measures of attentional allocation (Posner, Snyder, & Davidson, 1980) and decision-making (Kahneman & Tversky, 2013). By manipulating the similarity of targets and distractors, or the similarity amongst the set of distractors themselves (Desimone & Duncan, 1995), such research offers insights into the mechanisms of attention, decision-making, and human behavior.Controlling for stimuli in such experiments can be expensive in labor and time, however, resulting in a tendency to rely upon hyper-simplistic experimental designs; for instance, experimenters wishing to employ a task with highly similar targets and distractors may ask their participants to search for a 7 target among many T distractors. Such simplicity is not always preferred (or optimal) however, especially in research necessitating more complexity, such as tasks that use categorically defined targets (Hout et al., 2017;Madrid, et. al., 2019;Robbins & Hout, 2020), searches set in visually complex environments (Hout, et. al., 2021(Hout, et. al., , 2023)), or dynamic search tasks that are challenging to hyper-simplify (Scarince and Hout, 2018). Questions such as these often require complex stimuli, but when similarity must be manipulated in an experiment, it can be challenging to know how best to do so precisely. One may, for instance, reasonably assume that a 7 is more similar to the letter T than it is to the letter R; such an assumption is uncontroversial and may suffice if the researcher simply requires pairs of stimuli that are or are not alike. If a researcher is interested in categorical visual search, however -how people look for a class of items (e.g., "search for any coffee mug") rather than a particular exemplar -alphanumeric digits (or other simplistic stimuli) typically do not suffice. Moreover, researchers often require more than a simple dichotomy of similar vs. not similar, and instead need to be able to explore more nuance in their data by obtaining a continuum of similarity among the stimuli (one that can be quantified but is therefore much more subject to individual differences in perception).When scaling up to pictures of real-world objects, simple assumptions -regarding what objects are more or less similar to one another -are often not sufficient, straightforward, or uncontroversial.Rating feature rich, complex objects for similarity requires a significant empirical investment unto itself, but doing so affords the researcher experimental tools and precision of manipulation that is otherwise impossible to achieve. The suboptimal alternative, of course, is exclusive reliance on hyper-simplistic stimuli for which assumptions about similarity perception are deemed sufficient. To provide a concrete example : Hout, Goldinger, & Brady (2014) established a similarity database of approximately 4000 visual pictures of real-world items sampled from across 240 distinct categories; the images were sourced from the "Massive Memory Database" (Brady et al., 2008;Konkle et al., 2010;cvcl.mit.edu/MM/stimuli.html). After collecting similarity ratings, they employed multidimensional scaling (c.f., Hout, Papesh, & Goldinger, 2013) to model similarity relationships among the items, providing future researchers with continuous similarity relationships for 16-17 exemplars from each category. Since its initial publication, the similarity ratings from this database have been used to explore various cognitive processes such as attention, template formation, and memory. For instance, Hout and Goldinger (2015) systematically manipulated the similarity between a visual search cue (e.g., "look for this picture or something like it") and the subsequent item that appeared in the array. Similarity scores from the database (Hout, Goldinger, & Brady, 2014) predicted how efficiently eye movements could be directed to the target, and how quickly the target would be recognized once it was fixated.Guevara Pinto et al. (2020) gave participants a visual search task followed by a surprise alternative forced-choice task in which they were asked to select which of 16 category-matched exemplars was presented to them earlier in the experiment. The authors found that when the correct item was not chosen, similarity scores (i.e., the similarity of the incorrectly chosen "foil" relative to the item that had actually been shown previously) predicted the likelihood that an item would be selected. Simply, when subjects made a mistake, they were systematically more likely to choose visually similar foils (and the strength of this relationship was found to be modulated by the number of items that were being looked for).Because such similarity databases are so useful (but are also relatively rare) our goal in this project was to expand upon the natural category stimuli set created by Nosofsky et al. (2018; see also Meagher et al., 2017), by adding to it a set of similarity scores -isolated to each of the basic-level stimuli categories separately, as was done in Hout, Goldinger, & Brady (2014) -that can be used in cognitive experiments exploring attention, memory, categorization, and more.In Nosofsky et al. (2018), 360 rock images were collected for the purpose of studying the learning of complex natural science categories. These stimuli consist of high-resolution images, with each exemplar categorized as belonging to one of the three superordinate categories (based on the processes governing its formation): metamorphic, sedimentary, and igneous. Each superordinate category comprised 10 nested basic-level categories (e.g., marble, granite, obsidian), and each basic-level category contained 12 exemplars (or "tokens"), varying in similarity of appearance, but standardized in form (i.e., all were images of rocks isolated on a blank white background, uniform in overall size).In one experiment, Nosofsky et al. (2018) selected 30 representative tokens (one from each basiclevel category), and had participants rate the similarity of each pair (see also Nosofsky et al., 2017). In a second study, the researchers had participants rate the similarity between each of the 360 instances (from all categories), providing a larger matrix of similarity data, but one that was more sparsely populated (i.e., with fewer similarity ratings per pair of stimuli, given that there are 64,620 unique pairs in a 360x360 matrix). What is shared by both studies is the fact that exemplars were compared to one another across categories. In the second study, exemplars from within the same category were also compared to one another, but only in the context of the entire set of 360 tokens sampled from across 30 categories.Context effects -the tendency of similarity ratings to be determined not just by perceived similarity of a pair but to also be influenced by the participants' understanding of the entire set of stimuli (see Tversky, 1977;Goldstone, Medin, & Halberstadt, 1997) -might therefore may have resulted in items from within a single category being perceived as more similar to one another than they would be if that single category was being considered in isolation.It is inarguable that the data collected by Nosofsky et al. (2018) is useful for studying natural category learning (see Nosofsky et al., 2022, for example, for an application of this dataset to prototype and exemplar models of category formation). What is lacking, however, are complete similarity matrices for each basic-level category collected in a method that is free of potential context effects that might "compress" the perceived similarity of items within a single category. To add to the utility of this dataset, we had participants rate the similarity of all exemplars in a given basic-level category in isolation, allowing them to appreciate the nuance of the perceptual details contained within a category without being affected by the context of the greater set of categories/tokens.Frequently, similarity ratings are obtained via the pairwise method of comparison (Thurstone, 2017), whereby participants are tasked with numerically rating item pairs via Likert Scales (Likert, 1932), progressing through every possible pairing in a stimulus set. While the pairwise method has the advantage of task simplicity, a central drawback is that it burdens researchers (and participants) with a costly expense in time and data collection effort. This is especially the case when used for rating large stimulus sets, as the number of needed comparisons grows dramatically with every stimulus added to the set.Because of the length of time required to perform the ratings, researchers must be wary of both rater fatigue and criterion drift, generating concern over data reliability (Johnson, Lehmann, & Horne, 1990).By contrast, the Spatial Arrangement Method (SpAM) (Goldstone, 1994) presents participantswith multiple stimuli at once, tasking the rater with arranging them in space relative to the perceived similarity between each pair. Items that are of high perceived similarity are to be placed proportionally closer together than items considered dissimilar; the data matrix derived from this task simply includes recording Euclidean distances between each pair of items at the end of the arrangement. Whereas the rating of a 12-token set might require 5-10 minutes for a participant to rate all 66 unique pairs, a single SpAM trial could present all the items at once, and might require only 2-3 minutes of arrangement to obtain complete ratings. The SpAM task is intuitive, it takes advantage of the natural tendency to think of similarity in a spatial manner, and it is much faster and more efficient than pairwise methods. Indeed, while not right for every experimental scenario (see Verheyen et al., 2016), it has been shown that this speed and efficiency does not come at the cost of data quality for perceptual (Hout, Goldinger, & Ferguson, 2013;Hout & Goldinger, 2016) or conceptual (Richie et al., 2020) similarity rating tasks.Regardless of the similarity rating method used, obtained data are then most frequently subjected to multidimensional scaling (MDS) analysis, a statistical procedure used to model the similarity relationships among items in a set (see Hout, Papesh, &Goldinger, 2013 andHout et al., 2016, for tutorials). The goal of subjecting this "proximity data" (i.e., similarity ratings obtained via pairwise method, SpAM, or some other technique) to MDS is to produce a visually accessible interpretation of the similarity relationships present in the data (which can be useful in hypothesizing as to what features of the stimuli governed participant ratings), and to provide a comprehensive set of "distances" in MDS space that quantify the degree to which any given pair of stimuli are perceived as similar or dissimilar (relative to the other items/pairs in the set).In this project, we collected similarity ratings (via SpAM) for each of the 30 basic-level categories (separately) from the Nosofsky et al. (2018) dataset. Participants rated the similarity of all 12 exemplars in a category on each trial, and data were thereafter subjected to MDS analysis. This data report presents those data and MDS models; full results can be found on our OSF page. It is important to note that this database is intended merely as a supplement to Nosofsky et al. (2018). Specifically, this dataset is intended to focus on the discriminability and similarity of items within each category, which can vary significantly depending on the heterogeneity of the set (e.g., members of the Quartz category are highly similar to one another, whereas those in the Granite category vary more in appearance). By collecting similarity ratings within a single category at a time, within-category similarity can be better appreciated, without being "compressed" via comparison to members of other categories (e.g., Quartz items might clump together more than Granite in the context of all other categories present in Nosofsky et al., 2018).The cost of allowing for more nuance in within-category ratings, however, is that we are unable to provide across-category ratings for the stimuli; as have already been provided by Nosofsky et al. (2018). As such, this dataset is likely more useful for studies of visual attention (that necessitate withincategory similarity manipulations as in Hout & Goldinger, 2015) than it is for category-learning paradigms that require similarity comparisons across categories. Moreover, our approach does not allow for comparisons across categories (e.g., determination of the relative discriminability of Quartz items relative to Granite tokens), and might influence of the impact of "global" features that are common across many categories (e.g., light vs dark was shown to be a feature of high importance in the Nosofsky et al., 2018, dataset, but for universally dark categories like Obsidian, isolated similarity ratings are unlikely to reveal that featural dimension). Instead, the value of our dataset is that it provides relatively unperturbed similarity ratings within each category, allowing for subtle differences between tokens to be more impactful when not influenced by the features of other categories.This research was approved by the Institutional Review Board of New Mexico State University (on 7/2/22, IRB Protocol #2206002241), and was considered ''exempt'' in accordance with Federal regulations, 45 CFR Part 46. Upon arrival, informed consent detailing the aim and scope of the research (as well as the right to refuse or withdraw at any time without consequence) was obtained from all participants in writing. Participants were then instructed how to perform the task and were given an opportunity to ask questions before they began.Thirty-five participants were recruited from the student population at New Mexico State University. Students were compensated with participation credits that could be used as extra credit or as partial fulfillment of course requirements. All participants had normal or corrected-to-normal vision, and all reported normal color vision.All participants completed an experiment where they were given thirty SpAM trials (with categories selected in random order) containing 12 stimuli each, taking an average of 50 minutes to complete. All participants saw and arranged all items within each of the basic-level categories for similarity. In this manner, each item was scaled for similarity against every other item within their respective categories (e.g., all granite exemplars were rated against all other granite exemplars).The stimuli used were from the Nosofsky et al. ( 2018) study. There were a total of 360 stimuli, sampled from across 3 superordinate categories (metamorphic, sedimentary, igneous), and belonging each to one of 30 basic-level categories (e.g., marble). Each basic level category contained 12 exemplars; all pictures were standardized in format, with a single picture presented across a blank white background, and uniform in size.Data were collected on five computers simultaneously, all with identical hardware and software, and supervised by research assistants. The computers used were Dell Inspiron 14 5410 2-in-1 computers with Intel i7 processors (Dell, 2022) running Windows 10 (Microsoft Corporation, 2022) and highresolution ASUS VP28UQG LCD monitors operating on a 60 GHz refresh rate. The experiment was run on E-Prime v3.0 Professional Software (Schneider, Eschman, & Zuccolotto, 2002).In each trial, a new selection of images was shown to the participant on a computer screen (see Figure 1a). The screen was divided vertically by a black border into three distinct areas: left, right, and a central "arena." The left and right areas on the periphery of the screen were each populated by a row of rock images from top to bottom, placed in random order. Participants were asked to "drag and drop" each item from the periphery into the center arena and arrange them by perceived similarity such that items that were perceived of as similar would be placed proportionately closer together than items perceived as more dissimilar. Samples were not provided to participants because we did not want to bias the participants' perception of the stimuli, and our many prior experiments suggest that users find this interface intuitive and simple. That said, participants were also afforded the opportunity to ask research assistants any questions they had prior to beginning the task in order to ensure that they understood how to utilize the space. Participants were given as much time on the task as they needed but were not able to advance to the next trial until all items had been removed from the periphery and arranged in the arena (see Figure 1b). Typically, these trials lasted 2-5 minutes. Upon completion, participants were instructed to click on a Stop Sign icon at the bottom of the screen. To avoid accidental termination of trials, participants were then asked if they were satisfied with their arrangement of the stimuli or if they'd like more time. If they indicated satisfaction with a key press, they were then advanced to the next trial (if not, they were returned to their ongoing trial). The experiment continued in this way through to completion, with participants being allowed to take breaks as needed. Similarity scores were recorded at the end of each trial; we calculated the two-dimensional Euclidean distance between each pair of items at the end of the arrangement and used that information to populate each participants' similarity matrix (for each category). Upon completion, all participants were debriefed on the experimental aims and released. AnalysisAll data in the can be found in the The a with key a containing the the to the analysis, a containing all data for the in format, and a containing all from the in that were used to the each of the 30 rock we used multidimensional scaling (MDS) to a of MDS with was used to a for each within each of the categories. The MDS was in a & Hout, that takes the similarity data collected via SpAM as and the data through the MDS at a of through with the the value of as the to be each of the the a a of measures of and a for each item in the for the for the MDS is to the in as the of the is Such data can be in a approach to the the of an in the often value as a of This a at which an in a in the of & for the use of to more an between more and the results a of is provided for each rock in each of the MDS (see Figure a of in the context of the set of images being where the more central an object is within the the more similar it is to a To we calculated the center of of the space of objects for each MDS by each of the across all this approach to the center of the MDS space to the for in the context of data because it objects in the of the space. the MDS space of rocks where there is a between (i.e., and white and the rocks in the stimuli set. The rocks the and while the center of this space would between all objects, this would not any to the relatively of stimuli on the right of the of the however, does take into the relative within the space. In this example, the would to the right of the space than the resulting in the rocks being more central than the to the a of & Hout, for each object in each MDS (see Figure This a for this stimulus set that offers new by which the images can be is a that is of than each rock to the of the instead each rock to every other rock in the space. To this each rock in the space -the pairwise distance between a rock and every other rock in the space is most unique rock in the space be the one with the pairwise distance between it and any other rock in the space. The unique rock be the rock with the average pairwise distance in the total set of pairwise distances in the space. the it is possible to see the of the center of the as the for the on the right of space. The more the as being less for this particular space of The most rock in the space -the rock with the average pairwise distance to all other rocks be in to the unique rock which has the average pairwise the most central and most rocks are on their distance from the