Précis of <i>The Border between Seeing and Thinking</i>

Despite the title, this isn't so much a précis as an update about my current thinking on some of the most important issues of the book as they are relevant to the excellent critiques in this issue. (References to page numbers are to my book (Block, 2023a) unless specified otherwise.) The book is mainly about perception, the best understood aspect of the mind, and hence our best window into how the mind works and how it is grounded in the brain (Mandelbaum, 2024). The book starts with armchair indicators of perception, then moves to scientific indicators of perception, adaptation, speed, pop-out and illusory contours, using them to try to pinpoint the most basic characterization of perception and cognition. Of these indicators, the most controversial has been adaptation, mainly because of two issues: whether there can be cognitive adaptation (Helton, 2016) and whether all perceptual systems show it (Block, 2023d; Phillips & Firestone, 2023), The main claim of the book is that perception is constitutively iconic, non-conceptual and non-propositional while cognition is paradigmatically (but not constitutively) discursive (non-iconic), conceptual and propositional. Perception constitutively has these properties in that its nature does not allow for discursive, conceptual or propositional representation. Cognition is necessarily discursive, conceptual and propositional but it also can make use of perceptual materials in a cognitive envelope, as when we use perceptual imagery in the course of reasoning. This is my main approach to the perception/cognition border though I also accept a moderate level of modularity of perception (Mandelbaum, 2024), as explained in Chapter 11. Much of the book was devoted to distinguishing between low-level and high-level perception where low-level perceptual representations are the products of sensory transduction that are causally involved in the production of other (mid and high-level) perceptual representations. I argued that there are high-level perceptual representations and discussed how to distinguish them from cognitive representations, most importantly minimal immediate direct perceptual judgments. The latter notion comes up briefly in my reply to Jake Beck. I argued in Chapter 6 that infant color perception is non-conceptual on the basis of the fact that the infants in question do not normally deploy and probably don't have any concept of color. (My argument has been critiqued in (Green, 2024). I argued that the point generalizes at least to adult color perception. There has been a lot of controversy over claims that perception is discursive, conceptual and propositional (Block, 2023c; Quilty-Dunn et al., 2023). Jake Quilty-Dunn and EJ Green (Green & Quilty-Dunn, 2021; Quilty-Dunn & Green, 2021) argue that object perception—but not perception of features such as color– is discursive, conceptual and propositional, so although they are opposed to my main claims, non-conceptual color perception would be compatible with their view (Block, 2023b). One chapter of the book is devoted to core cognition. It has been argued that core cognition is a third category, intermediate between perception and cognition, but I argued that core cognition is a mongrel notion whose phenomena can be apportioned between perception and cognition, and in that sense core cognition is a heterogeneous mixture. Much of the book was devoted to perception, irrespective of whether the perception is conscious or unconscious. I argued that this perspective has a number of significant advantages in uncovering ideas that can then be applied in arguing against cognitive theories of consciousness. Although I argue against cognitive theories of consciousness in the book, I have recently started taking them more seriously, having been persuaded by the advantages of “pointer” versions of these theories (Block, 2024b; Michel, forthcoming). The concept in the book that is most in need of further explanation is the notion of a cognitive envelope and its relation to perception and working memory. I turn to that topic now. The term ‘cognitive envelope’ is best explained with respect to the “global neuronal workspace”, often offered as a theory of consciousness (Dehaene, 2014) but which I argue is better thought of as a theory of how perceptual representations are conceptualized. The idea of the global neuronal workspace is that activated representations in perceptual areas compete with one another, with a small number of “neural coalitions” winning out that then trigger “ignition”, linking them up with frontoparietal circuits via long range “workspace neurons” that connect the front with the back of the head1 (See pages 7–8 and p. 320 (of the book).) These activated neural coalitions involving both the front and back of the head make the contents of the representations immediately available to a range of cognitive faculties, reasoning, reporting, deciding, and the like, at least in creatures like adult humans that have these cognitive faculties. It is the links to cognitive faculties in the global workspace that constitute the “cognitive envelope”. This is all illustrated in Figure 1 (from early in the book—p.7). The concentric rings represent levels of processing. (see pp. 83–84 on the visual hierarchy.) The outer ring represents the sensory surfaces of the body whereas the inner rings represent frontal-parietal circuits that govern thinking, decision-making, planning and other cognitive functions. Processors are indicated by small disks, links between them by lines—activated processors and links are darker. Activations in the sensory rings compete for dominance, triggering mutually reinforcing reverberating activity linking inner and outer rings. What Figure 1 illustrates is that when sensory surfaces are stimulated, they cause activations of processors, indicated by darker disks. Those activated processors activate other processors via activated links. Some of those activations die out but others form mutually reinforcing coalitions—involving both feedforward and feedback connections– indicated by the large ovals, and some or all of those trigger long range activations in frontal/parietal workspace areas. We can think of global broadcasting in terms of 4 stages: (1) perceptual representations competing (outer dotted rings), (2) some mutually reinforcing coalitions winning (dark ovals), (3) winning coalitions triggering workspace activations (inside the central circle), (4) leading to mutually reinforcing neural activations involving both the front and back of the head. While the details of Dehaene's model are controversial, I think the broad outline of these stages would be widely accepted. Broadcasting in the global workspace is the main mechanism of “working memory”, the active maintenance of representations for use in reasoning. It is that use in reasoning that determines that they are in part discursive, conceptual and propositional. There may however be broadcasting in creatures whose reasoning capacities are substantially iconic. Ants and bees for example have famously prodigious memory capacities (Gallistel & King, 2011) but perhaps little or nothing in the way of propositional reasoning. Some of the reasoning abilities in infants may also be primarily iconic. I defined “working memory” as “active”. Sometimes the term “working memory” is used in a looser sense to refer to any temporarily retained perceptual information that can be used in a subsequent task. Active working memory is conscious but several studies suggest maintenance of information in unconscious perception for up to 4 seconds in what is sometimes termed “activity-silent working memory” (Soto et al., 2011) or a “blindsight effect” (Trübutschek et al., 2017). Subjects could localize a target presented in unconscious perception better than chance 4 seconds after presentation of the stimulus. There is some evidence that this information is maintained in temporarily changed synaptic weights (Trübutschek et al., 2017) but later work suggests that perhaps a low level of activation is involved (Barbosa et al., 2021; Fields, 2022) This form of information persistence is totally different in its fundamental properties from active working memory and it is just confusing to give it the same name (Carruthers, 2015, p. 91). I will ignore it in what follows. As I emphasize in the book, working memory is required for any content-based reasoning in which premises are put together to reason to a conclusion because the premises have to be held in working memory. How early do babies show signs of content-based reasoning? Kiley Hamlin showed 6 month old babies events involving small colored squares, triangles and circles with eyes made of wood. One figure, say the triangle, appeared to be “trying” to get up a slope. Later, another figure played the role of a “helper”, pushing the triangle up the slope while another played the hinderer, pushing the triangle down the slope. When offered the choice of blocks in these shapes, the babies preferred the helper and rejected the hinderer. They distinguished accidental from intentional helping and hindering (Hamlin et al., 2007). Most importantly for content-based reasoning, the infants preferred the hinderer of the hinderer. This was replicated with 8 month old bilinguals (Singh, 2020). 8 month olds were shown two scenarios in which an agent tried but failed to help, or alternatively, to hinder a “protagonist” open a box. Even though the protagonist did open the box, the infants preferred the helper to the hinderer. (Woo et al., 2022) report many results of this sort and they address concerns about replication showing that the main results have been replicated many times even though there have been some notable failures to replicate. My conclusion is that these results provide some reason to believe that the infants have content-based reasoning involving multiple items in working memory, categorized appropriately, put together to draw conclusions. Leaner accounts of this thinking in purely iconic terms are not ruled out however. This issue will come up in my reply to Jake Beck. What I've said so far might suggest that my view of working memory is that it is a hybrid: perceptual representations pretty much unchanged are part of a reverberant (self-sustaining feed forward and back) network with cognitive representations, allowing the perceptual representations to participate in perceptually based reasoning. However, part of this picture is likely wrong. The accepted view of working memory has shifted radically in the last few years. The outmoded “sensory recruitment” view of working memory (D'Esposito & Postle, 2015) assumed it could be understood in terms of purely perceptual representations and so could be studied using stimuli of shapes or colored circles or oriented lines. It was assumed that working memory capacity was fixed in terms of numbers of objects or a fixed pool of resources for each visual feature (Asp et al., 2021). More recent approaches (Chung et al., in press; Duan & Curtis, 2024; Kwak & Curtis, 2022) have treated working memory representations as task specific and as linked to knowledge structures in long term memory. Asp et al. presented ambiguous “Mooney faces” and related figures in paradigms in which the same figure would be seen as a face by some subjects and as a meaningless pattern by others. They found across a number of different methods higher working memory capacity for faces vs the same figures as meaningless patterns. As Chung et al. note, other studies have shown that when stimuli are seen in terms of semantically meaningful categories, working memory capacities are increased. One strong piece of evidence in favor of the sensory recruitment hypothesis was that working memory representations of perception could be decoded using brain imaging from the earliest visual area V1. However (Duan & Curtis, 2024) showed that even for oriented grids, the working memory representations in V1 are reformatted versions of the perceptual representations in V1. They showed this by biasing the stimuli with a sophisticated form of an “aperture bias” which in its simplest form is just looking at the stimuli through a shaped aperture which can have the same or orthogonal orientation of the grid. They used aperature biases that could be the same orientation as the stimuli or orthogonal to the orientation of the stimuli and contrasted a perception task with a working memory task. They found that the aperture biases affected the decoding of the perceptual representations in V1 in a perception task, but not in a comparable working memory task. When they trained a decoder on the perceptual representation of a grating in V1, the same decoder only worked during the working memory representation if the aperture bias was aligned with the grid. They conclude (p.1), “These results provide strong evidence that visual WM representations are abstractions of percepts, immune to perceptual aperture biases…” Working memory is highly resistant to masking of all kinds (Teeuwen et al., 2021). (A mask is a stimulus presented after (or before) a target stimulus that makes the target stimulus harder to see.) But perceptual representations at different stages in the processing stream can be obliterated by the right kinds of masks at different times after the stimulus. Maguire & Howe showed that lines and edges work well as masks when the perception is being processed by early vision, texture masks work well 50 ms later, and shaped objects mask perceptions during shape processing. Because of the need for visual cortex to process ongoing perception, working memory representations are often recoded outside the classic visual cortex (Rademaker et al., 2019) Color representations in perception when moved to working memory are biased away from category boundaries and towards category centers (Bae et al., 2015) Working memory representations for colors are related to color names in a way that perceptual representations of color are not. (Hasantash & Afraz, 2020) showed that although knowing more color names makes for more color representations in working memory, it has no effect on perceptual color matching. Also, subjects who have more color names in a particular area of the color wheel can hold more of those colors in working memory, but they did not do better on those colors in a perceptual matching task. Working memory representations are coarse grained. For example, we can hear many more pitches and see many more colors than we can hold in temporary memory Fundamental computational features of perception are absent in working memory representations. One such computational feature is divisive normalization, a computation governing an aspect of how the perceptions of two items influence one another, including center-surround (See p. and Chapter of is illustrated in Figure where it the of the central on the right where it is with the but not on the where the two are at Figure will see that the central in than the orthogonal central et al. et al., showed subjects central and of or one after the by one They found center-surround when the were presented at the same but not by 1 one to be held in working memory. They (p.1), memory representations a different of computational from normalization, a visual Working memory representations not to the iconic memory representations of perception. showed the classic iconic memory effect by subjects with an of For example, presented of 4 in each of a presentation of the subjects could or 4 even though they said they could see all or most of the showed that they a representation of all or most of the as presented the then after the was an for one of the found that subjects could name or 4 from any This classic iconic memory does not work when the is presented after they moved their eyes to a using a working memory representation of the (See and memory is much more perceptual than working memory, a fact many including that they have different Working memory with to that emphasize a pool of and also when subjects are to more items et al., 2019) showed subjects colored briefly and after a no up to 1 a where there been a colored The task was to pinpoint the color on a color found that as subjects could colors but with no of model the results much better than a model of the sort one would with working memory, a different pattern for the iconic memory and visual term memory in these Kwak and & Curtis, 2022) showed that perceptual representations are in working memory on the task. They showed subjects with a in one and oriented in The task was to the orientation of the or the grid. They the subjects during the that they could not the between the neural representations of these different the perceptual representations more in working memory. The cognitive of the task can in information to the working memory so that is a reason for thinking the process of the perception is in the sense of 2023), These between perception and working memory do not show a in and even if there is a in it could be the category of iconic representation than a of iconic into discursive representation. The perceptual representations of working memory may be or iconic, while in a cognitive envelope by the reverberating circuits involving cognition as in Figure Those cognition circuits will normally propositional reasoning, at least in adult content-based reasoning working memory. representations are held in working memory in a cognitive envelope in which the nature of the representations is more than perceptual representations and in which these representations are to a of cognitive in the global I was of much of this when I the book in but in the last few the evidence has in that even the most basic perceptions like the perception of oriented lines are recoded when used in reasoning. This has my that there are any perceptual representations used in cognition. in representations as to in properties in a way that is to the of For example, as objects are perceptual representations of the in a way that is to the of in representations to the that is in a way that on the of the properties are by representations that of those has a notion of also from not seen book my book was in notion in a of from the main is that that the relation be in a way that is to and my I give what I as a highly example in which in in are by that represent and of a at in the I to think that it can use irrespective of as I that we can an visual that an iconic both in and in this issue the claim that perception is iconic in my that adaptation provide some evidence that there is a perceptual category by and might be for or that are in but what about and that are not. How can there be an iconic representation that is between and suggests some of the results that I as and in need of further The most is that there are between and The mechanism of the might be that one any the representation at both and with the representation between the The details are in (Block, p. to some of the evidence for iconic representation in has long the role of iconic representations in reasoning. One role is the use of some in Jake in this issue. But there is another that has that also is substantially iconic, what the in which from in working memory represent the one of if infants see an put one large piece of in one and small in the other infants to the with more area showing the is being with the What these representations are in the is that if there are 4 or more in one of the the choice to As I for kinds of the working memory has a capacity of about so to chance that is in The has up to for and those can represent in an iconic further example of this is when the are a a and a the having seen those in the box. The is if items that like those are not there on this with 4 items than The and picture iconic representation at the level of the in working memory, but there is also For example, an in which there are two into which a can but what is in the objects have been put in the the to into the whereas if the number of items is 4 or the to chance as just The working memory representations as a the items in the as a in respect of More recent by and their et al., et al., show other For example, an to task in which there are two in one there are items all the in another all are different from one another, with no two the Subjects are one of the and are to from a number of which one the The intermediate all same with all the of all they their on all not the no two the or not all the to the two of the of The and to a of that is their are by all the same no two the same and intermediate the same lines. The point is that is a that properties of the and so iconic. these phenomena involving iconic representation in thinking in and also to the made by Jake in and in that there is an important category of non-conceptual and non-propositional My is that in of the sort just the iconic representations are in a cognitive envelope, so they are not non-conceptual or I do however that I might have the of iconic representation in thought but in many the thought of and babies over when they a form of in a cognitive envelope that a discursive, propositional and conceptual aspect to the Chapter that there is no fact of the as to whether perception is constitutively and then on pages I argued that both perceptual and perceptual are fundamental to perception and there is no fact of the as to whether one is more basic than the This is probably the topic on which my have the I think the conclusion is right but for different than in the book, and the concept of perceptual which I as in the book I see as importantly My of the to and is mainly with the issue than my with et al. and it is the of my than into the issues I to the