Cognitive Psychology and Cognitive Neuroscience/Neuroscience of Text Comprehension

Introduction

What is happening inside my head when I listen to a sentence? How do I process written words? This chapter will take a closer look on brain processes concerned with language comprehension. Dealing with natural language understanding, we distinguish between the neuroscientific and the psycholinguistic approach. As text understanding spreads through the broad field of cognitive psychology, linguistics, and neurosciences, our main focus will lay on the intersection of two latter, which is known as neurolinguistics.

Different brain areas need to be examined in order to find out how words and sentences are being processed. For long time scientist were restricted to draw conclusions from certain brain lesions to the functions of corresponding brain areas. During the last 40 years techniques for brain imaging and ERP-measurement have been established which allow for a more accurate identification of brain parts involved in language processing.

Scientific studies on these phenomena are generally divided into research on auditory and visual language comprehension; we will discuss both. Not to forget is that it is not enough to examine English: To understand language processing in general, we have to look at non-Indo-European and other language systems like sign language. But first of all we will be concerned with a rough localization of language in the brain.

Lateralization of language

Although functional lateralization studies and analyses find individual differences in personality or cognitive style don't favor one hemisphere or the other, some brain functions occur in one or the other side of the brain. Language tends to be on the left and attention on the right (Nielson, Zielinski, Ferguson, Lainhart & Anderson, 2013). There is a lot of evidence that each brain hemisphere has its own distinct functions in language comprehension. Most often, the right hemisphere is referred to as the non-dominant hemisphere and the left is seen as the dominant hemisphere. This distinction is called lateralization (from the Latin word lateral, meaning sidewise) and reason for it first was raised by experiments with split-brain patients. Following a top-down approach we will then discuss the right hemisphere which might have the mayor role in higher level comprehension, but is not quite well understood. Much research has been done on the left hemisphere and we will discuss why it might be dominant before the following sections discuss the fairly well understood fundamental processing of language in this hemisphere of the brain.

Functional asymmetry

Anatomical differences between left and right hemisphere

Initially we will consider the most apparent part of a differentiation between left and right hemisphere: Their differences in shape and structure. As visible to the naked eye there exists a clear asymmetry between the two halves of the human brain: The right hemisphere typically has a bigger, wider and farther extended frontal region than the left hemisphere, whereas the left hemisphere is bigger, wider and extends farther in it’s occipital region (M. T. Banich,"Neuropsychology", ch.3, pg.92). Significantly larger on the left side in most human brains is a certain part of the temporal lobe’s surface, which is called the planum temporale. It is localized near Wernicke’s area and other auditory association areas, wherefore we can already speculate that the left hemisphere might be stronger involved in processes of language and speech treatment.

In fact such a left laterality of language functions is evident in 97% of the population (D. Purves, "Neuroscience", ch.26, pg.649). But actually the percentage of human brains, in which a "left-dominance" of the planum temporale is traceable, is only 67% (D. Purves, "Neuroscience", ch.26, pg.648). Which other factors play aunsolved yet.

Evidence for functional asymmetry from "split brain" patients

In hard cases of epilepsy a rarely performed but popular surgical method to reduce the frequency of epileptic seizures is the so-called corpus callosotomy. Here a radical cut through the connecting "communication bridge" between right and left hemisphere, the corpus callosum, is done; the result is a "split-brain". For patients whose corpus callosum is cut, the risk of accidental physical injury is mitigated, but the side-effect is striking: Due to this eradicative transection of left and right half of the brain these two are not longer able to communicate adequately. This situation provides the opportunity to study differentiation of functionality between the hemispheres. First experiments with split-brain patients were performed by Roger Sperry and his colleagues at the California Institute of Technology in 1960 and 1970 (D. Purves, "Neuroscience", ch.26, pg.646). They lead researchers to sweeping conclusions about laterality of speech and the organization of the human brain in general.

A digression on the laterality of the visual system

A visual stimulus, located within the left visual field, projects onto the nasal (inner) part of the left eye’s retina and onto the temporal (outer) part of the right eye’s retina. Images on the temporal retinal region are processed in the visual cortex of the same side of the brain (ipsilateral), whereas nasal retinal information is mapped onto the opposite half of the brain (contralateral).

The stimulus within the left visual field will completely arrive in the right visual cortex to be processed and worked up. In "healthy" brains this information furthermore attains the left hemisphere via the corpus callosum and can be integrated there. In split-brain patients this current of signals is interrupted; the stimulus remains "invisible" for the left hemisphere.

The experiment we consider now is based on the laterality of the visual system: What is seen in the left half of the visual field will be processed in the right hemisphere and vice versa. Aware of this principle a test operator presents the picture of an object to one half of the visual field while the participant is instructed to name the seen object, and to blindly pick it out of an amount of concrete objects with the contralateral hand.

It can be shown that a picture, for example the drawing of a die, which has only been presented to the left hemisphere, can be named by the participant ("I saw a die"), but is not selectable with the right hand (no idea which object to choose from the table). Contrarily the participant is unable to name the die, if it was recognized in the right hemisphere, but easily picks it out of the heap of objects on the table with the help of the left hand.

These outcomes are clear evidence of the human brain’s functional asymmetry. The left hemisphere seems to dominate functions of speech and language processing, but is unable to handle spatial tasks like vision-independent object recognition. The right hemisphere seems to dominate spatial functions, but is unable to process words and meaning independently. In a second experiment evidence arose that a split-brain patient can only follow a written command (like "get up now!"), if it is presented to the left hemisphere. The right hemisphere can only "understand" pictorial instructions.

The following table (D. Purves, "Neuroscience", ch.26, pg.647) gives a rough distinction of functions:

Left Hemisphere	Right Hemisphere
analysis of right visual field language processing writing speech	analysis of left visual field spatial tasks visuospatial tasks object and face recognition

First it is important to keep in mind that these distinctions comprise only functional dominances, no exclusive competences. In cases of unilateral brain damage, often one half of the brain takes over tasks of the other one. Furthermore it should be mentioned that this experiment works only for stimuli presented for less than a second. This is because not only the corpus callosum, but as well some subcortical commissures serve for interhemispheric transfer. In general both can simultaneously contribute to performance, since they use complement roles in processing.

A digression on handedness

An important issue, when exploring the different brain organization, is handedness, which is the tendency to use the left or the right hand to perform activities. Throughout history, left-handers, which only comprise about 10% of the population, have often been considered being something abnormal. They were said to be evil, stubborn, defiant and were, even until the mid 20th century, forced to write with their right hand.

One most commonly accepted idea, as to how handedness affects the hemispheres, is the brain hemisphere division of labour. Since both speaking and handiwork require fine motor skills, the presumption here is that it would be more efficient to have one brain hemisphere do both, rather than having it divided up. Since in most people, the left side of the brain controls speaking, right-handedness predominates. The theory also predicts that left-handed people have a reversed brain division of labour.

In right handers, verbal processing is mostly done in the left hemisphere, whereas visuospatial processing is mostly done in the opposite hemisphere. Therefore, 95% of speech output is controlled by the left brain hemisphere, whereas only 5% of individuals control speech output in their right hemisphere. Left-handers, on the other hand, have a heterogeneous brain organization. Their brain hemisphere is either organized in the same way as right handers, the opposite way, or even such that both hemispheres are used for verbal processing. But usually, in 70% of the cases, speech is controlled by the left-hemisphere, 15% by the right and 15% by either hemisphere. When the average is taken across all types of left-handedness, it appears that left-handers are less lateralized.

After, for example, damage occurs to the left hemisphere, it follows that there is a visuospatial deficit, which is usually more severe in left-handers than in right-handers. Dissimilarities may derive, in part, from differences in brain morphology, which concludes from asymmetries in the planum temporale. Still, it can be assumed that left-handers have less division of labour between their two hemispheres than right-handers do and are more likely to lack neuroanatomical asymmetries.

There have been many theories as to find out why people are left-handed and what its consequences may be. Some people say that left-handers have a shorter life span or higher accident rates or autoimmune disorders. According to the theory of Geschwind and Galaburda, there is a relation to sex hormones, the immune system, and profiles of cognitive abilities that determine, whether a person is left-handed or not. Concludingly, many genetic models have been proposed, yet the causes and consequences still remain a mystery (M.T.Banich, "Neuropsychology", ch.3, pg. 119).

The right hemisphere

The role of the right hemisphere in text comprehension

The experiments with "split-brain" patients and evidence that will be discussed soon suggest that the right hemisphere is usually not (but in some cases, e.g. 15% of left handed people) dominant in language comprehension. What is most often ascribed to the right hemisphere is cognitive functioning. When damage is done to this part of the brain or when temporal regions of the right hemisphere are removed, this can lead to cognitive-communication problems, such as impaired memory, attention problems, and poor reasoning (L. Cherney, 2001). Investigations lead to the conclusion that the right hemisphere processes information in a gestalt and holistic fashion, with a special emphasis on spatial relationships. Here, an advantage arises for differentiating two distinct faces because it examines things in a global manner and it also reacts to lower spatial, and also auditory, frequency. The former point can be undermined with the fact that the right hemisphere is capable of reading most concrete words and can make simple grammatical comparisons (M. T. Banich,“Neuropsychology“, ch.3, pg.97). But in order to function in such a way, there must be some sort of communication between the brain halves.

Prosody - the sound envelope around words

Consider how different the simple statement "She did it again" could be interpreted in the following context taken from Banich: LYNN: Alice is way into this mountain-biking thing. After breaking her arm, you'd think she'd be a little more cautious. But then yesterday, she went out and rode Captain Jack's. That trail is gnarly - narrow with lots of tree roots and rocks. And last night, I heard that she took a bad tumble on her way down. SARA: She did it again Does Sara say that with rising pitch or emphatically and with falling intonation? In the first case she would ask whether Alice has injured herself again. In the other case she asserts something she knows or imagines: That Alice managed to hurt herself a second time. Obviously the sound envelope around words - prosody - does matter.

Reason to belief that recognition of prosodic patterns appears in the right hemisphere arises when you take into account patients that have damage to an anterior region of the right hemisphere. They suffer from aprosodic speech, that is, their utterances are all at the same pitch. They might sound like a robot from the 80ties. There is another phenomena appearing from brain damage: dysprosodic speech. In that case the patient speaks with disordered intonation. This is not due to a right hemisphere lesion, but arises when damage to the left hemisphere is suffered. The explanation is that the left hemisphere gives ill-timed prosodic cues to the right hemisphere, thus proper intonation is affected.

Beyond words: Inference from a neurological point of view

On the word level, the current studies are mostly consistent with each other and with findings from brain lesion studies. But when it comes to the more complex understanding of whole sentences, texts and storylines, the findings are split. According to E. C. Ferstl’s review “The Neuroanatomy of Text Comprehension. What’s the story so far?” (2004), there is evidence for and against right hemisphere regions playing the key role in pragmatics and text comprehension. On the current state of knowledge, we cannot exactly say how and where cognitive functions like building situation models and inferencing work together with “pure” language processes.

As this chapter is concerned with the neurology of language, it should be remarked that patients with right hemisphere damage have difficulties with inferencing. Take into account the following sentence:

With mosquitoes, gnats, and grasshoppers flying all about, she came across a small black bug that was being used to eavesdrop on her conversation.

You might have to reinterpret the sentence until you realize that "small black bug" does not refer to an animal but rather to a spy device. People with damage in the right hemisphere have problems to do so. They have difficulty to follow the thread of a story and to make inferences about what has been said. Furthermore they have a hard time understanding non-literal aspects of sentences like metaphors, so they might be really horrified when they hear that someone was "Crying her eyes out".

The reader is referred to the next chapter for a detailed discussion of Situation Models

The left hemisphere

Further evidence for left hemisphere dominance: The Wada technique

Before concerning concrete functionality of the left hemisphere, further evidence for the dominance of the left hemisphere is provided. Of relevance is the so-called Wada technique, allowing testing which hemisphere is responsible for speech output and usually being used in epilepsy patients during surgery. It is not a brain imaging technique, but simulates a brain lesion. One of the hemispheres is anesthetized by injecting a barbiturate (sodium amobarbital) in one of the patient’s carotid arteries. Then he is asked to name a number of items on cards. When he is not able to do that, despite the fact that he could do it an hour earlier, the concerned hemisphere is said to be the one responsible for speech output. This test must be done twice, for there is a chance that the patient produces speech bilaterally. The probability for that is not very high, in fact, according to Rasmussen & Milner 1997a (as referred to in Banich, p. 293) it occurs only in 15% of the left-handers and none of the right-handers. (It is still unclear where these differences in left-handers’ brains come from.)

That means that in most people, only one hemisphere “produces” speech output – and in 96% of right-handers and 70% of left-handers, it is the left one. The findings of the brain lesion studies about asymmetry were confirmed here: Normally (in healthy right-handers), the left hemisphere controls speech output.

Explanations of left hemisphere dominance

Two theories why the left hemisphere might have special language capacities are still discussed. The first states that dominance of the left hemisphere is due to a specialization for precise temporal control of oral and manual articulators. Here the main argument is that gestures related to a story line are most often made with the right and therefore by the left hemisphere controlled hand whilst other hand movements appear equally often with both hands. The other theory says that the left hemisphere is dominant because it is specialized for linguistic processing and is due to a single patient - a speaker of American Sign Language with a left hemisphere lesion. He could neither produce nor comprehend ASL, but could still communicate by using gestures in non-linguistic domains.

How innate is the organisational structure of the brain?

Not only cases of left-handers but also brain imaging techniques have shown examples of bilateral language processing: According to ERP studies (by Bellugi et al. 1994 and Neville et al. 1993 as cited in E. Dabrowska, "Language, Mind an Brain" 2004, p. 57), people with the Williams’ syndrome (WS) also have no dominant hemisphere for language. WS patients have a lot of physical and mental disorders, but show, compared to their other (poor) cognitive abilities, very good linguistic skills. And these skills do not rely on one dominant hemisphere, but both of them contribute equally. So, whilst the majority of the population has a dominant left hemisphere for language processing there are a variety of exceptions to that dominance. That there are different “organisation possibilities” in individual brains Dabrowska (p. 57) suggests that the organisational structure in the brain could be less innate and fixed as it is commonly thought.

Auditory Language Processing

This section will explain where and how language is processed. To avoid intersections with visual processes we will firstly concentrate on spoken language. Scientists have developed three approaches of conceiving information about this issue. The first two approaches are based upon brain lesions, namely aphasia, whereas the recent approach relies on results of on modern brain-image techniques.

Neurological Perspective

The Neurological Perspective describes which pathways language follows in order to be comprehended. Scientists revealed that there are concrete areas inside the brain where concrete tasks of language processing are taking place. The most known areas are the Broca and the Wernicke Area.

Broca’s aphasia

One of the most well-known aphasias is Broca’s aphasia that causes patients to be unable to speak fluently. Moreover they have a great difficulty producing words. Comprehension, however, is relatively intact in those patients. Because these symptoms do not result from motoric problems of the vocal musculature, a region in the brain that is responsible for linguistic output must be lesioned. Broca discovered that the brain region causing fluent speech is responsible for linguistic output, must be located ventrally in the frontal lobe, anterior to the motor strip. Recent research suggested that Broca`s aphasia results also from subcortical tissue and white matter and not only cortical tissue.

Example of spontaneous Speech - Task: What do you see on this picture?
„O, yea. Det‘s a boy an‘ girl... an‘ ... a ... car ... house... light po‘ (pole). Dog an‘ a ... boat. ‚N det‘s a ... mm ... a ... coffee, an‘ reading. Det‘s a ... mm ... a ... det‘s a boy ... fishin‘.“ (Adapted from „Principles of Neuroscience“ 4th edition, 2000, p 1178)

Wernicke‘s aphasia

Another very famous aphasia, known as Wernicke`s aphasia, causes opposite syndromes. Patients suffering from Wernicke`s aphasia usually speak very fluently, words are pronounced correctly, but they are combined senselessly – “word salad” is the way it is most often described. Understanding what patients of Wernicke`s aphasia say is especially difficult, because they use paraphasias (substitution of a word in verbal paraphasia, of word with similar meaning in semantic paraphasia, and of a phoneme in phonemic paraphasia) and neologisms. With Wernicke`s aphasia the comprehension of simple sentences is a very difficult task. Moreover their ability to process auditory language input and also written language is impaired. With some knowledge about the brainstructure and their tasks one is able to conclude that the area that causes Wernicke`s aphasia, is situated at the joint of temporal, parietal and occipital regions, near Heschl`s gyrus (primary auditory area), because all the areas receiving and interpreting sensory information (posterior cortex), and those connecting the sensory information to meaning (parietal lobe) are likely to be involved.

Example of spontaneous Speech - Task: What do you see on this picture?

„Ah, yes, it‘s ah ... several things. It‘s a girl ... uncurl ... on a boat. A dog ... ‘S is another dog ... uh-oh ... long‘s ... on a boat. The lady, it‘s a young lady. An‘ a man a They were eatin‘. ‘S be place there. This ... a tree! A boat. No, this is a ... It‘s a house. Over in here ... a cake. An‘ it‘s, it‘s a lot of water. Ah, all right. I think I mentioned about that boat. I noticed a boat being there. I did mention that before ... Several things down, different things down ... a bat ... a cake ... you have a ...“ (adapted from „Principles of Neuroscience“ 4th edition, 2000, p 1178)

Conduction aphasia

Wernicke supposed that an aphasia between Broca‘s area and Wernicke‘s area, namely conduction aphasia, would lead to severe problems to repeat just heard sentences rather than having problems with the comprehension and production of speech. Indeed patients suffering from this kind of aphasia show an inability to reproduce sentences since they often make phonemic paraphasias, may substitute or leave out words, or might say nothing. Investigations determined that the "connection cable", namely the arcuate fasciculus between Wernicke‘s and Broca‘s area is almost invariably damaged in case of a conduction aphasia. That is why conduction aphasia is also regarded as a disconnection syndrome (the behavioural dysfunction because of a damage to the connection of two connected brain regions).

Example of the repetition of the sentence „The pastry-cook was elated“:
„The baker-er was /vaskerin/ ... uh ...“ (adapted from „Principles of Neuroscience“ 4th edition, 2000, p 1178)

Transcortical motor aphasia and global aphasia

Transcortical motor aphasia, another brain lesion caused by a connection disruption, is very similar to Broca`s aphasia, with the difference that the ability to repeat is kept. In fact people with a transcortical motor aphasia often suffer from echolalia, the need to repeat what they just heard. Usually patients` brain is damaged outside Broca`s area, sometimes more anterior and sometimes more superior. Individuals with transcortical sensory aphasia have similar symptoms as those suffering from Wernicke`s aphasia, except that they show signs of echolalia. Lesions in great parts of the left hemisphere lead to global aphasia, and thus to an inability of both comprehending and producing language, because not only Broca`s or Wenicke`s area is damaged. (Barnich, 1997, pp. 276–282)

Type of Aphasia	Spontaneous Speech	Paraphasia	Comprehension	Repetition	Naming
Broca`s Wernicke`s Conduction Transcortical motor Transcortical sensory Global	Nonfluent Fluent Fluent Nonfluent Fluent Nonfluent	Uncommon Common (verbal) Common (literal) Uncommon Common Variable	Good Poor Good Good Poor Poor	Poor Poor Poor Good (echolalia) Good (echolalia) Poor	Poor Poor Poor Poor Poor Poor

Overview of the effects of aphasia from the neurological perspective

(Adapted from Benson, 1985,p. 32 as cited in Barnich, 1997, p. 287)

Psychological Perspective

Since the 1960‘s psychologists and psycholinguists tried to resolve how language is organised and represented inside the brain. Patients with aphasias gave good evidence for location and discrimination of the three main parts of language comprehension and production, namely phonology, syntax and semantics.

Phonology

Phonology deals with the processing of meaningful parts of speech resulting from the mere sound. More over there exists a differentiation between a phonemic representation of a speech sound which are the smallest units of sounds that leads to different meanings (e.g. the /b/ and /p/ in bet and pat) and phonetic representation. The latter means that a speech sound may be produced in a different manner at different situations. For instance the /p/ in pill sounds different than the /p/ in spill since the former /p/ is aspirated and the latter is not.

Examining which parts are responsible for phonetic representation, patients with Broca`s or Wernicke`s aphasia can be compared. As the speech characteristic for patients with Broca`s aphasia is non-fluent, i.e. they have problems producing the correct phonetic and phonemic representation of a sound, and people with Wernicke`s aphasia do not show any problems speaking fluently, but also have problems producing the right phoneme. This indicates that Broca`s area is mainly involved in phonological production and also, that phonemic and phonetic representation do not take place in the same part of the brain. Scientists examined on a more precise level the speech production, on the level of the distinctive features of phonemes, to see in which features patients with aphasia made mistakes.

A distinctive feature describes the different manners and places of articulation. /t/ (like in touch) and /s/ (like in such) for example are created at the same place but produced in different manner. /t/ and /d/ are created at the same place and in the same manner but they differ in voicing.

Results show that in fluent as well as in non-fluent aphasia patients usually mix up only one distinctive feature, not two. In general it can be said that errors connected to the place of articulation are more common than those linked to voicing. Interestingly some aphasia patients are well aware of the different features of two phonemes, yet they are unable to produce the right sound. This suggests that though patients have great difficulty pronouncing words correctly, their comprehension of words is still quite good. This is characteristic for patients with Broca`s aphasia, while those with Wernicke`s aphasia show contrary symptoms: they are able to pronounce words correctly, but cannot understand what the words mean. That is why they often utter phonologically correct words (neologisms) that are not real words with a meaning.

Syntax

Syntax describes the rules of how words must be arranged to result in meaningful sentences. Humans in general usually know the syntax of their mother tongue and thus slip their tongue if a word happens to be out of order in a sentence. People with aphasia, however, often have problems with parsing of sentences, not only with respect to the production of language but also with respect to comprehension of sentences. Patients showing an inability of comprehension and production of sentences usually have some kind of anterior aphasia, also called agrammatical aphasia. This can be revealed in tests with sentences. These patients cannot distinguish between active and passive voice easily if both agent and object could play an active part. For example patients do not see a difference between “The boy chased the girl” and “The boy was chased by the girl”, but they do understand both “The boy saw the apple” and “The apple was seen by the boy”, because they can seek help of semantics and do not have to rely on syntax alone. Patients with posterior aphasia, like for example Wernicke`s aphasia, do not show these symptoms, as their speech is fluent. Comprehension by mere syntactic means would be possible as well, but the semantic aspect must be considered as well. This will be discussed in the next part.

Semantics

Semantics deals with the meaning of words and sentences. It has been shown that patients suffering from posterior aphasia have severe problems understanding simple texts, although their knowledge of syntax is intact. The semantic shortcoming is often examined by a Token Test, a test in which patients have to point to objects referred to in simple sentences. As might have been guessed, people with anterior aphasia have no problems with semantics, yet they might not be able to understand longer sentences because the knowledge of syntax then is involved as well.

	anterior Aphasia (e.g. Broca)	posterior Aphasia (e.g. Wernicke)
Phonology	phonetic and phonemic representation affected	phonemic representation affected
Syntax	affected	no effect
Syntax	no effect	affected

Overview of the effects of aphasia from the psychological perspective

In general studies with lesioned people have shown that anterior areas are needed for speech output and posterior regions for speech comprehension. As mentioned above anterior regions are also more important for syntactic processing, while posterior regions are involved in semantic processing. But such a strict division of the parts of the brain and their responsibilities is not possible, because posterior regions must be important for more than just sentence comprehension, as patients with lesions in this area can neither comprehend nor produce any speech. (Barnich, 1997, pp. 283–293)

Evidence from Advanced Neuroscience Methods

Measuring the functions of both normal and damaged brains has been possible since the 1970s, when the first brain imaging techniques were developed. With them, we are able to “watch the brain working” while the subject is e.g. listening to a joke. These methods (further described in chapter 4) show whether the earlier findings are correct and precise.

Generally, imaging shows that certain functional brain regions are much smaller than estimated in brain lesion studies, and that their boundaries are more distinct (cf. Banich p. 294). The exact location varies individually, therefore bringing the results of many brain lesion studies together caused too big estimated functional regions before. For example, stimulating brain tissue electrically (during epilepsy surgery) and observing the outcome (e.g. errors in naming tasks) led to a much better knowledge where language processing areas are located.

PET studies (Fiez & Petersen, 1993, as cited in Banich, p. 295) have shown that in fact both anterior and posterior regions were activated in language comprehension and processing, but with different strengths – in agreement with the lesion studies. The more active speech production is required in experiments, the more frontal is the main activation: For example, when the presented words must be repeated.

Another result (Raichle et al. 1994, as referred to in Banich, p. 295) was that the familiarity of the stimuli plays a big role. When the subjects were presented well-known stimuli sets in well-known experimental tasks and had to repeat them, anterior regions were activated. Those regions were known to cause conduction aphasia when damaged. But when the words were new ones, and/or the subjects never had to do a task like this before, the activation was recorded more posterior. That means, when you repeat an unexpected word, the heaviest working brain tissue is about somewhere under your upper left earlap, but when you knew this word that would be the next to repeat before, it is a bit nearer to your left eye.

Visual Language Processing

The processing of written language is performed when we are reading or writing and is thought to happen in a distinct neural processing unit than auditory language processing. Reading and writing respectively rely on vision whereas spoken language is first mediated by the auditory system. Language systems responsible for written language processing have to interact with a sensory system different from the one involved in spoken language processing.

Visual language processing in general begins when the visual forms of letters (“c” or “C” or “c”) are mapped onto abstract letter identities. These are then mapped onto a word form and the corresponding semantic representation (the “meaning” of the word, i.e. the concept behind it). Observations of patients that lost a language ability due to a brain damage led to different disease patterns that indicated a difference between perception (reading) and production (writing) of visual language just like it is found in non-visual language processing.

Alexic patients possess the ability to write while not being able to read whereas patients with agraphia are able to read but cannot write. Though alexia and agraphia often occur together as a result of damage to the angular gyrus, there were patients found having alexia without agraphia (e.g. Greenblatt 1973, as cited in M. T. Banich, “Neuropsychology“, p. 296) or having agraphia without alexia (e.g. Hécaen & Kremin, 1976, as cited in M. T. Banich, “Neuropsychology“, p. 296). This is a double dissociation that suggests separate neural control systems for reading and writing.

Since double dissociations are also found in phonological and surface dyslexia, experimental results support the theory that language production and perception respectively are subdivided into separate neural circuits. The two route model shows how these two neural circuits are believed to provide pathways from written words to thoughts and from thoughts to written words.

Two routes model

1.1. Each route derives the meaning of a word or the word of a meaning in a different way

In essence, the two routes model contains two routes. Each of them derives the meaning of a word or the word of a meaning in a different way, depending on how familiar we are with the word.

Using the phonological route means having an intermediate step between perceiving and comprehending of written language. This intermediate step takes places when we are making use of grapheme-to-phoneme rules. Grapheme-to-phoneme rules are a way of determining the phonological representation for a given grapheme. A grapheme is the smallest written unit of a word (e.g. “sh” in “shore”) that represents a phoneme. A phoneme on the other hand is the smallest phonological unit of a word distinguishing it from another word that otherwise sounds the same (e.g. “bat” and “cat”). People learning to read or are encountering new words often use the phonological route to arrive at a meaning representation. They construct phonemes for each grapheme and then combine the individual phonemes to a sound pattern that is associated with a certain meaning (see 1.1).

The direct route is supposed to work without an intermediate phonological representation, so that print is directly associated with word-meaning. A situation in which the direct route has to be taken is when reading an irregular word like “colonel”. Application of grapheme-to-phoneme rules would lead to an incorrect phonological representation.

According to Taft (1982, as referred to in M. T. Banich,“Neuropsychology“, p. 297) and others the direct route is supposed to be faster than the phonological route since it does not make use of a “phonological detour” and is therefore said to be used for known words ( see 1.1). However, this is just one point of view and others, like Chastain (1987, as referred to in M. T. Banich, “Neuropsychology“, p. 297), postulate a reliance on the phonological route even in skilled readers.

The processing of written language in reading

1.2. Regularity effects are common in cases of surface alexia

Several kinds of alexia could be differentiated, often depending on whether the phonological or the direct route was impaired. Patients with brain lesions participated in experiments where they had to read out words and non-words as well as irregular words. Reading of non-words for example requires access to the phonological route since there cannot be a “stored” meaning or a sound representation for this combination of letters.

Patients with a lesion in temporal structures of the left hemisphere (the exact location varies) suffer from so called surface alexia. They show the following characteristic symptoms that suggest a strong reliance on the phonological route: Very common are regularity effects, that is a mispronunciation of words in which the spelling is irregular like "colonel" or "yacht" (see 1.2). These words are pronounced according to grapheme-to-phoneme rules, although high-frequency irregularly spelled words may be preserved in some cases, the pronunciation according to the phonological route is just wrong.

Furthermore, the would-be pronunciation of a word is reflected in reading-comprehension errors. When asked to describe the meaning of the word “bear”, people suffering from surface alexia would answer something like “a beverage” because the resulting sound pattern of “bear” was the same for these people as that for “beer”. This characteristic goes along with a tendency to confuse homophones (words that sound the same but are spelled differently and have different meanings associated). However, these people are still able to read non-words with a regular spelling since they can apply grapheme-to-phoneme rules to them.

1.3. Patients with phonological alexia have to rely on the direct route

In contrast, phonological alexia is characterised by a disruption in the phonological route due to lesions in more posterior temporal structures of the left hemisphere. Patients can read familiar regular and irregular words by making use of stored information about the meaning associated with that particular visual form (so there is no regularity effect like in surface alexia). However, they are unable to process unknown words or non-words, since they have to rely on the direct route (see 1.3).

Word class effects and morphological errors are common, too. Nouns, for example, are read better than function words and sometimes even better than verbs. Affixes which do not change the grammatical class or meaning of a word (inflectional affixes) are often substituted (e.g. “farmer” instead of “farming”). Furthermore, concrete words are read with a lower error rate than abstract ones like “freedom” (concreteness effect).

Deep Alexia shares many symptomatic features with phonological alexia such as an inability to read out non-words. Just as in phonological alexia, patients make mistakes on word inflections as well as function words and show visually based errors on abstract words (“desire” → “desert”). In addition to that, people with deep alexia misread words as different words with a strongly related meaning (“woods” instead of “forest”), a phenomenon referred to as semantic paralexia. Coltheart (as referred to in the “Handbook of Neurolinguistics”, ch.41-3, p. 563) postulates that reading in deep dyslexia is mediated by the right hemisphere. He suggests that when large lesions affecting language abilities other than reading prevent access to the left hemisphere, the right-hemispheric language store is used. Lexical entries stored there are accessed and used as input to left-hemisphere output systems.

The processing of written language in spelling

The phonological route is supposed to make use of phoneme-to-grapheme rules while the direct route links thought to writing without an intermediary phonetic representation

Just like in reading, two separate routes –a phonological and a direct route- are thought to exist. The phonological route is supposed to make use of phoneme-to-grapheme rules while the direct route links thought to writing without an intermediary phonetic representation (see 1.4).

It should be noted here that there is a difference between phoneme-to-grapheme rules (used for spelling) and grapheme-to-phoneme rules in that one is not simply the reverse of the other. In case of the grapheme “k” the most common phoneme for it is /k/. The most common grapheme for the phoneme /k/, however, is “c”. Phonological agraphia is caused by a lesion in the left supramarginal gyrus, which is located in the parietal lobe above the posterior section of the Sylvian fissure (M. T. Banich, “Neuropsychology“, p. 299). The ability to write regular and irregular words is preserved while the ability to write non-words is not. This, together with a poor retrieval of affixes (which are not stored lexically), indicates an inability to associate spoken words with their orthographic form via phoneme-to-grapheme rules. Patients rely on the direct route, which means that they use orthographic word-form representations that are stored in lexical memory. Lesions at the conjunction of the posterior parietal lobe and the parieto-occipital junction cause so called lexical agraphia that is sometimes also referred to as surface agraphia. As the name already indicates, it parallels surface alexia in that patients have difficulty to access lexical-orthographic representations of words. Lexical agraphia is characterised by a poor spelling of irregular words but good spelling for regular and non-words. When asked to spell irregular words, patients often commit regularization errors, so that the word is spelled phonologically correct (for example, “whisk” would be written as “wisque”). The BEST to CONNECT is to CAPITALISE the WORDS you WANT TO COMMUNICATE for readers to COMPREHEND.

Evidence from Advanced Neuroscience Methods

How can we find evidence for the theory of the two routes. Until now neuroscientific research is not able to ascertain that there are neural circuits representing a system like the one described above. The problem of finding evidence for visual language processing on two routes in contrast to one route ( as stated by e.g. from Seidenberg & McClelland as referred to in M. T. Banich,“Neuropsychology“, p. 308) is that it is not clear what characteristic brain activation would indicate that it is either happening on two or one routes. To investigate whether there are one or two systems, neuroimaging studies examine correlations between the activations of the angular gyrus, which is thought to be a crucial brain area in written language processing and other brain regions. It was found out that during reading of non- words ( which would strongly engage the phonological route) the activation is mostly correlated with brain regions which are involved in phonological processing e.g. superior temporal regions (BA 22) and Boca’s area. During reading of normal words (which would strongly engage the direct route) the highest activation was found in occipital and ventral cortex. That at least can imply that there are two distinct routes. However, these are conclusions drawn from highest correlations which do not ensure this suggestion. What neuroimaging studies do ascertain is that the usage of a phonological and a direct route strongly overlap, which is rather unspectacular since it is quiet reasonable that fluent speaker mix both of the routes. Other studies additionally provide data in which the activated brain regions during reading of non-words and reading of normal words differ. ERP studies suggest that the left hemisphere possesses some sort mechanism which response to combinations of letters in a string, or to its orthography and / or to the phonological representation of the string. ERP waves differ, during early analysis of the visual form of the string, if the string represents a correct word or just pronounceable nonsense (Posner & McCandliss, 1993 as referred in M.T. Banich, „Neuropsychology“p. 307-308). That indicates that this mechanism is sensitive to correct or incorrect words.

The opposite hemisphere, the right hemisphere, is in contrast to the left hemisphere, not involved in abstract mapping of word meaning but is rather responsible for encoding word specific visual forms. ERP and PET studies provides evidence that the right hemisphere responds in a stronger manner than the left hemisphere to letter like strings. Moreover divided visual field studies reveal that the right hemisphere can better distinguish between different shapes of the same letter (e.g. in different handwritings) than the left hemisphere. The contribution of visual language processing on both hemispheres is that the right hemisphere first recognizes a written word as letter sequences, no matter how exactly they look like, then the language network in the left hemisphere builds up an abstract representation of the word, which is the comprehension of the word.

Other symbolic systems

Most neurolinguistic research is concerned with production and comprehension of English language, either written or spoken. However, looking at different language systems from a neuroscientific perspective can substantiate as well as differentiate acknowledged theories of language processing. The following section shows how neurological research of three symbolic systems, each different from English in some aspect, has made it possible to distinguish - at least to some extent - between brain regions that deal with the modality of the language (and therefore may vary from language to language, depending on whether the language in question is e.g. spoken or signed) from brain regions that seem to be necessary to language processing in general - regardless whether we are dealing with signed, spoken, or even musical language.

Kana and Kanji

Kana and Kanji are the two writing systems used parallel in the Japanese language. Since different approaches are used in them to represent words, studying Japanese patients with alexia is a great possibility to test the hypothesis about the existence of two different routes to meaning, explicated in the previous section.

The English writing system is phonological – each grapheme in written English roughly represents one speech sound – a consonant or a vowel. There are, however, other possible approaches to writing down a spoken language. In syllabic systems like the Japanese kana, one grapheme stands for one syllable. If written English were syllabic, it could e.g. include a symbol for the syllable “nut”, appearing both in the words “donut” and “peanut”. Syllabic systems are sound-based – since the graphemes represent units of spoken words rather than meaning directly, an auditory representation of the word has to be created in order to arrive at the meaning. Therefore, reading of syllabic systems should require an intact phonological route. In addition to kana, Japanese also use a logographic writing system called kanji, in which one grapheme represents a whole word or a concept. Different from phonological and syllabic systems, logographic systems don’t comprise systematical relationships between visual forms and the way they’re pronounced – instead, visual form is directly associated with the pronunciation and meaning of the corresponding word. Reading kanji should therefore require the direct route to meaning to be intact.

The hypothesis about the existence of two different routes to meaning has been confirmed by the fact that after brain damage, there can be a double dissociation between kana and kanji. Some Japanese patients can thus read kana but not kanji (surface alexia), whereas other can read kanji but not kana (phonological alexia). In addition, there is evidence that different brain regions of Japanese native speakers are active while reading kana and kanji, although like in the case of English native speakers, these regions also overlap.

Since the distinction between direct and phonological route also makes sense in case of Japanese, it may be a general principle common to all written languages that reading them relies on two independent (at least partially) systems, both using different strategies to catch the meaning of a written word – either associating the visual form directly with the meaning (the direct route), or using the auditory representation as an intermediary between the visual form and the meaning of the word (the phonological route).

The Japanese Kanji sign for the concept "Book", "writing", or "calligraphy

Sign Language

From a linguistic perspective, sign languages share many features of spoken languages – there are many regionally bounded sign languages, each with a distinct grammar and lexicon. Since at the same time, sign languages differ from spoken languages in the way the words are “uttered”, i.e. in the modality, neuroscientific research in them can yield valuable insights into the question whether there are general neural mechanisms dealing with language, regardless of its modality.

Structure of SL

Sign languages are phonological languages - every meaningful sign consists of several phonemes (phonemes used to be called cheremes (Greek χερι: hand) until their cognitive equivalence to phonemes in spoken languages was realized) that carry no meaning as such, but are nevertheless important to distinguish the meaning of the sign. One distinctive feature of SL phonemes is the place of articulation – one hand shape can have different meanings depending on whether it’s produced at the eye-, nose-, or chin-level. Other features determining the meaning of a sign are hand shape, palm orientation, movement, and non-manual markers (e.g. facial expressions).

To express syntactic relationships, Sign Languages exploit the advantages of the visuo-spatial medium in which the signs are produced – the syntactic structure of sign languages therefore often differs from that of spoken languages. Two important features of most sign language's grammars (including American Sign Language (ASL), Deutsche Gebärdensprache (DGS) and several other major sign languages) are directionality and simultaneous encoding of elements of information:

Directionality

The direction in which the sign is made often determines the subject and the object of a sentence. Nouns in SL can be 'linked' to a particular point in space, and later in the discourse they can be referred to by pointing to that same spot again (this is functionally related to pronouns in English). The object and the subject can then be switched by changing the direction in which the sign for a transitive verb is made.

Simultaneous encoding of elements of information

The visual medium also makes it possible to encode several pieces of information simultaneously. Consider e.g. the sentence "The flight was long and I didn't enjoy it". In English, the information about the duration and unpleasantness of the flight have to be encoded sequentially by adding more words to the sentence. To enrich the utterance "The flight was long” with the information about the unpleasantness of the flight, another sentence (“I did not enjoy it") has to be added to the original one. So, in order to convey more information, the length of the original sentence must grow. In sign language, however, the increase of information in an utterance doesn’t necessarily increase the length of the utterance. To convey information about the unpleasantness of a long flight experienced in the past, one can just use the single sign for "flight" with the past tense marker, moved in a way that represents the attribute "long", combined with the facial expression of disaffection. Since all these features are signed simultaneously, no additional time is needed to utter "The flight was long" as compared to "The flight was long and I didn't enjoy it".

Neurology of SL

Since sentences in SL are encoded visually, and since its grammar is often based on visual rather than sequential relationships among different signs, it could be suggested that the processing of SL mainly depends on the right hemisphere, which is mainly concerned with the performance on visual and spatial tasks. However, there is evidence suggesting that processing of SL and spoken language might be equally dependant on the left hemisphere, i.e. that the same basic neural mechanism may be responsible for all language functioning, regardless of its modality (i.e. whether the language is spoken or signed).

The importance of the left hemisphere in SL processing indicated e.g. by the fact that signers with a damaged right hemisphere may not be aphasiacs, whereas as in case of hearing subjects, lesions in the left hemisphere of signers can result in subtle linguistic difficulties (Gordon, 2003). Furthermore, studies of aphasic native signers have shown that damage to anterior portions of the left hemisphere (Broca’s area) result in a syndrome similar to Broca’s aphasia – the patients lose fluency of communication, they aren’t able to correctly use syntactic markers and inflect verbs, although the words they sign are semantically appropriate. In contrast, patients with damages to posterior portions of the superior temporal gyrus (Wernicke’s area) can still properly inflect verbs, set up and retrieve nouns from a discourse locus, but the sequences they sign have no meaning (Poizner, Klima & Bellugi, 1987). So, like in the case of spoken languages, anterior and posterior portions of the left hemisphere seem to be responsible for the syntax and semantics of the language respectively. Hence, it’s not essential for the "syntax processing mechanisms" of the brain whether the syntax is conveyed simultaneously through spatial markers or successively through word order and morphemes added to words - the same underlying mechanisms might be responsible for syntax in both cases.

Further evidence for the same underlying mechanisms for spoken and signed languages comes from studies in which fMRI has been used to compare the language processing of:

1. congenitally deaf native signers of British Sign Language,
2. hearing native signers of BSL (usually hearing children of deaf parents)
3. hearing signers who have learned BSL after puberty
4. non-signing subjects

Investigating language processing in these different groups allows making some distinctions between different factors influencing language organization in the brain - e.g. to what amount does deafness influences the organization of language in the brain as compared to just having SL as a first language(1 vs. 2), or to what amount does learning of SL as a first language differ from learning SL as native language(1,2 vs.3), or to what amount is language organized in speakers as compared to signers(1,2,3 vs.4).

These studies have shown that typical areas in the left hemisphere are activated in both native English speakers given written stimuli and native signers given signs as stimuli. Moreover, there are also areas that are equally activated both in case of deaf subjects processing sign language and hearing subjects processing spoken language – a finding which suggests that these areas constitute the core language system regardless of the language modality(Gordon, 2003).

Different from speakers, however, signers also show a strong activation of the right hemisphere. This is partly due to the necessity to process visuo-spatial information. Some of those areas, however (e.g. the angular gyrus) are only activated in native signers and not in hearing subjects that learned SL after puberty. This suggests that the way of learning sign languages (and languages in general) changes with time: Late learner's brains are unable to recruit certain brain regions specialized for processing this language (Newman et al., 1998).]

We have seen that evidence from aphasias as well as from neuroimaging suggest the same underlying neural mechanisms to be responsible for sign and spoken languages. It ‘s natural to ask whether these neural mechanisms are even more general, i.e. whether they are able to process any type of symbolic system underlying some syntax and semantics. One example of this kind of more general symbolic system is music.

Music

Like language, music is a human universal involving some combinatorial principles that govern the organizing of discrete elements (tones) into structures (phrases) that convey some meaning – music is a symbolic system with a special kind of syntax and semantics. It’s therefore interesting to ask whether music and natural language share some neural mechanisms: whether processing of music is dependent on processing of language or the other way round, or whether the underlying mechanisms underlying them are completely separate. By investigating the neural mechanisms underlying music we might find out whether the neural processes behind language are unique to the domain of natural language, i.e. whether language is modular. Up to now, research in the neurobiology of music has yielded contradicting evidence regarding these questions.

On the one hand, there is evidence that there is a double dissociation of language and music abilities. People suffering from amusia are unable to perceive harmony, to remember and to recognize even very simple melodies; at the same time they have no problems in comprehending or producing speech. There is even a case of a patient who developed amusia without aprosodia, i.e. although she couldn't recognize tone in musical sequences, she nevertheless could still make use of pitch, loudness, rate, or rhythm to convey meanings in spoken language (Pearce, 2005). This highly selective problem in processing music (amusia) can occur as a result of brain damage, or be inborn; in some cases it runs on families, suggesting a genetic component. The complement syndrome of amusia also exists – after suffering a brain damage in the left hemisphere, the Russian composer Shebalin lost his speech functions, but his musical abilities remained intact (Zatorre, McGill, 2005).

On the other hand, neuroimaging data suggest that language and music have a common mechanism for processing syntactical structures. The P600 ERP`s in the Broca area, measured as a response to ungrammatical sentences, is also elicited in subjects listening to musical chord sequences lacking harmony (Patel, 2003) – the expectation of typical sequences in music could therefore be mediated by the same neural mechanisms as the expectation of grammatical sequences in language.

A possible solution to this apparent contradiction is the dual system approach (Patel, 2003) according to which music and language share some procedural mechanisms (frontal brain areas) responsible for processing the general aspects of syntax, but in both cases these mechanisms operate on different representations (posterior brain areas) – notes in case of music and words in case of language.

Outlook

Many questions are to be answered, for it is e.g. still unclear whether there is a distinct language module (that you could cut out without causing anything in other brain functions) or not. As Evely C. Ferstl points out in her review, the next step after exploring distinct small regions responsible for subtasks of language processing will be to find out how they work together and build up the language network.

References & Further Reading

Wikipedia has related information at Neurolinguistics

Books - english

Brigitte Stemmer, Harry A. Whitaker. Handbook of Neurolinguistics. Academic Press (1998). ISBN 0126660557
Marie T. Banich: Neuropsychology. The neural bases of mental function (1997).
Ewa Dąbrowska: Language, Mind and Brain. Edinburgh University press Ltd.(2004)
a review: Evelyn C. Ferstl, The functional neuroanatomy of text comprehension. What's the story so far?" from: Schmalhofer, F. & Perfetti, C. A. (Eds.), Higher Level Language Processes in the Brain:Inference and Comprehension Processes. Lawrence Erlbaum. (2004)

Books - german

Müller,H.M.& Rickert,G. (Hrsg.): Neurokognition der Sprache. Stauffenberg Verlag (2003)
Poizner, Klima & Bellugi: What the hands reveal about the brain. MIT Press (1987)
N. Chomsky: Aspects of the Theory of Syntax. MIT Press (1965). ISBN 0262530074
Neville & Bavelier: Variability in the effects of experience on the development of cerebral specializations: Insights from the study of deaf individuals. Washington, D.C.: US Government Printing Office (1998)
Newman et al.: Effects of Age of Acquisition on Cortical Organization for American Sign Language: an fMRI Study. NeuroImage, 7(4), part 2 (1998)

Links - english

Robert A. Mason and Marcel Adam Just: How the Brain Processes Causal Inferences in Text
Neal J. Pearlmutter and Aurora Alma Mendelsohn: Serial versus Parallel Sentence Comprehension
Brain Processes of Relating a Statement to a Previously Read Text: Memory Resonance and Situational Constructions
Clahsen, Harald: Lexical Entries and Rules of Language: A Multidisciplinary Study of German Inflection.
Cherney, Leora (2001): Right Hemisphere Brain Damage
Grodzinsky, Yosef (2000): The neurology of syntax: Language use without Broca's area.
Müller, H.M. & Kutas, M. (1996). What's in a name? Electrophysiological differences between spoken nouns, proper names and one's own name. NeuroReport 8:221-225.
Müller, H. M., King, J. W. & Kutas, M. (1997). Event-related potentials elicited by spoken relative clauses Cognitive Brain Research 4:193-203.

Links - german

University of Bielefeld:
- Müller, H. M., Weiss, S. & Rickheit, G. (1997). Experimentelle Neurolinguistik: Der Zusammenhang von Sprache und Gehirn In: Bielefelder Linguistik (Hrsg.) Aisthesis-Verlag, pp. 125–128.
- Müller, H.M. & Kutas, M. (1997). Die Verarbeitung von Eigennamen und Gattungsbezeichnungen: Eine elektrophysiologische Studie. In: G. Rickheit (Hrsg.). Studien zur Klinischen Linguistik - Methoden, Modelle, Intervention. Opladen: Westdeutscher Verlag, pp. 147–169.
- Müller, H.M., King, J.W. & Kutas, M. (1998). Elektrophysiologische Analyse der Verarbeitung natürlichsprachlicher Sätze mit unterschiedlicher Belastung des Arbeitsgedächtnisses. Klinische Neurophysiologie 29: 321-330.

Michael Schecker (1998): Neuronale "Kodierung" zentraler Sprachverarbeitungsprozesse --> Debates (only a criticism)

Organizational Issues

Group Members 2007

send eMail to all
- dberndt
- phenk
- tgeishau
- bschledd

Group Members 2006

send eMail to all
- jbuergle
- maebert
- hknepper
- hnasir
- npraceju
- msumpf