Multimodal History Learning Module: Text, Multimedia, Student Worksheet, and Rubric

Over the past six months, I’ve been fortunate to work with a wonderful group of scholars based at Universität Hamburg (Dr. Thorsten Logge, Dr. Philipp Wendler, and Catharina Köhnke) ; Universität Wien (Dr. Li Gerhalter); Macquarie University (Dr. Tanya Evans); and University of Missouri–Kansas City (Dr. Andrew Bergerson and Dr. Brian Frehner). We’re collaborators in The German Migration Research Network (GMRN), which offers a teaching-learning approach to the history of migration from Germany and Austria to Australia and the United States. The project, funded by the German Academic Exchange Service (DAAD), acts as a continuation of the international German Migration to Missouri digital courses. We are building Open Educational Resources for synchronous and asynchronous learning-teaching modules that will be hosted on the homepage of the Hamburg Open Online University. The GMRN courses are tuition-free and open to the public. Participants receive written confirmation of the skills and knowledge acquired during the courses.

I am developing several of the modules, one of which is on Multimodal History. What started our as a teaching module has now expanded into an essay on pedagogy, which I hope to finish by early October—right about the time the modules will go online. I’ve written with Fiona McDonald on multimodal anthropology for American Anthropologist in the past, and I’m looking forward to writing on multimodality in my home discipline. In the meantime, I’ve decided to post my module, Multimodal History, which is an extension of an earlier module that I created for my Digital Public History course.

What is Multimodality?

At its core, the concept of multimodality is relatively simple. Defined by Eve Bearne and Helen Wolstencroft, "Multimodality involves the complex interweaving of word, image, gesture and movement, and sound, including speech.  These can be combined in different ways and presented through a range of media." (Bearne 2007). The concept of multimodality recognizes that communication (and especially the narratives that we exchange with each other) is rarely limited to a single communicative mode. For example, when we talk to to each other, we express ideas with our eyes, our demeanor, our volume, and our movements, in addition to the words we use. 

A large interdisciplinary literature on multimodality has developed over the past few decades. However, outside of the realm of historical pedagogy, multimodality is a concept virtually absent in historical discourse. This module will not be a comprehensive introduction to the theory and application of multimodality. Rather, it will introduce some core concepts that have immediate relevance for historical practice—specifically the ways in which historians might use multimodal analysis and how they might utilize multimodality in communication. 

Let's start by defining three terms that are sometimes confusing when talking about multimodality: ”text, ”mode/modality,” and "medium." 

Text

Scholars often refer to a text as any thing that carries meaning. This could include a written text (e.g. a letter or a book) or a visual object (e.g. a painting or a photograph), but it can effectively be anything—a film, a building, a garden, or even a teaspoon. If a thing conveys meaning, it is a text. In the examples above—letters, buildings, teaspoons, etc.—texts are permanent or semi-permanent material objects. But, texts can also be ephemeral. For example, a gesture, such as a wink, can be a text because it carries meaning with it. Likewise, a Snapchat message is a text that may disappear after a certain period of time, but it nevertheless conveys meaning. 

When thinking of texts, there are two major things to keep in mind. First, inherent to our definition of a text as a thing-that-conveys-meaning is the idea that texts have the potential to be read, interpreted, or analyzed. This doesn’t necessarily mean that the person doing the reading, interpretation, or analysis will understand the intent of the author. For example, we’ve all experienced situations in which we’ve been puzzled by a communication (emoji communications can be particularly perplexing). What is important is the fact that a text has the possibility of being read, interpreted, or analyzed. 

Second, all texts are intersubjective. In other words, meaning is created through engagement with others. Or, to put it another way, meaning emerges only in a socio-cultural context. The most obvious example of this is language. When we speak or write, we are encoding our ideas through language. Languages emerge through ongoing engagement with others. They operate according to tacit or explicit rules—what we call a grammar—that shape how language can or should be used to convey meaning. Languages both reflect and shape their socio-cultural contexts, sometimes intentionally and sometimes unintentionally.

For a comprehensive understanding of a text, one not only needs to understand the language itself, but also the socio-cultural context in which it is being deployed. Take, for example, accents in Britain, which convey meaning beyond the words being spoken. In a recent study by Amanda Cole, she reports that 

"a group of almost 200 people aged 18 to 33 from south-east England were played ten-second audio clips of other young people reading the same sentence. Clips were played of over 100 people from different areas of London and across every county in the south-east.

The participants were not provided with any information about the people whose voices they heard. They were asked to make judgments on sliding scales about how friendly, intelligent and trustworthy they thought each person sounded.

My study shows that certain groups were evaluated more negatively than others. Based only on their accent, lower-working-class people were judged to be on average 14% less intelligent, 4% less friendly and 5% less trustworthy than upper-middle-class people. People from ethnic minority backgrounds were evaluated as 5% less intelligent than white people, regardless of class.

Compared with other areas of the south-east, negative judgments were made about people from London and Essex, places where the accents have been routinely devalued. For instance, people from Essex were judged on average 11% less intelligent than those from south-west London. Also, women were evaluated as being 2% less intelligent but 5% more friendly and trustworthy than men.”

In sum, Cole’s study reveals that accents in Britain reflect broader socio-cultural beliefs, dispositions, and practices. When one is speaking, their accent brings with it a host of signifiers that can be socially advantageous or disadvantageous. 

The structure and use of language itself can embody deep socio-cultural categories. Many languages, for example, reproduce gender binaries. To illustrate this, take the example of pronouns. In English, the most common first-person singular pronouns have traditionally been “he” and “she.” This gender binary in language reflects broader societal gender binaries. And, while early modern English speakers and writers had access to (and used) alternatives to “he” and “she,” including the pronouns “one” and “they,” the (largely male) grammarians of the eighteenth and nineteenth centuries generally rejected alternatives [Early modern writers often used “they” instead of the generic “he.” For example, in Jane Austen’s Pride and Prejudice (1913), she writes that “‘I cannot pretend to be sorry,’ said Wickham, after a short interruption, ‘that he or that any man should not be estimated beyond their deserts; but with him I believe it does not often happen.’” In Sense and Sensibility, Marianne Dashwood’s favorite maxim is, “no one can ever be in love more than once in their life.”]. As they argued for a standardized English, “he“ and “she” became the only acceptable first-person singular pronouns [This is not to say that this wasn’t contested. See Bodine 1975, Curzan 2003, Baron 2015]. Moreover, these grammarians codified a hierarchy of pronouns in which the male “he” was superior to the female “she,” and they argued that “he” should represent all genders when speaking in generalities. This preference is reflected in the U.S. Constitution where “he” is used to stand in for any person charged with a crime: 

“ A Person charged in any State with Treason, Felony, or other Crime, who shall flee from Justice, and be found in another State, shall on Demand of the executive Authority of the State from which he fled, be delivered up, to be removed to the State having Jurisdiction of the Crime.”

Another example comes from the British Parliament’s Abbreviation Act of 1850 (13 Vict. c. 21, 4), which stated explicitly that “in all Acts words importing the masculine gender shall be deemed and taken to include females.” In other words, the generic “he” would represent all humans in all legislation.

The emphasis on gender binaries and the use of “he” to stand in for all genders reflected a broader codification of gender categories and hierarchies. And, it was part of an increasing concern over policing gender categories and sexual beliefs and practices in the eighteenth and nineteenth centuries. 

While there are many more examples of language embodying larger socio-cultural patterns, the key point to remember is that texts are shaped by their contexts. If we want to understand a text, we need to place it in its broader context. 

Mode/Modality

Mode (or modality) refers to the way in which actors bundle together representational signals to convey meaning.  These meanings might be as simple as a warning (for example, the various cues to stop conveyed by a stop sign), or they might be complex (for example, the non-linear narrative embodied in René Magritte's "The Treachery of Images." These representational signals—what semioticians refer to as "signs”—are culturally and contextually specific. The signs we use to convey meaning depend on sociocultural lexicons and grammars. In an ideal situation, we work from the same lexicon and grammar and can understand each other. But, we have all experienced moments when we don't understand the cultural lexicon or misinterpret signs and have misread the social situation. 

When talking about a mode or modality, we are concerned with the various ways in which signs (and clusters of signs) can be represented. Signs might make use of color, sound, shape, or movement to convey an idea or set of ideas. Most of the time, meaning is made through the a combination of modalities—or, what we might call multimodality. Take, for example, the most basic page in the most basic book. In addition to the words themselves, there are other modes to convey meaning on every page: paragraph breaks, margins, font, color, and even paper stock (this sometimes conveys not only the monetary value of the book, but the value that the publishing community places in the ideas within the book). Users experience these modes with their eyes, through the feel of the book in their hands, and even the smell of the pages. This experience shapes the text’s meaning. 

Let's look at another example—a stop sign on a street in San Francisco. As basic as it is, the stop sign is actually a multimodal object. The word "STOP" is linguistic, while the layers of colors and octagonal shape are visual. And, when reading the stop sign in its context, there is a spatial component in that it is situated in reference to other signs and spaces—for example crosswalks, other signs, lines painted on the road, buildings, curbs, and street signs—all of which give more contextual meaning to the stop sign. 

Hiya111. A street in San Francisco, using an all-way stop intersection. 4 August 2019. Wikimedia Commons.  Creative Commons Attribution-Share Alike 4.0 International. https://commons.m.wikimedia.org/wiki/File:All-way_stop_intersection_in_San_Francisco.jpg

Scholars often refer to five primary modes: 

• Linguistic

• Aural / Audio

• Visual

• Gestural

• Spatial

But, scholars sometimes expand this list to include modes such as the olfactory, touch, and taste. To provide just one example of how this works, realtors in the United States will sometimes bake cookies when preparing to show a house to potential buyers. The purpose of this is to use the smell of the cookies as a signal that conveys complex ideas, including include warmth, coziness, and a feeling of welcome.

When two or more modes are brought together in the process of communication, we call this multimodality. Multimodal interpretation and communication pay attention to the ways that investigating multimodality can help us better understand complex meanings. 

Medium/Media

A medium (pl. media) is the technology used to communicate a message. Media include books (yes, books are technologies, and it's important to keep this in mind), radio signals, paintings, prints, websites, and more. 

In the most basic sense, the medium is the carrier of a sign. The medium can help determine which modes are relevant, or even possible, for conveying the meaning. And, usually, they include multiple modes.

Let's return to the example of the stop sign. As we already mentioned the modalities of the stop sign include linguistic, visual, and spatial features. The medium is the physical stop sign itself.

Sometimes--and this is especially relevant for digital humanities--communications scholars distinguish between old media and new media. To some extent, this division is entirely arbitrary. In the 1960s, television was a new medium, but in the 2010s, it is an old medium. In contemporary parlance "new media" is digital, and "old media" is analog. However, don't be surprised to see television referred to as "new media" when reading older work in communications. 

Putting the Terms Together

So far, we have three terms: text, mode/modality, and medium. How do they work together? You can think of it this way. The text is the thing-that-conveys-meaning. Its medium is the technology used to carry the meaning. Its modes are the ways its meanings are constructed (e.g. gestural, visual, auditory, etc.).

So, if we wanted to interpret a painting, we might decide that the painting is our text (though, depending on the questions we wanted to answer we might decide that the room in where the painting hangs is our text—an interpretive approach that would put the painting in another context). The medium includes the oil paints, canvas, and frame. The modalities include the visual and spatial—and, depending on the content of the painting, it might also utilize linguistic or gestural modalities.  

Multimodal History in Practice

As mentioned above, much of the literature on multimodality emerges from disciplines other than history. Historians have traditionally focused on linguistic modalities, and still, overwhelmingly, the media they work with is in manuscript or print form. This is not to say that historians do not practice multimodal analysis or multimodal communication. As historians have expanded the modes they study to include images, space, sound, and gestures, the media they engage with—video, audio recordings, and even computer code—has expanded as well. And, while historians have traditionally shared their work in the medium of print, in recent years, they have expanded the media through which they communicate to podcasts, tabletop games, and even VR.

Many historians have been deeply informed by interdisciplinary theories and methods that embody multimodal analysis—especially scholars of sensory history. And, fields such as public history and digital history have long practiced multimodal interpretive and communicative approaches. Examples include History Harvests, which digitize stories and objects in collaboration with communities; Jan Plamper’s recent work on the olfactory landscape of the Russian Revolution; and the video game Never Alone, which combines gameplay and narrative to tell the story of the Iñupiat people. Some scholars have even ventured into territory that goes beyond traditional scholarship—sometimes into the realm of the experimental. Excellent examples include the Virtual Paul’s Cross Website, which seeks to recreate the acoustics of a churchyard in the early seventeenth century.

Despite these examples, discussions of multimodal methods have been relatively absent in historiography (and, for that matter, undergraduate and graduate training). Because of this, it is often difficult for students to understand how to begin a multimodal analysis. The chart below offers some ways to think about multimodality in a historical context. In the first column are the various modes that we have introduced in this module. In the second column are some hints on how to think about reading texts from a multimodal perspective. For example, if you are analyzing sound, you can think about it from a number of perspectives, including rhythm, pitch, echo, patterns, and more. Likewise, spatial modalities involve scale, light, perspective, and form. 

READINGS: Modalities and Media

Historians have traditionally focused on linguistic modalities, so the media they typically have worked with have been books, letters, and journals. As scholars have expanded the modes they study to include images, space, sound, and gestures, the media with which they engage has expanded to include video, audio recordings, and even computer code. And, while historians have traditionally shared their work in a linguistic modality, in recent years, they have expanded the media through which they communicate. Podcasts, tabletop games, and even VR are just a few of the old and new media that historians (and especially public historians) employ in their work. This is all to say that the basics of communications theory are relevant to both the study of history and the communication with audiences. 

In this reading assignment, you will be introduced to a few classic texts in anthropology, cultural studies, and media studies to give you a sense of the importance in paying close attention to modalities and media in both historical research and communication.  

Reading: Gesture as a Modality

While gestures are a "mode" for communicating something, they are not stable entities. Gestures might mean one thing in one cultural context, but something else in another cultural context. Likewise, the same gesture might mean multiple things in the same cultural context, only to be differentiated in the moment. Clifford Geertz explores some of the complexities of gesture in his essay "Thick Description: Toward an Interpretive Theory of Culture" in Interpretation of Cultures (New York: Basic Books, 1972), 5-10  

In anthropology, or anyway social anthropology, what the practitioners do is ethnography. And it is in understanding what ethnography is, or more exactly what doing ethnography is, that a start can be made toward grasping what anthropological analysis amounts to as a form of knowledge. This, it must immediately be said, is not a matter of meth­ods. From one point of view, that of the textbook, doing ethnography is establishing rapport, selecting informants, transcribing texts, taking ge­nealogies, mapping fields, keeping a diary, and so on. But it is not these things, techniques and received procedures, that define the enterprise. What defines it is the kind of intellectual effort it is: an elaborate ven­ture in, to borrow a notion from Gilbert Ryle, "thick description." 

Ryle's discussion of "thick description" appears in two recent essays of his (now reprinted in the second volume of his Collected Papers) ad­dressed to the general question of what, as he puts it, "Le Penseur" is doing: "Thinking and Reflecting" and "The Thinking of Thoughts." Consider, he says, two boys rapidly contracting the eyelids of their right eyes. In one, this is an involuntary twitch; in the other, a conspiratorial signal to a friend. The two movements are, as movements, identical; from an l-am-a-camera, "phenomenalistic" observation of them alone, one could not tell which was twitch and which was wink, or indeed whether both or either was twitch or wink. Yet the difference, however unphotographable, between a twitch and a wink is vast; as anyone unfortunate enough to have had the first taken for the second knows. The winker is communicating, and indeed communicating in a quite precise and special way: (I) deliberately, (2) to someone in particular, (3) to impart a particular message, (4) according to a socially established code, and (5) without cognizance of the rest of the company. As Ryle points out, the winker has not done two things, contracted his eyelids and winked, while the twitcher has done only one, contracted his eye­ lids. Contracting your eyelids on purpose when there exists a public code in which so doing counts as a conspiratorial signal is winking. That's all there is to it: a speck of behavior, a fleck of culture, and­ voilà!-a gesture. 

That, however, is just the beginning. Suppose, he continues, there is a third boy, who, "to give malicious amusement to his cronies," parodies the first boy's wink, as amateurish, clumsy, obvious, and so on. He, of course, does this in the same way the second boy winked and the first twitched: by contracting his right eyelids. Only this boy is neither wink­ ing nor twitching, he is parodying someone else's, as he takes it, laugh­ able, attempt at winking. Here, too, a socially established code exists (he will "wink" laboriously, overobviously, perhaps adding a grimace-the usual artifices of the clown); and so also does a message. Only now it is not conspiracy but ridicule that is in the air. If the others think he is ac­tually winking, his whole project misfires as completely, though with somewhat different results, as if they think he is twitching. One can go further: uncertain of his mimicking abilities, the would-be satirist may practice at home before the mirror, in which case he is not twitching, winking, or parodying, but rehearsing; though so far as what a camera, a radical behaviorist, or a believer in protocol sentences would record he is just rapidly contracting his right eyelids like all the others. Com­plexities are possible, if not practically without end, at least logically so. The original winker might, for example, actually have been fake-wink­ ing, say, to mislead outsiders into imagining there was a conspiracy afoot when there in fact was not, in which case our descriptions of what the parodist is parodying and the rehearser rehearsing of course shift accordingly. But the point is that between what Ryle calls the "thin de­scription" of what the rehearser (parodist, winker, twitcher . . .) is doing ("rapidly contracting his right eyelids") and the "thick descrip­tion" of what he is doing ("practicing a burlesque of a friend faking a wink to deceive an innocent into thinking a conspiracy is in motion") lies the object of ethnography: a stratified hierarchy of meaningful structures in terms of which twitches, winks, fake-winks, parodies, rehearsals of parodies are produced, perceived, and interpreted, and without which they would not (not even the zero-form twitches, which, as a cultural category, are as much nonwinks as winks are nontwitches) in fact exist, no matter what anyone did or didn't do with his eyelids. 

Like so many of the little stories Oxford philosophers like to make up for themselves, all this winking, fake-winking, burlesque-fake-wink­ing, rehearsed-burlesque-fake-winking, may seem a bit artificial. In way of adding a more empirical note, let me give, deliberately unpreceded by any prior explanatory comment at all, a not untypical excerpt from my own field journal to demonstrate that, however evened off for didac­tic purposes, Ryle's example presents an image only too exact of the sort of piled-up structures of inference and implication through which an ethnographer is continually trying to pick his way: 

The French [the informant said] had only just arrived. They set up twenty or so small forts between here, the town, and the Marmusha area up in the middle of the mountains, placing them on promontories so they could sur­vey the countryside. But for all this they couldn't guarantee safety, espe­cially at night, so although the mezrag, trade-pact, system was supposed to be legally abolished it in fact continued as before. 

One night, when Cohen (who speaks fluent Berber), was up there, at Mar­musha, two other Jews who were traders to a neighboring tribe came by to purchase some goods from him. Some Berbers, from yet another neighbor­ ing tribe, tried to break into Cohen's place, but he fired his rifle in the air. (Traditionally, Jews were not allowed to carry weapons; but at this period things were so unsettled many did so anyway.) This attracted the attention of the French and the marauders fled. 

The next night, however, they came back, one of them disguised as a woman who knocked on the door with some sort of a story. Cohen was sus­picious and didn't want to let "her" in, but the other Jews said, "oh, it's all right, it's only a woman." So they opened the door and the whole lot came pouring in. They killed the two visiting Jews, but Cohen managed to barri­cade himself in an adjoining room. He heard the robbers planning to burn him alive in the shop after they removed his goods, and so he opened the door and, laying about him wildly with a club, managed to escape through a window. 

He went up to the fort, then, to have his wounds dressed, and complained to the local commandant, one Captain Dumari, saying he wanted his 'ar­ i.e., four or five times the value of the merchandise stolen from him. The robbers were from a tribe which had not yet submitted to French authority and were in open rebellion against it, and he wanted authorization to go with his mezrag-holder, the Marmusha tribal sheikh, to collect the indemnity that, under traditional rules, he had coming to him. Captain Dumari couldn't officially give him permission to do this, because of the French pro­hibition of the mezrag relationship, but he gave him verbal authorization, saying, "If you get killed, it's your problem." 

So the sheikh, the Jew, and a small company of armed Marmushans went off ten or fifteen kilometers up into the rebellious area, where there were of course no French, and, sneaking up, captured the thief-tribe's shepherd and stole its herds. The other tribe soon came riding out on horses after them, armed with rifles and ready to attack. But when they saw who the "sheep thieves" were, they thought better of it and said, "all right, we'll talk." They couldn't really deny what had happened-that some of their men had robbed Cohen and killed the two visitors-and they weren't prepared to start the serious feud with the Marmusha a scuffle with the invading party would bring on. So the two groups talked, and talked, and talked, there on the plain amid the thousands of sheep. and decided finally on five-hundred­ sheep damages. The two armed Berber groups then lined up on their horses at opposite ends of the plain, with the sheep herded between them, and Cohen, in his black gown, pillbox hat, and flapping slippers, went out alone among the sheep, picking out, one by one and at his own good speed, the best ones for his payment. 

So Cohen got his sheep and drove them back to Marmusha. The French, up in their fort, heard them coming from some distance ("Ba, ba, ba" said Cohen, happily, recalling the image) and said, "What the hell is that?" And Cohen said, "That is my 'ar." The French couldn't believe he had actually done what he said he had done, and accused him of being a spy for the re­bellious Berbers, put him in prison, and took his sheep. In the town, his family, not having heard from him in so long a time, thought he was dead. But after a while the French released him and he came back home, but without his sheep. He then went to the Colonel in the town, the Frenchman in charge of the whole region, to complain. But the Colonel said, "I can't do anything about the matter. It's not my problem." 

Quoted raw, a note in a bottle, this passage conveys, as any similar one similarly presented would do, a fair sense of how much goes into ethnographic description of even the most elemental sort-how extraor­dinarily "thick" it is. In finished anthropological writings, including those collected here, this fact-that what we call our data are really our own constructions of other people's constructions of what they and their compatriots are up to-is obscured because most of what we need to comprehend a particular event, ritual, custom, idea, or whatever is in­sinuated as background information before the thing itself is directly ex­amined. (Even to reveal that this little drama took place in the high­ lands of central Morocco in 1912--and was recounted there in 1968--is to determine much of our understanding of it.) There is noth­ing particularly wrong with this, and it is in any case inevitable. But it does lead to a view of anthropological research as rather more of an ob­servational and rather less of an interpretive activity than it really is. Right down at the factual base, the hard rock, insofar as there is any, of the whole enterprise, we are already explicating: and worse, explicating explications. Winks upon winks upon winks. 

Analysis, then, is sorting out the structures of signification--what Ryle called established codes, a somewhat misleading expression, for it makes the enterprise sound too much like that of the cipher clerk when it is much more like that of the literary critic--and determining their social ground and import. Here, in our text, such sorting would begin with distinguishing the three unlike frames of interpretation ingredient in the situation, Jewish, Berber, and French, and would then move on to show how (and why) at that time, in that place, their copresence produced a situation in which systematic misunderstanding reduced tradi­tional form to social farce. What tripped Cohen up, and with him the whole, ancient pattern of social and economic relationships within which he functioned, was a confusion of tongues. 

I shall come back to this too-compacted aphorism later, as well as to the details of the text itself. The point for now is only that ethnography is thick description. What the ethnographer is in fact faced with­ except when (as, of course, he must do) he is pursuing the more auto­matized routines of data collection-is a multiplicity of complex con­ceptual structures, many of them superimposed upon or knotted into one another, which are at once strange, irregular, and inexplicit, and which he must contrive somehow first to grasp and then to render. And this is true at the most down-to-earth, jungle field work levels of his ac­tivity: interviewing informants, observing rituals, eliciting kin terms, tracing property lines, censusing households . . . writing his journal. Doing ethnography is like trying to read (in the sense of "construct a reading of') a manuscript-foreign, faded, full of ellipses, incoher­encies, suspicious emendations, and tendentious commentaries, but written not in conventionalized graphs of sound but in transient examples of shaped behavior. 

To drive the point home a bit, there is an episode of Seinfeld in which the difference between a twitch and a wink makes all the difference. I think Clifford Geertz would have approved. 

Reading: Encoding/Decoding

When thinking about multimodality, it's important to recognize that meaning-making is never unidirectional. In other words, the creator of meaning (for example, a writer or artist) can never fully control how the people receiving their messages will interpret them, ignore them, or use them in unintended ways. In the Seinfeld example, George speaks clearly and directly, but Kramer misinterprets his body language. Another classic essay that examines these complexities is Stuart Hall's "Encoding and Decoding in the Television Discourse" (September 1972). https://core.ac.uk/download/pdf/81670115.pdf 

Hall’s approach is a powerful way for thinking about and analyzing texts. He reminds us that not only do texts sit within broader sociocultural contexts, but that they may reflect and reproduce systems of economic, political, and social power (as we saw in the example of gender binaries in the lecture). 

Reading: Medium, Message, and Algorithm

Despite historians' expanded use of media to communicate, an expansion in their critical and historical analysis of these media hasn't always accompanied this work (with the notable exceptions by scholars in the history of science and technology and in digital humanities). With so little focus on media, it's worthwhile to take a moment to examine why thinking about media is important. To do this, let's travel back to the 1960s to read a bit from Marshall McLuhan's, Understanding Media: The Extensions of Man, (McGraw Hill, 1964). 

In a culture like ours, long accustomed to splitting and dividing all things as a means of control, it is sometimes a bit of a shock to be reminded that, in operational and practical fact, the medium is the message. This is merely to say that the personal and social consequences of any medium - that is, of any extension of ourselves - result from the new scale that is introduced into our affairs by each extension of ourselves, or by any new technology. Thus, with automation, for example, the new patterns of human association tend to eliminate jobs, it is true. That is the negative result. Positively, automation creates roles for people, which is to say depth of involvement in their work and human association that our preceding mechanical technology had destroyed. Many people would be disposed to say that it was not the machine, but what one did with the machine, that was its meaning or message. In terms of the ways in which the machine altered our relations to one another and to ourselves, it mattered not in the least whether it turned out cornflakes or Cadillacs. The restructuring of human work and association was shaped by the technique of fragmentation that is the essence of machine technology. The essence of automation technology is the opposite. It is integral and decentralist in depth, just as the machine was fragmentary, centralist, and superficial in its patterning of human relationships. 

The instance of the electric light may prove illuminating in this connection. The electric light is pure information. It is a medium without a message, as it were, unless it is used to spell out some verbal ad or name. This fact, characteristic of all media, means that the "content" of any medium is always another medium. The content of writing is speech, just as the written word is the content of print, and print is the content of the telegraph. If it is asked, "What is the content of speech?" it is necessary to say, "It is an actual process of thought, which is in itself nonverbal." An abstract painting represents direct manifestation of creative thought processes as they might appear in computer designs. What we are considering here, however, are the psychic and social consequences of the designs or patterns as they amplify or accelerate existing processes. For the "message" of any medium or technology is the change of scale or pace or pattern that it introduces into human affairs. The railway did not introduce movement or transportation or wheel or road into human society, but it accelerated and enlarged the scale of previous human functions, creating totally new kinds of cities and new kinds of work and leisure. This happened whether the railway functioned in a tropical or a northern environment, and is quite independent of the freight or content of the railway medium. The airplane, on the other hand, by accelerating the rate of transportation, tends to dissolve the railway form of city, politics, and association, quite independently of what the airplane is used for. 

Let us return to the electric light. Whether the light is being used for brain surgery or night baseball is a matter of indifference. 

It could be argued that these activities are in some way the "content" of the electric light, since they could not exist without the electric light. This fact merely underlines the point that "the medium is the message" because it is the medium that shapes and controls the scale and form of human association and action. The content or uses of such media are as diverse as they are ineffectual in shaping the form of human association. Indeed, it is only too typical that the "content" of any medium blinds us to the character of the medium. It is only today that industries have become aware of the various kinds of business in which they are engaged. When IBM discovered that it was not in the business of making office equipment or business machines, but that it was in the business of processing information, then it began to navigate with clear vision. The General Electric Company makes a considerable portion of its profits from electric light bulbs and lighting systems. It has not yet discovered that, quite as much as AT&T, it is in the business of moving information. 

The electric light escapes attention as a communication medium just because it has no "content." And this makes it an invaluable instance of how people fail to study media at all. For it is not till the electric light is used to spell out some brand name that it is noticed as a medium. Then it is not the light but the "content" (or what is really another medium) that is noticed. The message of the electric light is like the message of electric power in industry, totally radical, pervasive, and decentralized. For electric light and power are separate from their uses, yet they eliminate time and space factors in human association exactly as do radio, telegraph, telephone, and TV, creating involvement in depth. 

A fairly complete handbook for studying the extensions of man could be made up from selections from Shakespeare. Some might quibble about whether or not he was referring to TV in these familiar lines from Romeo and Juliet: 

But soft! what light through yonder window breaks? 

It speaks, and yet says nothing. 

McLuhan's argument is that technologies are not neutral--that they are not simply carriers of meaning, but makers of meaning. In so doing, they transform human societies. Let's build on these insights just a bit by jumping ahead 55 years after McLuhan's initial insights to turn to the work of Safiya Umoja Noble, who is the author of Algorithms of Oppression: How Search Engines Reinforce Racism  (NYU Press, 2018). Noble's book forces us to rethink McLuhan by 1) emphasizing the role that the builders of media (programmers and corporations) play in shaping the ways technologies function and 2) expanding the argument that technologies are never neutral. To get a sense of the scholarship in Umoja Noble’s text, watch this video:

While we will not read it in this module, I encourage you to read Ruha Benjamin's book, Race after Technology: the New Jim Code (Polity Press, 2019). Benjamin looks at the algorithms built into the digital media we use, which are effectively a "black box" that we never get to see inside but which present themselves as neutral. Benjamin argues that these algorithms reproduce racial inequities and are, in effect,  a "New Jim Code”: 

"the employment of new technologies that reflect and reproduce existing inequities but that are promoted and perceived as more objective or progressive than the discriminatory systems of a previous era. Like other kinds of codes that we think of as neutral, 'normal' names have power by virtue of their perceived neutrality."

A Little Something Extra

For historians, working with written and visual texts is often second nature. Working in other modalities requires a bit study and practice. To get you started on how to think about a text, I am including this short example of how one might analyze music in the context of a multimodal history. Music and sound are never there to fill an aural void. Rather, they are themselves rhetorical devices that might evoke mood, emphasize an idea, or generate an argument. To give you some sense of the complex ways in which music functions in communication, watch Adam Neely's analysis of Lady Gaga's and Michael Bearden's mixed meter ”Star Spangled Banner“ at the presidential inauguration of Joe Biden.

Assignment

Learning Objectives

In this assignment, students will apply the information that they have learned in this module to a set of historical documents. Students will first analyze a set of documents through the lens of multimodality and then generate a set of potential research approaches to better understand them. 

Directions

In this assignment, you will be introduced to the central architectural space in Indianapolis known as monument circle.” While the circle was included in the original plat map of Indianapolis created by Alexander Ralston, the contemporary design of the circle is dominated by Soldiers and Sailors Monument, constructed between 1888 and 1901.Bruno Schmitz (1858-1916), a German architect, designed the monument and the space around it, with the major sculpture groups provided by Austrian sculptor Rudolf Schwarz (1866-1912). 

Below are a series of historical and contemporary documents that include 1) a short history of the monument, 2) a small collection of primary textual and visual documents; 3) a video and and some audio of the space that was recorded in May 2022. 

Read, watch, and listen to the documents and ask yourself the following:

1. How has the space developed over the past 130+ years?

2. How do people move through the space? How might one’s position in the space transform one’s experience of the space?

3. In what ways have the monument creators embedded a multimodal text?

4. How might a historian read the monument through a multimodal lens?

5. How might the historical documents be interpreted through a multimodal lens?

After surveying the textual, visual, and audio documents, fill in the “multimodal analysis matrix” below. Your job is to explain how Monument Circle might be analyzed using a multimodal approach using the documents. To get you started, I have included an example in the “Linguistic” field.

Documents

Introduction 

• Selm, William L. “Soldiers and Sailors Monument.” In Encyclopedia of Indianapolis, 2nd ed. Indianapolis: Polis Center at IUPUI, 2021.

Historical Documents

• Alexander Ralston. “Plat of the Town of Indianapolis.” 1821.

• Charles F. Bretzman. Monument Circle Farewell Parade, 150th Artillery, August 26, 1917. Digital scan of nitrate negative. 14x46 in. Indiana Historical Society, NHPRC Cirkut Negative Preservation Collection, P0431. Description; During World War I countless parades were held in Indianapolis for bond rallies and sometimes to send soldiers off to the war. This view of Monument Circle shows a Farewell Parade of the 150th Artillery on August 26, 1917, as recorded by Bretzman. Groups of soldiers are marching past the crowds assembled around Monument Circle.

• W. H. Bass Photo Company. Monument Circle, East Half, 27 July 1920. Bass Photo Company Collection, P0130, Indiana Historical Society. Digital scan of nitrate negative. All Rights Reserved.

• Rose, Ernestine Bradford. The Circle, “The Center of Our Universe.” Vol. 18. Indiana Historical Society Publications 4. Indianapolis: Indiana Historical Society, 1957.

Video File

• Jason M. Kelly and Kelly Kerr. Monument Circle in Indianapolis on 12 May 2022. CC BY-NC-SA.

Audio File

• Jason M. Kelly and Kelly Kerr. Sounds from the Foot of the Benjamin Harrison Statue at Monument Circle in Indianapolis on 12 May 2022. CC BY-NC-SA.

Assignment Handout

After surveying the textual, visual, and audio documents, fill in the “multimodal analysis matrix” below. Your job is to explain how Monument Circle might be analyzed using a multimodal approach using the documents. To get you started, I have included an example in the “Linguistic” field.

Assignment Grading Rubric

Bibliography

Baron, Dennis. “The Politics of ‘He.’ Literally.” The Web of Language, December 25, 2015. https://blogs.illinois.edu/view/25/300287.

Benjamin, Ruha. Race After Technology: Abolitionist Tools for the New Jim Code. Medford, MA: Polity, 2019.

Bodine, Ann. “Androcentrism in Prescriptive Grammar: Singular ‘They’, Sex-Indefinite ‘He’, and ‘He or She’1.” Language in Society 4, no. 2 (August 1975): 129–46. https://doi.org/10.1017/S0047404500004607.

Castro, Jason B., Arvind Ramanathan, and Chakra S. Chennubhotla. “Categorical Dimensions of Human Odor Descriptor Space Revealed by Non-Negative Matrix Factorization.” PLOS ONE 8, no. 9 (September 18, 2013): e73289. https://doi.org/10.1371/journal.pone.0073289.

Cole, Amanda. “Disambiguating Language Attitudes Held towards Sociodemographic Groups and Geographic Areas in South East England.” Journal of Linguistic Geography 9, no. 1 (April 2021): 13–27. https://doi.org/10.1017/jlg.2021.2.

———. “Working-Class and Ethnic Minority Accents in South-East England Judged as Less Intelligent – New Research.” The Conversation, October 7, 2021. http://theconversation.com/working-class-and-ethnic-minority-accents-in-south-east-england-judged-as-less-intelligent-new-research-162886.

Collins, Samuel Gerald, Matthew Durington, and Harjant Gill. “Multimodality: An Invitation.” American Anthropologist 119, no. 1 (2017): 142–46. https://doi.org/10.1111/aman.12826.

Curzan, Anne, ed. “A History of Gender, People, and Pronouns: The Story of Generic He.” In Gender Shifts in the History of English, 58–82. Studies in English Language. Cambridge: Cambridge University Press, 2003. https://doi.org/10.1017/CBO9780511486913.004.

Fulkerson, Matthew. “Touch.” In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta, Summer 2020. Metaphysics Research Lab, Stanford University, 2020. https://plato.stanford.edu/archives/sum2020/entriesouch/.

Haythornthwaite, Caroline, and Richard Andrews. E-Learning Theory and Practice. London: SAGE Publications Ltd., 2011. https://doi.org/10.4135/9781446288566.

Jewitt, Carey, ed. The Routledge Handbook of Multimodal Analysis. Second Edition. Routledge Handbooks. Milton Park, Abingdon, Oxon: Routledge, 2014.

Jewitt, Carey, and Kerstin Leder Mackley. “Methodological Dialogues across Multimodality and Sensory Ethnography: Digital Touch Communication.” Qualitative Research 19, no. 1 (February 1, 2019): 90–110. https://doi.org/10.1177/1468794118796992.

Kress, Gunther R.. “Multimodal Discourse Analysis.” In The Routledge Handbook of Discourse Analysis, edited by Michael Handford and James Paul Gee. London: Routledge, 2013.

Kress, Gunther R., and Theo Van Leeuwen. Multimodal Discourse: The Modes and Media of Contemporary Communication. Oxford: Oxford University Press, 2001.

Noble, Safiya Umoja. Algorithms of Oppression: How Search Engines Reinforce Racism. New York: NYU Press, 2018.

Pink, Sarah. “Multimodality, Multisensoriality and Ethnographic Knowing: Social Semiotics and the Phenomenology of Perception.” Qualitative Research 11, no. 3 (June 1, 2011): 261–76. https://doi.org/10.1177/1468794111399835.

Plamper, Jan. “Sounds of February, Smells of October: The Russian Revolution as Sensory Experience.” The American Historical Review 126, no. 1 (March 1, 2021): 140–65. https://doi.org/10.1093/ahr/rhaa575.

Smith, Mark M. “Producing Sense, Consuming Sense, Making Sense: Perils and Prospects for Sensory History.” Journal of Social History 40, no. 4 (2007): 841–58.

———. The Smell of Battle, the Taste of Siege: A Sensory History of the Civil War. Oxford: Oxford University Press, 2014.

Takaragawa, Stephanie, Trudi Lynn Smith, Kate Hennessy, Patricia Alvarez Astacio, Jenny Chio, Coleman Nye, and Shalini Shankar. “Bad Habitus: Anthropology in the Age of the Multimodal.” American Anthropologist 121, no. 2 (2019): 517–24. https://doi.org/10.1111/aman.13265.

The New London Group. “A Pedagogy of Multiliteracies: Designing Social Futures.” Harvard Educational Review 66, no. 1 (February 8, 2010): 60–93. https://doi.org/10.17763/haer.66.1.17370n67v22j160u.

Tullett, William. “State of the Field: Sensory History.” History 106, no. 373 (2021): 804–20. https://doi.org/10.1111/1468-229X.13246.