Multiform Grammar: From Illustrated Historiography to the Classroom
Noa Yaari
In my dissertation titled “Multiform Arguments in the Historiography of Individualism in Pre-Modern Europe,” I introduced a new conceptual framework to analyze arguments that are composed of both words and images. This approach, which I term “multiform grammar,” is applicable to illustrated texts in any discipline and to spaces such as classrooms, art galleries and stores that may also juxtapose verbal and visual means of representation. Multiform grammar argues that illustrated arguments are constructs of two kinds of relationships between words and images. The first relationship is semantic, and it relates to the meanings that the words and the images in the argument denote. The second relationship is spatial, and it relates to the physical distances between those words and images across the body of the argument or the illustrated pages (Fig. 1). Multiform grammar can explain how authors create and communicate knowledge through illustrated arguments, using simultaneously the semantic and the physical relationships between the words and the images in their arguments. In a pedagogical context, it is a useful tool with which one can analyze, think about and design multiform arguments and activities while using words and images through varied technologies.
In my dissertation, I presented three new concepts to deepen our understanding of how words and images create unified meaning, both despite and because of the differences between them: “multiform arguments” for arguments that are composed of both words and images; “multiform references” for terms in the main text and captions that have the potential to shift the readers’ attention to images; and “multiform grammar” for the mechanism that coordinates words and images into multiform sequences that make sense. As mentioned above, “multiform grammar” is also the name of this new approach (Fig. 2). Multiform references, that have the potential to shift readers’ attention from words to images, are rhetorical devices whose role is to “bridge” between the verbal and visual systems within the same utterance. Greater awareness of how multiform references work, based on the semantic and the spatial relationships between the words and the images within a single utterance, can lead to an effective use of multiformity in the classroom, and consequently to student engagement with the material.
How can multiform references that shift students’ attention from words to images across the classroom influence student engagement with the learned material? There are three kinds of multiform references: explicit, implicit and indeterminate. An explicit multiform reference takes place when the author or the teacher explicitly asks their audience to pay attention to the image. In books, authors convey this message through using brackets in the main text, in which the abbreviation “Fig.” and the number of the figure are written. This convention is used in both the main text and at the beginning of the caption under the image. In the classroom, a teacher who says “as this image shows” explicitly draws the students’ attention to the image. Like the brackets in the main text, the phrase “this image” stops the flow of the teachers’ speech and refers the attention to something that is “outside” of the sequence uttered by the teacher and yet is inherent to it.
In some cases, teachers use a term that has a similar meaning to that of an image or to that of a detail in the image that the teachers show while using the term. Semantic similarity can take place, for example, if teachers say “love” and on the screen there are visual forms that seem to denote the same meaning (Fig. 3). Such a semantic relation constructs an implicit multiform reference, which has the potential to shift attention from words to images by virtue of a certain semantic relation rather than by an explicit convention. Indeterminate multiform reference is a rhetorical device that links words and images in a vague, associative way, that is hard to define. Any word that teachers utter and any image that they show either simultaneously or consecutively maintain a semantic and spatial relationship. The meanings that the word and the image denote and the physical space between them create notions in the students’ minds. Being aware of those multiform notions and how they are created by multiform grammar can be utilized to capture students’ attention, direct it to the most relevant points, and consequently keep the students engaged.
An example of how I used multiform grammar in my teaching can be found in the way I divided the class into teams, in a week in which we read about gambling. I prepared a big envelope on which I drew a happy figure calling “I’m lucky!” In the envelop, I put notes on which I drew $ in five different colours, to divide the class into five teams. Then, during the class, I invited each student to pull out a note at random from the envelope. In this case, multiform grammar coordinated the theme learned that week (gambling); the act of pulling blindly a note and the following consolidation of teams (gambling); and the symbol on the notes and the figure’s joyful reaction (successful gambling) into a learning process that crossed media differences. Since my direction of the students’ attention to the theme of the reading was the only explicit reference within that array of materials and activities, it was implicit and indeterminate multiform references that made that array coherent and absorbing.
Recommended reading on word and image relationship and multimodality:
Bal, Mieke. Reading “Rembrandt”: Beyond the Word-Image Opposition. Cambridge: Cambridge University Press, 1991.
Barthes, Roland. “Rhetoric of the Image.” In Image, Music, Text, edited and translated by Stephen Heath, 32-51. New York: Hill and Wang, 1977.
Foucault, Michel. This Is not a Pipe. Translated and edited by James Harkness. Barkley: University of California Press, 1982.
Kress, Gunther. Multimodality: A Social Semiotics Approach to Contemporary Communication. New York: Routledge, 2010.
—. and Carey Jewitt, Jon Ogborn and Charalampos Tsatsarelis, eds. Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. New York: Continuum, 2001.
Mitchell, W. J. T. “Image X Text.” In Image Science: Iconology, Visual Culture, and Media Aesthetic. 40-47. Chicago: University of Chicago Press, 2015.
Yaari, Noa. The Multiform Grammar Lab (blog). http://noayaari.com. 2018.
About the author
Noa Yaari has a PhD in History from York and she is an artist. She also has degrees in Humanities, Education, and History and Philosophy of Science. In her research, teaching and art, she explores word and image relationship. Her project, “The Multiform Grammar Lab,“ aims to help faculty and students combine words and images effectively and creatively in their essays, presentations, websites and other formats. It intends to do so through workshops and counselling, emphasizing the coherence and aesthetics of hybrid documents. The project also endeavors to develop programs that will integrate artistic activity in face-to-face and online teaching.