Multimodality, ethnography and education in south america

(Viewed 26.10.2017).
Thorlacius, Lisbeth (2018). Visuel kommunikation på digitale medier. [Visual communication on digital media my translation]. Copenhagen: Forlaget Samfundslitteratur.

MULTILITERACIES AND MULTIMODALITY: A FOCUS ON ADDITIONAL LANGUAGE TEACHING

Vanessa Tiburtino

Federal University of Espírito Santo (UFES)
[e-mail]

This work aims to reflect on the semiotic landscape of communication and its changes because of the characteristics of contemporary society, marked by continuous and rapid transformations, becoming increasingly technologized or digitized (LANKSHEAR &KNOBEL, 2007). The work deals with multiliteracies and multimodality (KRESS, 2003) for the teaching context and questions what such changes mean in the conception of what will become a literate subject in this new communicational landscape. It focuses on the discussion of the new conditions for literacy and how it is conceived in the current literature, bringing the perspectives of the studies of the New Literacies, Multiliteracies and multimodality. In this perspective, the work brings, in its discussion, how literacy events can be created and considered in the appreciation of the text as well as interpreting the social and cultural contexts of circulation and production of design and statements. More specifically, the work encourages the way in which texts, which circulate socially as well as the chosen way and means, bring the representation of the subjects in conditions of social vulnerability, treated as refugees. In these parameters, this paper approaches as well as defends the perspective of teaching that considers the multimodal dimension of the communication, the negotiation of meanings, the production of meanings, not being restricted to a teaching perspective centered in the materiality of the language. As a result, it can be seen that multimodality and multiliteracies can support the pedagogical task of developing students' explicit understanding of a diversity of multimodal systems and their design. It is necessary to move away from a monocultural and monomodal vision of literacy. Multimodal texts can be used by teachers in the classroom as a basis for critical engagement, design, or explicit teaching of how modes construct meanings in specific genres.

Keywords: Multiliteracies, multimodalities, teaching

References

KRESS, G. (2003) Literacy in the new media age. London: Routledge.

LANKSHEAR, C. & KNOBEL, M. (2007) New literacies: changing Knowledge and classroom research. Buckingham: Open University Press.

Multimodality in the teaching of Social Sciences in the Spanish educational system: current landscape, challenges and proposals

Laura Triviño-Cabrera

Faculty of Education at the University of Malaga
[e-mail]

This paper addresses the research lines we are currently developing in a Research Project entitled Multimodal Literacy and Cultural Studies: Towards a citizenship education in postmodern society, in the subject of ​​Didactics of Social Sciences at the Faculty of Education of the University of Malaga.

This project focuses on a quadruple difficulty stemming from scarce or non-existent student knowledge on the following topics: the introduction of media culture in classrooms; connecting with the present day; the application of inter-disciplinarity and the visibility of otherness.

Some of the hypotheses raised revolve around:

• 
  A lack of development of literacy in the teaching-learning process. Students are educated via media but they are not made to think about the media.
  

• 
  popular culture excluded from subjects, either as material or as a resource, or as content in itself. Students neither contemplate, examine nor reflect in the classroom on the reality of media that surrounds them.
  
  
  
• 
  the invisibility of otherness as a controversial issue that must be addressed in the teaching-learning process to develop the competence of empathy.
  
  
  
• 
  the very limited practical dimension given to subjects: students should be able to take part in the production of their own knowledge through role play and digital technologies.
  

The objective of this paper is to present the proposals that we are implementing at different educational levels: these initiatives advance multimodal instruction, leading to educational innovation in school subjects.

References

Ávila, R.M. (2004). Tendencias epistemológicas y sociales en la enseñanza del arte y sus implicaciones didácticas. Una propuesta alternativa desde la educación artística. Cultura y Educación, 16: 1-2, 115-125. DOI 10.1174/1135640041752849

Freire, P. y Macedo, D. (1989). Alfabetización, lectura de la palabra y lectura de la realidad. España: Paidós.

Giroux, H. A. (2001). Cultura, política y práctica educativa. Barcelona: Gráo.

Groupe de Recherche en Littératie Médiatique Multimodale (2009). Recuperado de http://litmedmod.ca/

Kress, G. & Van Leeuwen, T. (2001). Multimodal discourse. The modes and media of contemporary communication. London: Arnold.

Kaltenbacher, M. (2007). Perspectivas en el análisis de la multimodalidad : desde los inicios al estado del arte. ALED 7 (1), 31-57.

Melo, Maria do Céu (2008). O Triunfo do Olhar sobre o “Triunfo da Morté” de Bruegel. Melo, Maria do Céu (Org.). (2008). Imagens na Aula de História. Diálogos e Silêncios. Ramada: Edições Pedago, 19-41.

Rowsell, J. & Walsh, M. (2012). Repensar la lectoescritura para nuevos tiempos: Multimodalidad, multiliteracidades y nuevas alfabetizaciones. Enunciación. Vol., No 1 (enero-junio 2015), pp. 141-150.

Sant, E. y Pagès, J. (2011). ¿Por qué las mujeres son invisibles en la enseñanza de la Historia? Revista Historia y Memoria, No. 3, 129-146. Recuperado de http://revistas.uptc.edu.co/index.php/historia_memoria/article/view/802/801

Santisteban Fernández, Antoni (2010). La formación de competencias de pensamiento histórico. Clío & Asociados, no. 14, 34-56. Recuperado de http://www.memoria.fahce.unlp.edu.ar/art_revistas/pr.4019/pr.4019.pdf

Segall, Avner, Heilman, Elizabeth, Cherryholmes, Cleo H. (ed.) (2006). Social Studies. The Next Generation. Re-searching in the Postmodern. New York: Peter Lang Publishing.

Valtin, Renate (2015). Declaration of European Citizens´ Right to Literacy. European Literacy Policy Network. Recuperado de http://www.eli-net.eu/fileadmin/ELINET/Redaktion/Amsterdam_conference/The_declaration_of_European_Citizens__Right_to_Literacy.pdf

Wilson, C., Grizzle, A., Tuazon. R., Akyempong, K. y Cheung. C.K. (2011). Media and Information Literacy. Curriculum for Teachers. París: UNESCO.

Towards Multimodality: Transmedial Nature of Film

Sunčana Tuksar, mag. educ. philol. angl.

Juraj Dobrila University of Pula, Faculty for Interdisciplinary, Italian and Cultural Studies, Pula, Croatia
[e-mail]

The aim of this paper is to discuss the development of narratology as a discipline in respect to film as a form. Such critical analysis draws from the works of Genette to concrete changes brought about by the debates between Chatman and Rimmon-Kenan to Bordwell’s poststructuralist concerns. There are at least two levels which point to relations between the principles of a narrative construction and understanding the image: a) icons, symbols, and literary figures on a connotative level, and b) space and time on the level of frame and shot.

Narrative form is most common in fictional films. Film embodies the stories we experience in our lives and presents them by a cause-effect chain relationship that occurs in time and space (Bordwell and Thompson, 2004). This paper provides the following two transmedial aspects of a film. Firstly, there is a multimodal nature of film as a form. Due to Genette’s narrative schemes, special emphasis is put on interpretation of modes and context as central to communication. Therefore Chatman’s work Story and Discourse comes into consideration. Secondly, the model for construction of the storyline cues follows the paradigmatic and syntagmatic approach, stemming from structuralist criticism in relation to understanding the image. In the end such an approach traces back to the works of Eisenstein and Metz and is reflected in the contemporary work of researchers concerned not only with the multimodal nature of film but also with other (non)digital texts (e.g. Kress and van Leeuwen, Burn, Nørgaard, Bateman and Schmidt, etc.).

Multimodality in digital visualizations used in education; how semiotic and technological affordances interplay in meaning making

Elise Seip Tønnessen, Professor

Department of Nordic and Media Studies, University of Agder, Kristiansand, Norway
[e-mail]

This paper will take its outset in a project investigating how students in upper secondary school read, understand and learn from digital visual representations in the subject of social sciences. Data visualizations available from the Internet, such as Gapminder or national statistics (i.e. Statistics Norway) are applied in project based learning in Norwegian upper secondary schools, and this project aims at exploring the affordances of data visualizations as well as study empirically how they function in a context of learning.

Meaning-making from such digital visualizations depends not only on the interpretation of the multimodal text appearing on the screen, but also on the understanding of the interactive choices available with digital technology, and on conventions established in statistics and graphic design.

The question this paper sets out to discuss is: How do digital visual representations of numeric data combine semiotic and technological resources for meaning making? This question will first be answered through a comparative analysis of examples from two digital resources used in the classrooms we have visited (Gapminder and Statistics Norway). The main question in the comparative analysis will be how the multimodal design of semiotic resources is integrated in the interactive digital design allowing the users to choose variables and different ways to display them.

In the next instance this will open a theoretical discussion of how these examples may inform our understanding of the concept of mode. According to Gunther Kress (2014, p. 64) “modes consist of bundles of (highly diverse) features”, some of which are mode specific, others are resources that work across modes. With new semiotic practices appearing in digital media, we may need to “readjust our terminologies and theoretizations accordingly” (van Leeuwen 2015, p. 102).

References

Gapminder (2018). Accessed from https://www.gapminder.org/tools/#_chart-type=bubbles

Kress, G. (2014). What is a mode? In Jewitt (ed.) The Routledge Handbook of Multimodal Analysis. London and New York: Routledge.

Statistics Norway (2018). Key figures for the population. Accessed from https://www.ssb.no/en/befolkning/nokkeltall/population

Van Leeuwen, T. (2015). Interview with Theo van Leeuwen in Andersen, T.H., Boeriis, M., Maagerø, E. and Tønnessen, E.S.: Social Semiotics: Key Figures, New D

Image Creation and Localization on the Corporate Homepage: a Multimodal approach

Hui Wang, Cheng-Hung Lo

Xi’an Jiaotong-Liverpool University

With the increasing world-wide e-commerce proliferation, leveraging the web presence and localization of companies is considered as online advertising for the company and one of the quickest ways for the company to tap the global markets, the design of the corporate homepage, is hence vital to the image-building of the company. This paper explores the construction and localization of corporate images on its homepage, aiming at helping to enhance the global presence of companies and thereby expand their internal and external markets. Informed by the multimodal approach, Systemic Functional Linguistics and theories of User Experience (UX), this paper compares and analyzes the image of Huawei, a multinational telecommunication company ranking in the top global 100 in July 2017 and 3rd in the world in mobile phone shipments in 2015, as presented on the corporate homepage, with a focus on how the verbal and the visual codes work together to create and transfer meaning and construct the corporate image. The Chinese and English advertisements of HUAWEI Mate10 are extracted from homepages respectively for analysis, of which the Chinese advertising is deemed as the source text, and the English version is the localized, given that HUAWEI is a China-based company. The analysis reveals that the two advertisements for the same product select different information for and display distinct patterns of presentation in the image creation and localization process, and give weight to the interaction between the visual and the verbal texts. The findings of the comparative analyses are then discussed in relation to the rationales proposed by Garrett（2010）. It is then argued that the creation and localization of online corporate images should be made in a user-centered manner, so as to be better tailored to the needs of the target market.

Keywords: Corporate image, multimodality, user experience, web localization

References

Garrett, J. J. The Elements of User Experience: User-Centered Design for the Web and Beyond. CA: New Riders, 2010.

Visualizing English as a Symbol: A Multimodal Discourse Analysis of the Early Education Program Flyers in China

Jin Wang, PhD. Associate Professor

School of Foreign Languages, Shenzhen University, China
[e-mail]

Utilizing tools from Kress and van Leeuwen’s ( 2006) inter-semiosis framework and Halliday’s (1994) systemic functional grammar, this study explores how the English language is visualized as a symbol of internationalization and modernity in the early education program flyers in China as multimodal discourse. While it is widely perceived that English is closely related to internationalization and modernity in countries of the expanding circle (Kachru 1985), little has been said with reference to this symbol being discursively and multimodally construed. Flyers from ten well-known early education companies in China are collected in that the expectations of Chinese parents for their children are displayed nowhere better than the early education program flyers, an arena where expectations as naturalized ideologies of the country are reflected and construed. Multimodal discourse analysis reveals that two types of discursive practice, i.e. code-switching with embedded English elements and images of white children as native English speakers are employed to visualize English as a symbol, which is interwoven into and enhanced by semantics realized through other modal elements such as layout, color of the pictures, distance of characters in the pictures, and the language. The study adds insights into understanding the discursive construal of ideology of the English language in the contemporary Chinese consumerist culture.

References

Halliday, M.A.K. (1994) An Introduction to Functional Grammar, 2nd edn. London: Arnold.

Kachru, B. B. 1985. Standards, codification, and sociolinguistic realism: the English language in the outer circle. In R. Quirk and H. Widdowson (eds.). English in the World: Teaching and Learning the Language and the Literature. Cambridge: Cambridge University Press, 11-30.

Kress, G. and Van Leeuwen, T. (2006) Reading Images: The Grammar of Visual Design, 2nd edn. London: Routledge.

An Embodied View of Physical Signs in News Cartoons

Xiaoping Wang

Soochow University, China
[e-mail]

The centric position of the body in cognition has long been the paradigm for theories of embodiment; its research, however, has been with two limitations: 1) many studies have primarily worked on a specific, single dimension of the body, lacking a dynamic and comprehensive perspective; 2) the multimodal nature of embodied cognition has been tremendously under-estimated, particularly in terms of the multimodal representation of physical signs. In view of the above two problems, the present research, selecting news cartoons as its research objects, takes a multi-layered perspective to viewing the semiosis of physical signs, for the role of the “body” amid the semiotization process might differ in different layers of representation. It is revealed that: 1) on the formal layer, physical signs, which depend on the perceptual senses, are realized by virtue of embodied simulation, which can help divert interpreters’ attention from the form to the cognitive content; 2) on the conceptual layer, the semiotization of physical signs can be considered as a modularization process, absorbing modules from different sources and then being integrated as a whole, which can be explained by metonymy and multimodal metaphor theory; 3) on the contextual layer, the referent of the “body” is believed to be gradually replaced by its contextual symbolic meaning, which, by nature, arises from the process of reinterpretation of physical signs in specific cognitive situations.

Keywords: Physical sign, semiotization, modularization, embodiment, news cartoon

When the object is the speaker: a test case for multimodal analysis

Penny Wheeler

Department of Linguistics

Macquarie University, Australia

[e-mail]
Rhetoric has long had explanatory power in multimodal analyses and theoretical considerations of multimodality. Given this history, an examination of multimodal artefacts which share one rhetorical device could test the ‘edges’ of multimodality theory by probing the functionality, materiality and communication (Bateman, Wildfeuer, & Hiippala, 2017, p. 89) of these objects.

The rhetorical device shared by these artefacts is ‘prosopopoeia’, a class of metaphor “endowing inanimate objects with speech” (Shelestiuk, 2006), that is, a kind of personification. Prosopoetic objects are inscribed with a text such as “AELFRED MEC HEHT GEWYRCAN” (“Alfred ordered me to be made”). This first-person text (“MEC”, “me”) can be considered with other modes, such as the visual and tangible, but there is an uncanny twist. The function of the rhetorical device is to simultaneously emphasise the “thingness” of the object, but also its “personhood”. So Mock (2016) is right when he claims “the creators’ desire to give the artefact a speaking identity … directs the audience to focus on the materiality of the object itself”, but it is also true that the first person voice of the artefact directs the reader to focus on their own relationship with the object, the personal connection.

This ambiguous effect of prosopopoeia is mentioned in Wheeler (2015), but in this presentation I would like to extend a multimodal analysis to the inscription’s placement in space. My contention is that where the inscription features on the object, or where the inscribed object is located in its context, can heighten the rhetorical effect of the prosopopoeia. There is an optional and playful element to these inscriptions: they can be discovered by the reader, or be overlooked without penalty. What might this mean for the concept of a communicative situation?

References

Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality foundations, research and analysis: A problem-oriented introduction. Berlin, Germany: De Gruyter Mouton.

Mock, S. R. (2016). Say what I am called: a corpus of Anglo-Saxon self-referential inscriptions. University of Oregon.

Shelestiuk, H. V. (2006). Approaches to metaphor: Structure, classifications, cognate phenomena. Semiotica, 161, 333–343. https://doi.org/10.1515/SEM.2006.069

Wheeler, P. J. (2015). When things speak. Macquarie University.
Using Information Experience Design and Multimodality to make sense of experiences in physical and virtual curated spaces

Dylan Yamada-Rice and Kevin Walker

Royal College of Art

[e-mail] / [e-mail]

This paper will show how our use of multimodal theory in relation to the discipline of Information Experience Design (IED) that we teach and direct at the Royal College of Art contributes to the development of the multimodal paradigm.

Information Experience Design is a new, hybrid field unrelated to informatics or commercial ‘user experience’ where research is undertaken using theories of information and experience drawn from the social sciences, physics, computer science and communications research (including multimodal social semiotic theory). The discipline is forward thinking centring on multisensory forms of analysis and understanding, that have been made necessary by the proliferation of technologies and the growing capacity to gather data, which has raised questions about how people experience information, and how information is both communicated in and generated from embodied experiences. IED makes use of advances in digital technologies, combined with analogue making to produce a range of multimodal data that is then analysed in order to design means for people to experience data in innovative ways such as through physical-digital installations.

After outlining IED we will draw on our interests in physical and virtual curated spaces to show how we have combined multimodality with hands-on methods of drawing, animation and making to inform changes in the museums and digital games industry. Specifically, Walker’s work focuses on visitor interactions with museum displays that incorporate both physical and digital exhibits. Yamada-Rice will draw examples from her commercially-funded study on children’s interaction and engagement with a wide range of virtual reality content and devices.
Intercultural Blending in English Digital Storytelling

Yu-Feng (Diana) Yang

National Sun-Yat Sen University, Taiwan
[e-mail]

Valuing digitally-mediated multimodal composing as a newly developed literacy practice, this research investigates how English language learners (ELLs) serve as multimodal designers when working on one type of multimodal composition, digital storytelling. Grounded in “literacy as social practice,” and the notion of “designing,” (The New London Group, 1996, 2000) it explores how ELLs negotiate sociocultural, historical and ideological orientation to the digital storytelling project, re-contextualize culturally and linguistically diversified resources, and dialogue with the local and the global community during the process of multimodal composing. Study participants include 39 ELLs who work on group English digital stories and introduce aesthetics and culture stories of their local community to the global audience. Data collections include students’ digital stories, interviews, class discussions, questionnaires, and documents.

This presentation plans to discuss one of the preliminary findings of the study and focuses on ELLs’ intercultural blending practices in digitally-mediated multimodal compositions. How students select, utilize, and orchestrate local and global cultural resources in their digital story to demonstrate ideas to their global audience will be reported. For example, in their video composition situated in Qi-Shan, a town that was greatly impacted by the 921 earthquake in southern Taiwan, students struggled to arrange Taiwanese (e.g., local resources) and English (e.g., global resources) in their story. How to take advantage of the affordances of the Taiwanese oral narrations, the English oral narrations, and the English subtitles to reach the global audience and to uncover the local cultures of family relationships remain crucial for the students. In addition, images and dialogues emphasizing a stubborn father (e.g., local resource), a gossiping neighbor (e.g., local resource), and a caring daughter (e.g., local resource) are orchestrated to present the image of home (e.g., global resource) and what it can mean to the global audience. The relationships between local and global resources in relation to multimodality and designing will be discussed.