Meeting the Challenges of Designing Multimodal Texts to Promote Multiliteracies Pedagogy

Íris Susana Pires Pereira; Maitê Gil; Cristina Maria Sylla

doi:10.17231/comsoc.43(2023).4481

Authors

Íris Susana Pires Pereira Centro de Investigação em Educação, Instituto de Educação, Universidade do Minho, Braga, Portugal https://orcid.org/0000-0003-0647-2319
Maitê Gil Centro de Investigação em Estudos da Criança, Instituto de Educação, Universidade do Minho, Braga, Portugal https://orcid.org/0000-0002-2814-2540
Cristina Maria Sylla Centro de Investigação em Estudos da Criança, Instituto de Educação, Universidade do Minho, Braga, Portugal/Interactive Technologies Institute, Laboratory for Robotics and Engineering Systems, Lisbon, Portugal https://orcid.org/0000-0003-2159-7566

DOI:

https://doi.org/10.17231/comsoc.43(2023).4481

Keywords:

multiliteracies, multimodality, interculturality, digital literacy, interactive story apps

Abstract

This article aims to add to the understanding of the challenges involved in designing digital texts to promote multiliteracies pedagogy. A multiliteracies approach calls for multimodal meaning-making and cultural diversity to be integrated into new school curriculum content, and accordingly, we analyse an interactive children’s story app, named Mobeybou in Brazil. The research question addressed was: what can we learn about the design of multimodal texts aimed at promoting intercultural learning from the design of this story app? The app incorporates tangible and digital storytelling materials to promote intercultural skills among young children, focusing specifically on Brazil. Mobeybou in Brazil was studied to characterize the design of its multimodal representations of meaning, using categories from the grammar of storytelling and multimodal meaning-making, particularly those concerned with representing the experiential diversity and personal positioning of the app users. The findings provide evidence of the complexity involved in designing multimodal texts to meet the challenges of promoting multiliteracies pedagogy, highlighting the urgent need to narrow the interface between research undertaken in education, semiotics, and digital media design. The article concludes by identifying the study’s limitations and some future developments.

Downloads

Download data is not yet available.

Author Biographies

Íris Susana Pires Pereira, Centro de Investigação em Educação, Instituto de Educação, Universidade do Minho, Braga, Portugal

Íris Susana Pires Pereira is an assistant professor at the Institute of Education, University of Minho, Portugal. She is currently the director of the Revista Portuguesa de Educação. She holds a PhD in Child Studies (University of Minho). Her research focuses on multimodal literacy, pedagogy of multiliteracies, and teachers’ professional development. She has published several articles in peer-reviewed journals, book chapters, and has recently co-edited The Routledge Handbook of Digital Literacies in Early Childhood. She supervises PhD theses, master and practicum reports. She was a research member and the Portuguese representative in the COST action — The Digital Literacy and Multimodal Practices of Young Children.

Maitê Gil, Centro de Investigação em Estudos da Criança, Instituto de Educação, Universidade do Minho, Braga, Portugal

Maitê Gil is a researcher at the Research Centre on Child Studies, University of Minho, Portugal. She has a long experience as a teacher at different levels. She is a professor at the Federal Institute of Rio Grande do Sul (Brazil), currently on leave. Her research focuses on the relationship between linguistics and language teaching, especially on the linguistic education of Portuguese teachers, as well as on grammar and conceptualization within the framework of cognitive linguistics. She holds a PhD in linguistics from the Federal University of Rio Grande do Sul (Brazil) with a collaborative period at the University of Birmingham (PDSE-Capes). Recently, she developed a postdoctoral project at the Universidade Católica Portuguesa.

Cristina Maria Sylla, Centro de Investigação em Estudos da Criança, Instituto de Educação, Universidade do Minho, Braga, Portugal/Interactive Technologies Institute, Laboratory for Robotics and Engineering Systems, Lisbon, Portugal

Cristina Sylla is a principal researcher at the Research Centre in Child Studies, University of Minho, and principal investigator of two funded projects: Mobeybou: Moving Beyond Boundaries - Designing Narrative Learning in the Digital Era (http://mobeybou.com) and the project Once Upon a Time: A Kit of Tools for Fostering Children’s Development of Multiliteracies, Collaboration Skills, and Intercultural Sensitivity. Her research interests are the conceptualization, design, and evaluation of technologically innovative tools that foster playful learning and innovative teaching approaches targeting children’s development of essential 21st-century learning and innovation skills. She has over 50 international publications in peer-reviewed international scientific indexing conferences and journals. She has chaired the International Conference on Child Computer interaction 2022 (https://idc.acm.org/2022/), is part of several international networks, and integrates the scientific committee of several high ranked conferences. She is a research member of the International Collective of Research and Design in Children’s Digital (https://www.childrensdigitalbooks.com) and a member of the editorial board of the International Journal of Child Computer Interaction (cite score 3.8).

References

Bal, M. (2017). Narratology. Introduction to the theory of narrative. University of Toronto Press.

Bezemer, J., & Kress, G. (2016). Multimodality, learning and communication. A social semiotic frame. Routledge. DOI: https://doi.org/10.4324/9781315687537

Brasil, J. A., & Cabecinhas, R. (2019). Intercultural dialogue and intergroup relations in Europe: Contributions of cultural studies and social psychology. Comunicação e Sociedade, 105–118. https://doi.org/10.17231/comsoc.0(2019).3063 DOI: https://doi.org/10.17231/comsoc.0(2019).3063

Cope, B., & Kalantzis, M. (2009). Multiliteracies: New literacies, new learning. Pedagogies, 4(3),164–195. https://doi.org/10.1080/15544800903076044 DOI: https://doi.org/10.1080/15544800903076044

Council of Europe. (2008). Living together as equals in dignity [White paper]. Council of Europe Ministers of Foreign Affairs.

Frederico, A. (2021). Reading an app: Dimensions of meaning-making in digital literary reading in early childhood. Perspectiva, 39(1), 1–25. https://doi.org/10.5007/2175-795X.2021.e66013 DOI: https://doi.org/10.5007/2175-795X.2021.e66013

Gee, J. P. (2007). Social linguistics and literacies: Ideology in discourse (3.ª ed.). Taylor & Francis. DOI: https://doi.org/10.4324/9780203944806

Gibson, J. J. (1979). The ecological approach to visual perception. Houghton Mifflin.

Gil, M., Pereira, í. S. P., & Sylla, C. (no prelo). La construcción del posicionamiento lector en una narrativa digital interactiva. Una discusión a la luz de la teoría de las multiliteracidades. Cultura, Lenguaje y Representación.

Hagen, A., & Mills, K. A. (2022). Rhythm in literary apps. Visual Communication. Publicação eletrónica antecipada. https://doi.org/10.1177/14703572221078038 DOI: https://doi.org/10.1177/14703572221078038

Halliday, M., & Matthiessen, C. (2004). An introduction to functional grammar. Routledge.

Kalantzis, M., & Cope, B. (2012). New learning. Elements of a science of education. Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781139248532

Kress, G. (1997). Before writing: Rethinking the paths to literacy. Routledge.

Kress, G. (2010). Multimodality. A social semiotic approach to contemporary communication. Routledge.

Kress, G., & van Leeuwen, T. (2006). Reading images. The grammar of visual design. Routledge. DOI: https://doi.org/10.4324/9780203619728

Kulju, P., Kupiainen, R., Wiseman, A. M., Jyrkiäinen, A., Koskinen-Sinisalo, K.-L., & Mäkinen, M. (2018). A review of multiliteracies pedagogy in primary classrooms. Language and Literacy, 20(2), 80–101. https://doi.org/10.20360/langandlit29333 DOI: https://doi.org/10.20360/langandlit29333

Lim, F. V., Toh, W., & Nguyen, T. T. H. (2022). Multimodality in the English language classroom: A systematic review of literature. Linguistics and Education, 69, 101048. https://doi.org/10.1016/j.linged.2022.101048 DOI: https://doi.org/10.1016/j.linged.2022.101048

Lim, F. V., Towndrow, P. A., & Min Tan, J. (2021). Unpacking the teachers’ multimodal pedagogies in the primary English language classroom in Singapore. RELC Journal. Publicação eletrónica antecipada. https://doi.org/10.1177/00336882211011783 DOI: https://doi.org/10.1177/00336882211011783

Martin, J. R., & Rose, D. (2007). Genre relations. Mapping culture. Equinox.

Martins, M. de L., Marques, M. A., & Ramos, R. (2020). Discursos sobre migrantes, refugiados e minorias na esfera pública: O século XXI em debate. Comunicação e Sociedade, 38, 9–14. https://doi.org/10.17231/comsoc.38(2020).3134 DOI: https://doi.org/10.17231/comsoc.38(2020).3134

Mills, K. (2016). Literacy theories for the digital age: Social, critical, multimodal, spatial, material and sensory lenses. Multilingual Matters. DOI: https://doi.org/10.21832/9781783094639

Mills, K., Unsworth, L., & Scholes, L. (2023). Literacy for digital futures: Mind, body, text. Routledge. DOI: https://doi.org/10.4324/9781003137368

Mobeybou. (2020). Mobeybou in Brazil (6.8) [Jogo].

New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60–92. https://doi.org/10.17763/haer.66.1.17370n67v22j160u DOI: https://doi.org/10.17763/haer.66.1.17370n67v22j160u

New London Group. (2000). A pedagogy of multiliteracies. In B. Cope & M. Kalantzis (Eds.), Multiliteracies (pp. 1–28). Routledge.

O’Brien, A. (2014). Using focalisation to manipulate audience viewpoint in 3-D animation narratives: What do student authors need to know? In L. Unsworth & A. Thomas (Eds.), English teaching and new literacies pedagogy. Interpreting and authoring digital multimodal narratives (pp. 123–150). Peter Lang.

Painter, C., Martin, J. R., & Unsworth, L. (2012). Reading visual narratives: Image analysis of children’s picture books. Equinox.

Pereira, Í. S. P. (2019). Para uma pedagogia da leitura realizada on-line para construir conhecimento. Texto Digital, 15(2), 28–56. DOI: https://doi.org/10.5007/1807-9288.2019v15n2p28

Pereira, Í. S. P., Gil, M., & Sylla, C. (2022). Para a construção da teoria dos multiletramentos. Dimensões ideacionais (e implicações pedagógicas) da multimodalidade textual numa storyapp para crianças. Calidoscópio, 20(1), 65–89. https://doi.org/10.4013/cld.2022.201.04

Rowsell, J., Kress, G., Pahl, K., & Street, B. (2013). The social practice of multimodal reading: A new literacy studies – Multimodal perspective on reading. In D. E. Alvermann, N. Unrau, & R.B. Rudell (Eds.), Theoretical models and processes of reading (pp. 1182–1207). International Reading Association. DOI: https://doi.org/10.1598/0710.43

Sheets-Johnstone, M. (1999). The primacy of movement. John Benjamins. DOI: https://doi.org/10.1075/aicr.14

Sheets-Johnstone, M. (2011). Embodied minds or mindful bodies? A question of fundamental, inherently inter-related aspects of animation. Subjectivity, 4, 451–466. DOI: https://doi.org/10.1057/sub.2011.21

Silva, A. M., Cabecinhas, R., & Evans, R. (2019). Cultures, memories, dialogues under construction. Comunicação e Sociedade, 13–17. https://doi.org/10.17231/comsoc.0(2019).3057 DOI: https://doi.org/10.17231/comsoc.0(2019).3057

Stein, P. (2008). Multimodal pedagogies in diverse classrooms. Representations, rights and resources. Routledge. DOI: https://doi.org/10.4324/9780203935804

Sylla, C., Gil, M., & Pereira, Í. S. P. (2022). Narrating by doing: A bridging concept for understanding and informing the design of tangible interfaces for storytelling. Interacting with Computers, 33(6), 627–640. https://doi.org/10.1093/iwc/iwac016 DOI: https://doi.org/10.1093/iwc/iwac016

United Nations General Assembly. (2015). Transforming our world: The 2030 Agenda for sustainable development, A/RES/70/1. https://documents-dds-ny.un.org/doc/UNDOC/GEN/N15/291/89/PDF/N1529189.pdf?OpenElement

Unsworth, L. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning-making resources of language-image interaction. English Teaching: Practice and Critique, 5(1), 55–76.

Wilson, F. R. (1998). The hand: How its use shapes the brain, language, and human culture (1.ª ed.). Pantheon Books.

Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625– 636. https://doi.org/10.3758/BF03196322 DOI: https://doi.org/10.3758/BF03196322

Zhao, S., & Unsworth, L. (2016). Touch design and narrative interpretation: A social semiotic approach to picture book apps. In N. Kucirkova & G. Falloon (Eds.), Apps, Technology and Younger Learners: International evidence for teaching (pp. 89–102). Routledge.