Multimodal Analysis Spring 2016

SLS 750 Seminar in Second Language Acquisition -- Multimedia Analysis Spring 2016

Spring 2016 | CRN: 89063

Dongping Zheng, Ph.D. Associate Professor

zhengd@hawaii.edu | Twitter: @zhengdo | Wechat: dongpinghawaii | Cell: 808.635.0279 (in case of emergency)

Class meets on Tuesdays 3:00-5:50 at Moore Hall 251 (We will meet in Moore 152 on 1/26, 3/15 and 4/5; On lab days will be arranged and notified impromptu ) |

Office Hours: Thursdays 3:00-5:00 | Moore Hall 555

Description:

In this course, we will explore new ways of looking at communication and interaction (e.g., student-teacher, human-computer, doctor-patient interactions, and other professorial and mundane interaction) by using an array of multimodal analytic toolkits. A common thread in these approaches is illuminated by Edward Hutchins’ seminal work on “Cognition in the Wild”, which considers material artifacts as part and parcel of human cognition and communication. Rather than treating them as decorations or backgrounds of communication, material artifacts or external representations augment our thinking and communication, extending cognition beyond the skull.  “They allow us to think the previously unthinkable” (Kirsh 2010). For example, a particular choice of color, in combination with other features, indexes a particular evaluative language stance; A particular gesture or body movement signals a pattern of meaning-making and sense-making along and/or in combination with language; A particular type of technology (such as Facebook social media, Youtube video, or Second Life virtual world) invites different trajectories of interaction and meaning-making practices.

Backgrounded on these perspectives, we will look at multimodality from different aspects of literature; for example, Baldry and Thibault’s multimodal transcription and text analysis, Charles Goodwin’s embodied interaction, Carey Jewitt’s multimodal approach to technology, literacy and learning. We will explore software packages that have been used for multimodal transcription and analysis, such as CHILDES (open source), Mutlimodal Web Analyzer (open source), ELAN (Open source,http://tla.mpi.nl/tools/tla-tools/elan/) and Transana (http://www.transana.org/, free to SLS 750 students), Discursis (Purchase, http://www.discursis.com/)

Who should take this course?

• Students from the College of Languages, Linguistics and Literature, the College of Education, the Department of Information and Computer Sciences, College of Business, School of Medicine, etc.
• Students who are interested in exploring interaction patterns from video and audio data and other textual data.
• Students who are interested in web technologies, and curious how social medias provide new learning opportunities that are extended by multimodal analysis.
• Students who are interested in material development and instructional design, such as designing courses within Laulima or using other course management and delivery systems.
• Professionals who are interested in investigating how understanding of interaction processes  can help with any learning and training situations.

Learning Outcomes:

• Can perform multimodal analysis of any given text genre (print, hypertext, audio, video).
• Can engage in intellectual discussion of how people communicate in multimodal ways with gestures, and body movements.
• Can skillfully use one of the transcription and analytic tools.
• Can design for meaningful interactions situated in different contexts: such as classrooms, computer assisted learning environments, online virtual environments.

Note: Students who would like to take this seminar for SLS730 credit can do a project that focus on teaching.

Textbook and materials:

Required TextBook:

Baldry, A., and P. J. Thibault. 2005. Multimodal transcription and text analysis: A multimedia toolkit and coursebook. London: Equinox Publishing Ltd.  (MTTA hereafter). Available at Amazon 

Ebrary: http://site.ebrary.com.eres.library.manoa.hawaii.edu/lib/uhmanoa/detail.action?docID=10871885

Other required journal articles and book chapters can be accessed from Zotero

Recommended Books:

Coiro, J., M. Knobel, C. Lankshear, and D. J. Le, eds. 2008. Handbook of Research on New Literacies. Philadelphia, PA: Lawrence Erlbaum.

Bednarek, M., and Martin, J. R. (eds) (2010). New Discourse on Language: Functional

Perspectives on Multimodality, Identity, and Affiliation. London and New York: Continuum. Available at UHM ebrary, http://uhmanoa.lib.hawaii.edu:7008/vwebv/holdingsInfo?searchId=1169&recCount=10&recPointer=2&bibId=3240808#

Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach, London:Routledge.

Jewitt, C. (2009). An Introduction to Multimodality. in C. Jewitt (ed), The Routledge Handbook of Multimdal Analysis. London and New York: Routledge, pp. 14-27.

Unsworth, L. (2008). Multimodal semiotics: functional analysis in contexts of education. London ; New York: Continuum. 

Other course materials are available on Zotero. 

Readings and Activities 

References

*Angus D, Watson B, Smith A, Gallois C, Wiles J (2012) Visualising Conversation Structure across Time: Insights into Effective Doctor-Patient Consultations. PLoS ONE 7(6): e38014. doi:10.1371/journal.pone.0038014

*Angus, D.; Smith, A.; Wiles, J.; , "Conceptual Recurrence Plots: Revealing Patterns in Human Discourse," Visualization and Computer Graphics, IEEE Transactions on , vol.18, no.6, pp.988-997, June 2012. doi: 10.1109/TVCG.2011.100

Atkinson, D. (2011). A Sociocognitive approach to second language acquisition: How mind, body, and world work together in learning additional languages. In Atkinson, D. (Ed), Alternative approaches to second language acquisition. London: Routledge.

*Cowley, S. & Zheng, D. (2011). The turning of the tide: Rethinking language, mind and world  [Review article of Linell, P. (2009), Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense-making]. Journal of Multicultural Discourses, 6(02), 197-210.

Cowley, S. J. (2011). Distributed language. In S. J. Cowley (Ed.), Disitributed language (pp 1-14). Amsterdam: John Benjamins. Retrieved August 26, from, http://www.academia.edu/1977912/Distributed_Language

Cowley, S. (2009). Distributed language and dynamics. Pragmatics & Cognition, 17(3),495-507.

Cowley, S. (2007). The cognitive dynamics of distributed language. Language Sciences, 29, 575-583.

*Goodwin, C. (2013). The co-operative, transformative organization of human action and knowledge. Journal of Pragmatics. 46. pp. 8-23. http://dx.doi.org/10.1016/j.pragma.2012.09.003

Goodwin, C. (2007). Participation, Stance, and Affect in the Organization of Activities. Discourse and Society, 18(1). pp. 53-73.

*Hodges, B (2009). Ecological Linguistics: Values, dialogical arrays, complexity, and caring. Pragmatics & Cognition 17:3 (2009), 628–652.

Hutchins, E. (1995). Cognition in the wild. Cambridge, MA: MIT Press. Chap 7: Learning in Context.

Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. London: Routledge.  Chapter 1, Introduction.

Jones, R. (2010) Creativity and discourse. World Englishes. 29 (4): 467-480.

Kravchenko, A. (2007). Essential properties of language, or why language is not a code. Language Sciences, 29, 650-671.

Linell, P. (2009). Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense-making. Information Age Publishing, INC. Charlotte, NC.

Chapter 5 — Dialog and the Other

Chapter 7 —

Chapter 13 — Rethinking language in dynamic terms.

Linell, P. (2000) What is dialogism? Aspects and elements of a dialogical approach to language, communication and cognition. Lecture first presented at Växjö University, October 2000. This version: 2003-02-26

Magnani, L. (2006). Multimodal abduction: External semiotic anchors and hybrid representations Logic Journal of the IGPL14 (2):107-136doi:10.1093/jigpal/jzk009. 

Norris, S. (2006) Multiparty interactions: A multimodal perspective on relevance. Discourse Studies, 8 (3), pp. 401-42.

O‟Halloran, K. L. (2011). Multimodal Discourse Analysis. In K. Hyland and B. Paltridge (eds) Companion to Discourse. London and New York: Continuum.

Piirainen-Marsh, A. & Tainio, L. (2009). Collaborative Game-play as a Site for Participation and Situated Learning of a Second Language, Scandinavian Journal of Educational Research, 53: 2, 167 — 183.

*Streeck, J. (2013). Interaction and the living body. Journal of Pragmatics. 46(69-90)

*Street, B. Pahl, K. and Rowsell. J. (2009) Multimodality and New Literacy Studies. In Jewitt, C. ed. The Routledge Handbook of Multimodal Analysis.  London: Routledge. Pp191-200.

*Unsworth, L. and Cleirigh, C. (2009). Multimodality and reading: The construction of meaning through image-text interaction. In Jewitt, C. ed. The Routledge Handbook of Multimodal Analysis. London: Routledge. Pp151-163.

Steffensen, S. (2013). Human interactivity: Problem-finding, problem-solving, and verbal patterns in the wild.

Steffensen, S. V. (2012). Care and conversing in dialogical systems. Language Sciences.

Stuart, S. (2010). Enkinaesthesia, biosemiotics and the ethiosphere. In S.J. Cowley, J.C. Major, S.V. Steffensen & A. Dinis, (Eds), Signifying bodies: Biosemiosis, interaction and health (305–330). Braga: Portuguese Catholic University Press.

Thurlow, C., & Jaworski, A. (2014). “Two hundred ninety-four”: Remediation and multimodal performance in tourist placemaking. Journal of Sociolinguistics, 18(4), 459–494.

Walsh, M. (2010). Multimodal literacy: What does it mean for classroom practice. Australian Journal of Language and Literacy, 33(3), 211–239.

* readings are available in Laulima

** Any additions or changes will be colored in green