UDC: 784.4(545)
     78.091(545)
COBISS.SR-ID 139170569 CIP - 12

Received: Jan 15, 2024
Reviewed: Jan 23, 2024
Accepted: Feb 07, 2024

#7 Taxonomy of gesture-sound relationships in Manual Interactions with Imaginary Objects (MIIO) in Hindustani Dhrupad vocal improvisation

Stella PaschalidouDepartment of Music Technology and Acoustics, Hellenic Mediterranean University, E. Daskalaki, 74133 Rethymno, Greecepashalidou@hmu.gr

Citation: Paschalidou Stella. 2024. "Taxonomy of gesture-sound relationships in Manual Interactions with Imaginary Objects (MIIO) in Hindustani Dhrupad vocal improvisation." Accelerando: Belgrade Journal of Music and Dance 9:7

Acknowledgement: I would like to express my sincere gratitude to all the musicians who have generously participated in my research over the years. The observations of their gestures have played a crucial role in shaping the content of this study. The work received no funding.

Abstract

Over the years, multiple taxonomies have emerged to categorize the diverse range of movements, particularly those linked to music, with a wealth of literature in music performance exploring this topic. Despite extensive discussions, conflicting opinions persist, and a consensus on a single taxonomy remains elusive, highlighting the intricate nature of movement types. In Hindustani (North Indian Classical) Dhrupad vocal music, singers often appear performing manual interactions with imaginary objects, referred to as "MIIOs" (Manual Interactions with Imaginary Objects) in this study. This paper revisits existing taxonomies in respect to gesture-sound relationships, examining perspectives in classifying MIIOs within these frameworks. Based on the results, it underscores the challenge of classifying MIIOs into these established, emphasizing their unique status in the realm of music-related movements.

Keywords:

gesture-sound relationships, taxonomy, systematic musicology, hindustani vocal music, ethnomusicology, embodied music cognition

Introduction

Over the years, various taxonomies have been proposed to categorize the wide array of movement types, particularly those related to music. A rich body of literature on this subject exists within music performance, comprising edited books, volumes, and individual discussions. Despite the extensive exploration of the topic, conflicting opinions persist, and a unanimous consensus on a single taxonomy has yet to be reached, underscoring the complexity of movement types.

In the realm of Dhrupad vocal music, a sub-genre within Hindustani (North Indian Classical) music, during improvisation singers frequently engage in the simulation of manual interactions with imaginary objects, termed as "MIIOs" (Manual Interactions with Imaginary Objects) in this work. Specifically, despite the absence of a real, tangible object (Rahaim, 2013), vocalists appear to move as if they are stretching, pulling, pushing, or throwing something. These movements comprise gripping, main action and releasing phases and convey a sense of contending with or yielding to an imaginary resistive force, suggesting that distinctive patterns of acoustic features in the voice may correspond to the effortful interactions associated with the physical properties of imagined objects, such as viscosity, elasticity, weight, and friction.

This paper aims to emphasize that MIIOs represent a unique category of music-related movement, making their classification and alignment within existing taxonomies challenging.

Literature Review

The term "gesture" holds diverse interpretations across various disciplines, encompassing cognitive sciences, experimental psychology, philosophy, linguistics, biomechanics, human computer interaction, and musicology. Despite its widespread use, these interpretations often lack clarity. This section elucidates the current research context's application of the term and its involvement in examining relationships between gesture and sound, specifically in MIIOs.

Gesture

Traditionally, gesture was broadly defined as body movement conveying information (Kurtenbach & Halteen 1990). More recently, this definition has expanded beyond physical movement to include sonic and imagistic aspects. Presently, gesture is conceptualized as both a mental and corporeal phenomenon (Leman & Godøy 2010, 8), conveying expressive information and meaning. The term now encompasses a range of manifestations, from empty handed movements to those involving real object manipulation, general body movements, dynamic auditory contours in music, visual imagery, feelings of motion (including kinaesthetic, tactile, proprioceptive sensations, and ideomotor responses), speech, and even experiences in touch, taste, and smell (Luciani 2004).

Hatten (2004, 95) contends that musical gestures signify intrinsic musical qualities characterized by 'significant energetic shaping through time with unique expressive force.' Contrary to viewing melody as a sequence of specific pitches, music theorist Ernst Kurth (1922) conceptualizes its essence as arising from internal patterns of energetic tension and release. Kurth suggests that these patterns manifest in the acoustic realm as curves of musical intensification and abatement, resulting in a 'play of [psychological] tensions.' Musical experience is often perceived as a fluctuation of tension, giving rise to emotional responses (Lerdahl & Krumhansl 2007; Krumhansl & Schenk 1997; Vines et al. 2006).

These dynamic configurations of musical gestures can be depicted through physical hand movements. As per Godøy (2009, 205), the hands not only follow the geometric aspects of sound, such as pitch contours, pitch spread, rhythmic patterns, textures, and timbral features, but also convey the sensations of effort embedded in the music. These termed 'motion bells' (Camurri et al. 2003) can be conceptualized as successive event units, their energy centered around moments of heightened emphasis. Psychologist Daniel Stern (2010) describes them as peaks and decays or impulses and rebounds, believed to have originated from our innate ability to navigate through approach and withdrawal. Stern (Ibid.) regards these units as a dynamic expression of 'vitality,' resulting from a combination of five elements: force, motion, time, space, and intentionality.

Despite recognizing variations in sensory modalities among individuals, adopting a singular, shared term offers the advantage of transcending the Cartesian divide between the physical and the mental, delving into the realm of embodied cognition. In the context of music, Jensenius et al. (2010) utilize this term to encompass the amalgamation of sensations arising from physical movement and sound. Similarly, Delalande (1988) points out that the term 'musical gesture' exists at the intersection of observable actions and mental representations.

While I endorse the idea of employing a unified term for gesture, for the sake of clarity, this paper adopts distinct terms for each modality:

• 'gesture' or 'physical gesture' is used to describe voluntary, goal-directed, and coherent movement units or chunks in music performance (even if they don’t directly produce sound). These encompass a prefix, the (main) excitation phase, and a suffix (referred to as 'action' by Jensenius ( 2007)).
• 'melodic phrase' or 'melodic movement' is utilized for sound gestures (sonic event units).

Additionally, I will reserve the more general term 'movement' for discussions that broadly pertain to motion, specifically the displacement of a body part of the human skeleton. Additionally, I will retain the term 'action' for goal-oriented, coherent body movement units that may not necessarily be associated with music production.

Event units are perceived as outcomes influenced by a combination of music-related and biomechanical factors (Hogan and Sternad 2007; Nelson 1983), along with the body's motor control dynamics and our attention and short-term memory capabilities (Snyder 2000). The subjective experience of these event units is often dominated by dynamics, seen as being instigated and guided by some propelling force (Runeson & Frykholm 1983; Stern 2010), a phenomenon particularly prominent in instrumental excitatory movements.

Hand gestures in Hindustani singing

Hand movements serve various functions in both performance and teaching contexts, including expressing the performer's personal connection with musical sounds (Clayton 2005; Rahaim 2013; Leante 2009; Fatone et al. 2011), communicating with fellow performers (Moran 2007), and engaging with the audience (Clayton 2007) or students (Pearson, 2013). Hindustani singers, as observed by Rahaim (2009) and based on personal observations, appear to interact with melodic content in two primary ways: employing open hands or closed hands (forming an initial grip).

In the open-handed mode, hands trace curves effortlessly in space, creating a melographic representation of the sound, with the hand shape assumed to mirror the voice's timbre (Rahaim, 2013 and interview testimony). Conversely, the closed-handed mode involves hands forming a grip before seemingly manipulating an imaginary object with effort, acting against an envisioned opposing force associated with altering its shape, size, or orientation. For instance, gestures may resemble the singer moving an object like a ball around in space, bouncing it on the floor, creating the shape of a knot with both hands, or pulling a heavy object or stretching something like a rubber band (Fatone et al., 2011; Rahaim, 2013), as seen in Figure 1(a,b).

Figure 1a. Snapshots of MIIOs during vocal improvisation for Afzal Hussain

Figure 1b. Snapshots of MIIOs during vocal improvisation for Lakhan Lal Sahu

This paper specifically focuses on interactions involving closed hands and imaginary objects, such as pushing, pulling, or exerting force on a substantial object in space, as well as stretching and compressing an elastic material. When these gestures occur, the emphasis shifts from the hands themselves to the envisioned object, maintaining a connection with which the performer engages. It appears as though the body is temporarily extended through an interface to the real world, allowing individual notes to be perceived as graspable objects. It remains uncertain whether vocalists possess a vivid visual imagery of these objects as clearly defined geometrical construction, or if their mental imagery is limited to the proprioceptive sensation of physicality (an aspect that could be explored in future research).

Nevertheless, in each instance, vocalists manipulate notes as if they were handling them like smooth pitch glides, holding, extending, and then releasing them. This contact persists as long as the object serves the purpose of vocalization, after which vocalists seem to unconsciously let it go. As outlined by Rahaim (2009, 82), in such interactions, the connection between gesture and voice appears to be rooted in the consistent physicality of the imagined object held by the hands, rather than its spatial and acoustic dimensions. While it is not expected for a performer to consciously compute real-time mathematical calculations of physical laws, it seems that prior recurring experiences are linked to the melodic activity during singing. Previous research work has revealed that when Dhrupad singers engage with an imaginary object, the relationship to the sound is embedded in the object’s affordances (Rizzolatti & Sinigaglia 2008: 34; Norman 1988), i.e. the interaction possibilities that the object provides—determined by its physical properties such as viscosity, elasticity, weight, friction, etc.—and the perceived effort it entails.

Affordance in interaction design, a concept describing the practical opportunities a real object invites for the user (Rizzolatti & Sinigaglia 2008, 34; Norman 1988), plays a crucial role in this examination. Introduced by Gibson (1977; 1979) in ecological psychology, affordance refers to a set of potential motor actions evoked by the intrinsic properties of a real object.

In essence, while various ways exist for an individual to interact with an object, these are constrained by its physical or geometrical properties and the surrounding context. This concept underscores the significance of prior sensorimotor experience, challenging the traditional cognitive theory's view that our understanding of a real object is solely based on its physical properties and its relation to other real objects. Instead, it emphasizes that our knowledge, termed action-based knowledge or know-how, is derived from the actions we have previously performed in connection with this type of real object. This perspective aligns with the fundamentals of enactive theories (O'Regan & Noë 2001; Thompson & Varela,2001; Varela et al. 1991) and ecological psychology by Gibson (1977; 1979).

Taxonomies and classifications

Instead of presenting an exhaustive list, only two taxonomies are discussed here. The first exclusively addresses movements accompanying singing, while the second pertains to general music-related movements. Notably, existing taxonomies for singing movements by scholars such as Clayton (2005; 2017) and Davidson (2001; 2005) do not necessarily encompass the classification of MIIOs. The second, more general movement taxonomy is elaborated upon to underscore the inherent ambiguity in classifying MIIOs.

1. Clayton's taxonomy (2005, 2017) specifically delves into the movements of vocalists in Khyāl singing, categorizing them based on their function in performance or teaching. Given the pronounced similarity between singing movements and speech movements, this taxonomy was crafted by employing a procedure analogous to McNeill's extension of 'Kendon's continuum' (McNeill, 1992; 2000; 2005) and the research conducted by Ekman & Friesen (1972) in the study of speech movements. The taxonomy encompasses 'markers' denoting musical processes or structure, 'emblems' for communication with other musicians or the audience, and 'illustrators,' which are 'tied to the content of singing, appearing analogous to the melodic flow or "motion"' (Clayton 2017, 75). Of particular interest to this work is only the last type, also referred to as 'gesticulations' by Kendon and McNeill.
2. Godøy et al. (2012) have formulated a widely acknowledged movement classification, wherein music-related movements are categorized according to their (a) content, (b) timescale, and (c) function. (a) The content pertains to the performer's training level, sociocultural background, and performance practice (solo or ensemble) and does not hold particular significance for the present study. (b) The timescale corresponds to the numerous temporal and structural intricacies inherent in movement, as considered in movement analysis methods (Francoise et al. 2012). This involves the subdivision of continuous body motion into smaller chunks characterized by varying timescales: (b1) The micro-timescale refers to durations of 0-0.5 seconds for individual tones and stationary movement features, as defined by Cambouropoulos (2001); (b2) The meso-timescale encompasses durations ranging from 0.5 to 5 seconds, referring to movement events (gestures) and the interaction between gestures and sonorous objects, as defined by Schaeffer (1966), Godøy (2006), Bonini & Rodà (2001), Mion (2003), and Hsieh & Luciani (2005); (b3) The macro-timescale pertains to longer durations, encompassing entire sections or even entire works, as delineated by Camurri et al. (2005), Suzuki & Hashimoto (2004), Canazza et al. (2004), and Bresin & Friberg (2000). Studying the relationships between movement and sound presents a challenge due to the need to comprehend information at various levels of detail (Godøy 2009). The fundamental premise of the present research is that the most prominent connections between movement and sound are anticipated at the meso-level timescale (Godøy et al. 2016). At this level, co-articulation phenomena play a role in merging distinct gestures into more extensive movement units or trajectories, and conversely, in consolidating continuous streams of information into cohesive movement units or 'atoms,' which are meaningful entities, concrete motor images, trajectories, or Gestalts (Godøy 2010; Buxton 1986; Cox 2006; Godøy et al. 2006a; Küssner 2014; Noyce et al. 2013). This concept is applicable to both sonorous and movement modalities and extends from Schaeffer's (1966) notion of the sonorous object to Godøy's (2006) concept of the gestural-sonorous object. (b4) At the meso-level timescale, the focus in sound analysis revolves around patterns or events, including motives, ornaments, melodic shapes, and dynamic profiles (Schaeffer 1952; Bregman 1990), which 'may be primordial to the singular sound or tone' (Godø, 2009: 209). Consequently, the analysis concentrates on movements typically spanning between 0.5-5 seconds, and this perspective has implications for the processes of annotation, segmentation, and feature extraction (Henbing & Leman 2007). (c) The function aspect delves into the purpose of movement in music performance and serves as the basis for our exploration of MIIOs. The functional categorization can be subdivided into communicative, sound-producing, ancillary, and sound-accompanying. (c1) Communicative movements are designed for communication and can be further categorized into endogenous (within the performer), performer-performer, or performer-audience interactions (Jensenius 2007). However, these are not relevant to the scope of this study. (c2) Sound-producing movements encompass human actions that either generate (excitation gestures) or alter (modification gestures) the resonant characteristics of an instrument. These movements exhibit a distinct cause-and-effect relationship with the sound, dictated by physical laws, and a precise energetic connection with the instrument. Excitation gestures involve initiating the sound, while modification gestures entail subsequent adjustments to it (Cado, 1988). (c3) Ancillary movements (Jensenius 2007; Wanderley & Depalle 2004), also known as sound-facilitating movements (Jensenius et al. 2010), coincide with sound producing movements, but do not directly contribute to sound generation. Ancillary movements may encompass actions that performers undertake to enhance and convey expressiveness in a performance, such as articulating, dynamically shaping, and phrasing the sound. This category of movements has undergone extensive examination by Caramiaux et al. (2012), Nusseck & Wanderley (2009), Wanderley (2002), and Vines et al. (2004). Alternatively, ancillary movements may also refer to supportive actions that instrumental music performers employ to aid sound-producing movements in various ways, without being directly responsible for sound production. These movements primarily aim for physiological comfort, such as preventing fatigue and avoiding strain injuries. This aspect has been studied, for example, by Dahl et al. (2010, 38) in the context of drumming gestures. (c4) Sound-accompanying movements do not contribute to the direct production of sound, but instead follow music, as seen in listening situations, dancing, or acting to music (Jensenius et al. 2010; Jensenius 2007). Such movements may involve the approximate mimicry of the motor activity underlying a sound-producing gesture, referred to as 'motor mimicry' or 'motormimetic sketching' by Godøy et al. (2006b, 258). They can also encompass tracing some acoustic aspect that follows the evolution of a sounding (real) object's resonance, termed 'sound tracing' by Godøy et al. (2006a, 27). In the first case, it involves imitating the excitation gesture (the presumed cause) that might have produced the sound, while in the second case, it entails tracing features of the resulting sound (the effects of the action, i.e., images of how materials resonate). Rocchesso et al. (2015) and Caramiaux et al. (2010; 2011; 2014) argue that listeners tend to mimic the imagined sound-producing gesture when they can identify the sound source (the "causal" character of movement in relation to sound). In contrast, in "non causal" situations, they resort to manually describing the sound. Further discussion on this topic is provided in the next section.

In practical terms, the delineation between the mentioned functional categories is not always straightforward, and distinctions are often challenging. This implies that a movement could potentially align equally well with multiple categories or be considered a combination of different classes. Jensenius (2007, 57) illustrates the relationship between movement and sound as a continuum in Figure 2, spanning from cases with close relationships to those with more loosely connected associations. While the four function classes are positioned on this continuum, movements in practice may fall in between any of these classes, highlighting a frequent ambiguity in terms of movement classification.

Figure 2. The 4 different music-related movement types (based on function) and their connection to sound on a continuum reflecting link to sound (Jensenius 2007, 57).

It is worth noting that scholars from the research group that introduced the previously mentioned taxonomy have also recently employed an alternative classification (Godøy et al. 2016). This alternative taxonomy comprises three categories: a) sound-producing b) sound-accompanying c) amodal, affective, or emotive

The last category encompasses any type of movement, including the physical, sonic, and imaginary (motor-imagery), associated with global sensations of the sound. This involves concepts like images of effort, balance, and other elements commonly encountered in dance. While this category could indeed encompass MIIOs, the use of this alternative taxonomy underscores the complexity of developing a classification system that is comprehensive and capable of capturing the wide array of nuances in movement.

Examining the broader movement classification based on function, categorizing singing movements as sound-accompanying appears intuitive, given their role in adding expressivity and supporting vocalization (as mentioned in interviews and discussed). However, in the previously referenced taxonomy, the term has only been applied in conjunction with instrumental movements, making it inappropriate for use in the context of singing in this research.

There seems to be some confusion in defining ancillary and sound-accompanying movements in terms of whether they must be linked to sound producing movements or not. This ambiguity is evident in Delalande (1998) using the term 'accompanying gestures' for gestures accompanying sound-producing gestures, while Jensenius et al. (2010) use the term exclusively in response to music in a listening context.The variation in terminology highlights that research in music-related movements has predominantly focused on instrumental music making.

At this juncture, I contend that the ancillary type of movements in the aforementioned functional taxonomy should not be exclusively tied to sound-producing (instrumental) movements, as they can also be observed in vocal performance. For example, singers may employ movements to facilitate voice production (Brunkan 2015) or for articulation and phrasing purposes (Pearson 2016: 200).

Therefore, I propose that sound accompanying movements encompass all types of movements that do not directly produce sound, regardless of whether they pertain to producing or reacting to sound. In other words, I suggest defining a movement class called 'sound-accompanying' to include expressive, supporting, motor-mimetic, and sound-tracing movements as sub-categories accompanying movements that produce or respond to sound. In the following section, I will aim to assert that not only does the previously mentioned 'functional' taxonomy lack the ability to classify singing movements, but also that MIIOs should be regarded as a unique case, adding further complexity to their classification.

Ambiguity in classifying ΜΙΙΟs

The functional classification mentioned earlier makes a clear separation between movements directly involved in sound production or enhancing expressiveness during music performance and those that respond to sound by either imitating sound-producing movements or tracing sound features. However, I contend that making such a functional distinction is not feasible or straightforward in the case of singing, particularly in the context of MIIOs. Classifying MIIOs solely as sound-producing movements poses challenges, and categorizing them as imitating sound-producing or sound-tracing movements also presents difficulties. I will now elaborate on the reasons behind this assertion.

The direct production of sounds pertains to instrumental (effective) movement, characterized by a strong energetic coupling between movement and sound governed by strict physical laws dictating how the sounding body vibrates in response to excitation. In the case of MIIOs, where no tangible object is involved, there is no evident energetic coupling between the hands and the imagined object, or at least it is not readily apparent. Although the performers' movements seem directed toward this imagined object, it is not the object itself that vibrates and produces the sound. Thus, labeling these movements as sound-producing gestures becomes challenging.

Simultaneously, there may be a direct or indirect physical impact on the mechanism of voice production. For example, a rapid downward push might induce a desirable wobbling effect in the voice (as in gamaks), or increased muscle tension in the arms could aid in controlling the diaphragm. However, there is certainly no direct energetic coupling between the hands and the supposed source of sound. Therefore, characterizing MIIOs as sound-producing movements may not be accurate. One could argue that MIIOs are better described as sound-accompanying movements that rely on imitation.

Godøy (2006b) distinguishes between the imitation of sound-producing gestures and sound tracing gestures. However, it remains unclear whether the performer is engaging with the melody by mimicking gestures that could potentially produce these sounds or by tracing certain aspects of the voice. Despite the absence of a tangible object, the connection to the sound appears to be mediated by the imagined object. Consequently, it seems more plausible that the performer is not primarily interacting with the outcome of their movements (tracing the sound) but is primarily focused on gestures that could potentially produce the sound.

There is an observable synchronization and temporal congruency between the voice and the apparent manipulative movements in terms of various features, such as melody, dynamics, and timbre. Vocalists seem to draw upon prior sensorimotor experiences derived from recurrent patterns and familiar interactions with the real world to execute these imitations. Fundamental embodied experiences like grasping, stretching, and releasing an elastic object serve as examples. However, the imagined object in use would not typically produce any sounds under normal conditions; an elastic band, for instance, is not expected to make a sound when stretched. Consequently, it is also not accurate to characterize these movements as imitating sound-producing gestures.

Similarly, Cox (2011) distinguishes between observed sound-producing gestures, analogous sound-producing gestures, and other analogous exertions when considering imagery. The first case involves imagining playing the violin while listening to a violin piece ('intra-modal MMI'), the second case pertains to imagining playing the same piece on a different instrument or singing it ('cross-modal MMI'), and the third case involves imagining the use of any other modality that is not primarily acoustic to match the pattern of the audio stimulus (also cross modal MMI). Perhaps the third category by Cox (imagining the use of another modality to match the pattern of the audio stimulus) would be the most suitable way to classify MIIOs.

However, this classification lacks the essence of MIIOs, which involves the mediation of movement by the imagined object and the imitation of effort as a means to simulate the imagined energy coupling between the hands and the object. Whether this energetic coupling can extend to include sound and to what extent this sound aligns with what the imagined object could potentially produce or the actual sound of the voice remains an open question.

---

Conclusion

The implied ambiguity suggests that it is unclear whether in MIIOs there exists a relationship between manual gestures and the voice, and how robust this relationship is. Furthermore, it is uncertain whether this relationship, if present, can be detected by an observer. In essence, while MIIOs are rooted in recurrent patterns of our interaction with the environment, they are not necessarily grounded in learned sensorimotor contingencies. This implies that, although the object may be envisioned as producing the sound, and the gesture may be envisioned as potentially sound-producing, the ultimate connection to the sound remains unclear. Further investigating this relationship calls for systematic empirical research work.

---

References

1. Bonini, F., and Antonio Roda. 2006. "Expressive content analysis of musical gesture: an experiment on piano improvisation." In Workshop on Current Research Directions in Computer Music, 23-28. Barcelona.
2. Bregman, Albert S. 1994. Auditory scene analysis: The perceptual organization of sound. MIT press, Bradford Books.
3. Bresin, Roberto, and Anders Friberg. 2000. "Rule-based emotional coloring of music performance." In Proceedings of the 2000 International Computer Music Conference, 364-367. San Francisco, USA.
4. Brunkan, Melissa C. 2016. "Relationships of a circular singer arm gesture to acoustical and perceptual measures of singing: A motion capture study." Update: Applications of Research in Music Education 34(3): 56-62.
5. Buxton, William, and Brad Myers. 1986. "A study in two-handed input." ACM SIGCHI Bulletin 17 (4): 321-326.
6. Cadoz, Claude. 1988. "Instrumental gesture and musical composition." In ICMC 1988-International Computer Music Conference, 1-12. Kologne, Germany.
7. Cambouropoulos, Emilios. 2001. "The local boundary detection model (LBDM) and its application in the study of expressive timing." In Proceedings of the 2001 International Computer Music Conference, 232-235. Havana, Cuba.
8. Camurri, Antonio, Gualtiero Volpe, Giovanni De Poli, and Marc Leman. 2005. "Communicating expressiveness and affect in multimodal interactive systems." Ieee Multimedia 12(1): 43-53.
9. Camurri, Antonio, Ingrid Lagerlöf, and Gualtiero Volpe. 2003. "Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques." International journal of human-computer studies 59(1-2): 213-225.
10. Canazza, Sergio, Giovanni De Poli, Carlo Drioli, Antonio Roda, and Alvise Vidolin. 2004. "Modeling and control of expressiveness in music performance." In Proceedings of the IEEE 92(4): 686-701.
11. Caramiaux, Baptiste, Frédéric Bevilacqua, and N. Schnell. 2010. Mimicking sound with gesture as interaction paradigm. Tech. rep., IRCAM-Centre Pompidou.
12. Caramiaux, Baptiste, Patrick Susini, Tommaso Bianco, Frédéric Bevilacqua, Olivier Houix, Norbert Schnell, and Nicolas Misdariis. 2011. "Gestural Embodiment of Environmental Sounds: an Experimental Study." In Proceedings of the 11th international conference on New Interfaces for Musical Expression, 144-148, vol. 11. Oslo, Norway.
13. Caramiaux, Baptiste, Marcelo M. Wanderley, and Frédéric Bevilacqua. 2012. "Segmenting and parsing instrumentalists' gestures." Journal of New Music Research 41(1): 13-29.
14. Caramiaux, Baptiste, Frédéric Bevilacqua, Tommaso Bianco, Norbert Schnell, Olivier Houix, and Patrick Susini. 2014. "The role of sound source perception in gestural sound description." ACM Transactions on Applied Perception (TAP) 11(1): 1-19.
15. Clayton, Martin. 2005. "Communication in Indian raga performance." In Musical Communication. Edited by D. Miell, D. Hargreaves, & R. MacDonald. 361-381. Oxford: Oxford University Press.
16. ________. 2017. "Time, Gesture, and Attention in a Khyāl Performance." In Ethnomusicology: A Contemporary Reader (Volume II): 249-266. Routledge.
17. Cox, Arnie. 2006. "Hearing, feeling, grasping gestures." In Music and gesture. 45-60. Edited by A. Gritten & E. King. Routledge.
18. _______. 2011. "Embodying music: Principles of the mimetic hypothesis." Music Theory Online 17, no. 2, 2011.
19. Dahl, Sofia, Frédéric Bevilacqua, and Roberto Bresin. 2010. "Gestures in performance." In Musical Gestures, pp. 48-80. Routledge.
20. Davidson, Jane W. 2001. "The role of the body in the production and perception of solo vocal performance: A case study of Annie Lennox." Musicae Scientiae 5(2): 235-256.
21. _______. 2005. "Bodily communication in musical performance." In Musical communication, 215-237. Oxford University Press.
22. Delalande, François. 1988. "La gestique de Gould: éléments pour une sémiologie du geste musical." Glenn Gould Pluriel (1988): 85-111. Québec: Louise Courteau.
23. Ekman, Paul, and Wallace V. Friesen. 1972. "Hand movements." Journal of communication 22(4): 353-374.
24. Kurth, Ernst. 1922. Grundlagen des linearen Kontrapunkts. Berlin: M. Hesse.
25. Fatone, Gina A., Martin Clayton, Laura Leante, and Matt Rahaim. 2016. "Imagery, melody and gesture in cross-cultural perspective." In New perspectives on music and gesture, 203-220. Routledge, 2016.
26. Françoise, Jules, Baptiste Caramiaux, and Frédéric Bevilacqua. 2012. "A hierarchical approach for the design of gesture-to-sound mappings." In Proceedings of the 9th Sound and Music Computing Conference, 233-240. Copenhagen, Denmark.
27. Gibson, J. 1977. "The theory of affordances.” In Perceiving, Acting and Knowing: toward an ecological psychology. 67-82. Edited by R. Shaw & J. Bransford. Hillsdale, NJ: Lawrence Erlbaum.
28. Gibson, James. 1985. The ecological approach to visual perception. Boston: Houghton Mifflin.
29. Godøy, R. I. 2006. ”Gestural-Sonorous Objects: embodied extensions of Schaeffer's conceptual apparatus.” Organised Sound, 11(02): 149-157.
30. Godøy, Rolf Inge. 2009. "Geometry and effort in gestural renderings of musical sound." In Gesture Based Human-Computer Interaction and Simulation: 7th International Gesture Workshop, GW 2007, Lisbon, Portugal, May 23-25, 2007, Revised Selected Papers 7, 205-215. Springer Berlin Heidelberg, 2009.
31. Godøy, Rolf Inge, Egil Haga, and Alexander Refsum Jensenius. 2006. "Exploring music-related gestures by sound-tracing: A preliminary study." In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems. 27-33. Edited by K. Ng. Leeds, UK.
32. Godøy, Rolf Inge, Egil Haga, and Alexander Refsum Jensenius. 2005. "Playing “air instruments”: mimicry of sound-producing gestures by novices and experts." In International Gesture Workshop, 256-267. Berlin, Heidelberg: Springer Berlin Heidelberg.
33. Godøy, Rolf Inge, Alexander Refsum Jensenius, and Kristian Nymoen. 2010. "Chunking in music by coarticulation." Acta Acustica united with Acustica 96 (4): 690-700.
34. Godøy, Rolf Inge, Alexander Refsum Jensenius, Arve Voldsund, Kyrre Harald Glette, Mats Erling Høvin, Kristian Nymoen, Ståle Andreas van Dorp, Skogstad, and Jim Tørresen. 2012. "Classifying music-related actions." In Proceedings of the ICMPC-ESCOM 2012 Joint Conference: 12th International Conference on Music Perception and Cognition & the 8th Triennial Conference of the European Society for the Cognitive Sciences of Music, 352–357. Thessaloniki, Greece.
35. Godøy, Rolf Inge, Minho Song, Kristian Nymoen, Mari Romarheim Haugen, and Alexander Refsum Jensenius. 2016. "Exploring sound-motion similarity in musical experience." Journal of New Music Research 45(3): 210-222.
36. Hatten, Robert S. 2004. Interpreting musical gestures, topics, and tropes: Mozart, Beethoven, Schubert. Indiana University Press.
37. Henbing, Li, and Marc Leman. 2007. "A gesture-based typology of sliding-tones in guqin music." Journal of New Music Research 36 (2): 61-82.
38. Hogan, Neville, and Dagmar Sternad. 2007. "On rhythmic and discrete movements: reflections, definitions and implications for motor control." Experimental brain research (181): 13-30.
39. Hsieh, Chi-Min, and Annie Luciani. 2005. "Physically-based particle modeling for dance verbs." In Proceedings of the 15th International Conference on Computer Graphics and Applications, 85-92, 2005, Novosibirsk, Russia.
40. Jensenius, Alexander Refsum. 2007. "Action-sound: Developing methods and tools to study music related body movement." PhD diss. University of Oslo, 2007.
41. Jensenius, Alexander Refsum, and Marcelo M. Wanderley. 2010. "Musical gestures: Concepts and methods in research." In Musical gestures, 24-47. Routledge, 2010.
42. Krumhansl, Carol L., and Diana Lynn Schenck. 1997. "Can dance reflect the structural and expressive qualities of music? A perceptual experiment on Balanchine's choreography of Mozart's Divertimento No. 15." Musicae Scientiae 1(1): 63-85.
43. Kurtenbach, Gordon. 1990. "Gestures in human-computer communication." In The art of human-computer interface design. Edited by B. Laurel, 309-317. Addison-Wesley.
44. Leman, Marc, and Rolf Inge Godøy. 2010. "Why study musical gestures." In Musical gestures: Sound, movement, and meaning. Edited by M. Leman & R. I. Godøy, 3-11. New York: Routledge.
45. Küssner, Mats. 2014. "Shape, drawing and gesture: Cross-modal mappings of sound and music." PhD diss. University of London, 2014.
46. Leante, Laura. 2009. "The lotus and the king: imagery, gesture and meaning in a Hindustani rāg." In Ethnomusicology forum, vol. 18, no. 2, 185-206. Taylor & Francis Group.
47. Lerdahl, Fred, and Carol L. Krumhansl. 2007. "Modeling tonal tension." Music perception 24 (4): 329-366.
48. Luciani, Annie. 2004. "Dynamics as common criterion to enhance the sense of presence in virtual environments." In Proceedings of the 7th Annual International Workshop Presence 2004, 96-103. ISPR, 2004, New Orleans, Louisiana, USA.
49. McNeill, David. 1992. Hand and mind: What gestures reveal about thought. University of Chicago press.
50. McNeill, David, ed. 2000. Language and gesture. Vol. 2. Cambridge University Press.
51. McNeill, D. 2005. Gesture and thought. Chicago, IL, US.
52. Mion, Luca. 2003. "Application of Bayesian Networks to automatic recognition of expressive content of piano improvisations." In Proceedings of the SMAC03 Stockholm Music Acoustics Conference, 557-560. Stockholm, Sweden.
53. Moran, Nicola Stephanie. 2007. Measuring musical interaction: Analysing communication in embodied musical behaviour. Open University (United Kingdom), 2007.
54. Nelson, Winston L. 1983. "Physical principles for economies of skilled movements." Biological cybernetics (46): 135-147.
55. Norman, Donald A. 1988. The psychology of everyday things. Basic books, 1988.
56. Noyce, Genevieve L., and Peter Sollich. 2013. "Quantifying shapes: Mathematical techniques for analysing visual representations of sound and music." Empirical Musicology Review (2013): 128-154.
57. Nusseck, Manfred, and Marcelo M. Wanderley. 2009. "Music and motion—how music-related ancillary body movements contribute to the experience of music." Music Perception 26(4): 335-353.
58. O'Regan, J. Kevin, and Alva Noë. 2001. "Acting out our sensory experience." Behavioral and Brain Sciences 24(5): 1011-1021.
59. Pearson, Lara. 2013. "Gesture and the sonic event in Karnatak music." Empirical Musicology Review (2013): 2-14.
60. Pearson, Lara. 2016. "Gesture in Karnatak music: Pedagogy and musical structure in South India." PhD diss., Durham University, 2016.
61. Rahaim, Matthew. 2013. Musicking bodies: Gesture and voice in Hindustani music. Wesleyan University Press.
62. Rizzolatti, Giacomo, and Corrado Sinigaglia. 2008. Mirrors in the brain: How our minds share actions and emotions. Oxford University Press, USA.
63. Rocchesso, Davide, Guillaume Lemaitre, Patrick Susini, Sten Ternström, and Patrick Boussard. 2015. "Sketching sound with voice and gesture." Interactions 22(1): 38-41.
64. Runeson, Sverker, and Gunilla Frykholm. 1983. "Kinematic specification of dynamics as an informational basis for person-and-action perception: expectation, gender recognition, and deceptive intention." Journal of experimental psychology: general 112(4): 585.
65. Schaeffer, Pierre. 2016. A la recherche d'une musique concrète. Paris: Média Diffusion.
66. Schaeffer, Pierre. 2022. "Traité des objets musicaux." In Lexikon Schriften über Musik, 735- 744. Bärenreiter, Kassel, Germany.
67. Snyder, Bob. 2000. Music and memory: An introduction. MIT press.
68. Stern, Daniel N. 2010. Forms of vitality: Exploring dynamic experience in psychology, the arts, psychotherapy, and development. Oxford University Press, USA.
69. Suzuki, Kenji, and Shuji Hashimoto. 2004. "Robotic interface for embodied interaction via dance and musical performance." Proceedings of the IEEE 92(4): 656-671.
70. Thompson, Evan, and Francisco J. Varela. 2001. "Radical embodiment: neural dynamics and consciousness." Trends in cognitive sciences 5(10): 418-425.
71. Varela, Francisco J., Evan Thompson, and Eleanor Rosch. 1991. The embodied mind, revised edition: Cognitive science and human experience. MIT press.
72. Vines, B.W., Wanderley, M.M., Krumhansl, C.L., Nuzzo, R.L. and Levitin, D.J. 2004. “Performance gestures of musicians: what structural and emotional information do they convey?” In Gesture-Based Communication in Human-Computer Interaction: 5th International Gesture Workshop, GW 2003, Genova, Italy, April 15-17, 2003, Selected Revised Papers 5, 468-478. Springer Berlin Heidelberg.
73. Vines, Bradley W., Carol L. Krumhansl, Marcelo M. Wanderley, and Daniel J. Levitin. 2006. "Cross-modal interactions in the perception of musical performance." Cognition 101(1): 80-113.
74. Wanderley, Marcelo M. 2001. "Quantitative analysis of non-obvious performer gestures." In International Gesture Workshop, 241-253. Berlin, Heidelberg: Springer Berlin Heidelberg.
75. Wanderley, Marcelo M., and Philippe Depalle. 2004. "Gestural control of sound synthesis." Proceedings of the IEEE 92, no.4 (2004): 632-644.