P. Physical (senses) - Redefining Musicking Through Sight, Touch, Notation (Reading), Hearing and Movement
Physical sense dimension is the first of the seven analytical dimensions that make up the PACMMAN framework. It foregrounds the multisensory channels through which performers and audience experience a work or any musicking practice that incorporates computational media, sensors and interactive technologies that directly engage the human senses.
Core Analytical Insights
The DigiScore corpus analysis distilled a set of sense-specific analytical questions that map each channel onto a semiotic function. • Sight tracked visual grammars (e.g. colour-coded symbols, animated gestures, spatial layout) as notation, metaphor and timing cues. • Touch probed how material objects, haptic controllers and amplified gestures translate bodily force into sonic or visual output, recording pressure, resistance and texture. • Reading examined non-traditional notations (e.g. dynamic icons, mnemonic graphics, AR overlays) as memory aids rather than prescriptive scripts. • Hearing focused on timbre, spatialisation and process-driven texture as relational traces of gesture, algorithm and space. • Bio-movement investigated how human or machine embodied agency (e.g. gestures, breath, gaze, machine motion) shapes the unfolding narrative.
Across these five senses, three recurring themes emerged. 1) First, a mapping logic binds sensory symbols to musical parameters, for example, colour to mode (e.g. amber = major-7 tonality), shape to tension, spirals, pressure to dynamic intensity and gestural trajectories to harmonic direction. 2) Second, an immersive grammar emerged in which performers internalise visual, tactile and notational cues as an intuitive language, creating bidirectional feedback loops that close the gap between agency and material. 3) Third, each sense carries semiotic weight that can cue, constrain or amplify the others. For example, visual metaphors evoke affective states, material texture encodes narrative meaning, procedural notation encodes transformation rather than fixed pitches, process-derived timbre mirrors gesture and embodied movement generates inter-agency dynamics that co-construct meaning.
Sub-field analysis
In this section I expand on the five key themes listed above: sight, touch, notation (reading), hearing and movement & bio-feedback. In each subsection I present a short explanatory paragraph that analyses the corpus findings with a little more granularity followed by three key themes that emerged from the corpus. Visual material - “Seeing” meaning Traditionally, this sub-field would be the preserve of looking at sheets of paper, PDF images of scores, or graphic symbols anchored onto a fixed medium. In the DigiScore corpus visual material rarely served solely a notational device. For example, animated scores, colour-coded icons, three-dimensional projections and fluid-surface images acted as either notation, cue generators or metaphorical narratives, sometimes in unison. For instance, a flower icon that expanded and faded could signal a gradual harmonic shift; a spiral that tightened may indicate rising tension. This allowed performers who lacked a conventional conservatory background (e.g. due to disability, limited access, or differing cultural training) to participate more fully and because the visual logic was learned on-stage with the media, it functioned as reliably as notated instructions.
Key themes P1 Colour-to-mode and shape-to-gesture mapping P2 Real-time visual grammar that performers learned by immersion through performance P3 Narrative-driven metaphor that cued affective states
Tactile material - “Touching” the score
Several scores in the DigiScore corpus used a range of tactile interfaces. For example, haptic controllers, pressure-sensitive floors and wearables translated bodily forces into data streams that drove sound or visual output. The materials themselves “pushed back”, for example a stiff metal plate vibrates differently from a soft fabric, giving the performer a range of audible-tactile cues to explore. Amplified gestures (e.g. microphones attached to the hands) made the tactile contact audible, consequently exposing the causal chain to the audience and reinforcing the dialogic relationship between performer and medium.
In this sense, the tactile dimension can turn the score into a lived instrument: bidirectional feedback loops let performers feel resistance, receive vibration and see visual responses, creating a closed sensory circuit. A feature here is that embodied agency empowered musicians to co-author meaning through pressure, friction and timing. Additionally, material semiotics give substrates (e.g. water, rope, mycelium-dyed cloth) symbolic weight that shapes narrative.
Key themes P4 Bidirectional feedback loops (e.g. resistance <-> vibration <-> visual response) P5 Embodied agency: the performer co-authoring meaning through pressure, friction and timing P6 Material semiotics: the substrate (water, rope, mycelium-dyed cloth) carried symbolic and material weight
Notation (Reading) - “Reading” as memory & suggestion
Beyond reading traditional notation on screens that were either fixed or dynamically moving across a “play-head”, digital scores in the corpus behaved as guides that updated in real time. For example, hybrid scores mixed icons, animated glyphs and textual prompts; AR overlays could also appear on a tablet or headset as the piece progressed. Mnemonic graphics (colour-coded shapes, spirals) served as memory anchors that performers internalised before the live event, while on-the-fly updates collapsed the traditional boundary between reading and executing.
Key themes P7 Procedural notation that encoded processes (“grow”, “obliterate”) rather than of specifying fixed pitch or literal rhythm, shifting the focus from static data to dynamic transformation. Some scores did employ traditional notation, but we are not concentrating on that format here. P8 Negotiated meaning: performers co-constructed (or completed) the unfolding score, balancing freedom and constraint P9 Cognitive embedding: repeated exposure created a shared visual-language repertoire that can be summoned without a visual aid
Auditory material - “Hearing” relational information
In many of the case studies in the corpus, sound was treated as a trace of multimodal interaction rather than as a functional backing track. For example, real-time processing, live-coding environments (Max/MSP, Ableton) and field recordings blended with acoustic sources to produce hybrid soundscapes. In other pieces, a shifting icon triggered a spectral filter sweep; a pressure gesture launched a granular synthesis engine. In this auditory dimension sound can function as a relational trace of gesture. Here process-derived timbre revealed the physical act that produced it (e.g. friction yields grainy noise, breath creates airy shimmer). Additionally, spatial localisation reinforced visual metaphors and help build synesthetic maps. We also found that an auditory feedback loop lets performers hear the immediate consequence of their motion and adjust in real time.
Key themes P10 Process-derived timbre that revealed the type of gesture that generated it (e.g. friction -> grainy noise; breath -> airy shimmer) P11 Spatial narrative: localisation of sound reinforced visual metaphors, creating a synesthetic map P12 Auditory feedback loop: performers listened for the acoustic consequence of their gestures and adjusted continuously
Movement & bio-feedback - Embodiment and agency
For several digital scores in the corpus movement was a primary driver of musical intent. Examples include how human gestures (walking, breathing, facial expression) were mapped to algorithmic parameters, while machine agents (neural networks, generative robots) responded in turn, blurring the composer-performer boundary. Shared visual timelines (e.g. projected progress bars) give the ensemble a collective sense of time without a traditional conductor. We found that movement and bio-feedback turned gesture into a musical language: size, speed and trajectory encode semantic cues (e.g. rise == major shift, collapse == minor), while human gestures and algorithmic agents’ movement patterns formed a kinaesthetic approach to notation.
Key themes P13 Inter-agency dynamics: human gestures and algorithmic agents co-created meaning that neither could anticipate alone P14 Gesture as language: size, speed and trajectory of movement encoded semantic categories (e.g. “rise” -> major shift; “collapse” -> minor shift) P15 Embodied memory: repeated bodily patterns form a kinaesthetic archive that were drawn upon in improvisation
Take-away points
There are four key takeaways from the analysis of this dimension in action:
- Multisensory layering - each sense can contribute a semiotic layer that can cue, constrain or amplify the others, or operate in isolation.
- Distributed agency - meaning can be co-produced by the body, the material (hardware, surface, projection) and the code that mediates them.
- Pedagogical potential - visual-musical mappings and tactile feedback lower barriers for learners who lack traditional conservatory training.
- Research implications - any analysis of contemporary digital performances should treat the five senses as separable but interlinked components of a single ecological system.