The Digital Score – The Medium and its Message

Challenge: with mixed media and active computation underpinning many digital scores, how do we discuss (or create with) these agents and objects and how do they shape meaning?

Key Insights:

  1. the ‘content’ of a digital score is not the same as its ‘message’
  2. The ‘message’ should be understood through the way that a digital score shapes the scale and form of human association and action (i.e. how it makes us behave, feel, think, create)
  3. digital score creativity should consider its message rather than focussing solely on the content, as deep meaning is communicated here
  4. the media used to construct a digital score are in themselves individual media with independent ‘messages’ and ‘contents’
  5. organising and interpreting the inter-related media in a digital score could consider them as objects or agents with mass, gravitational pull, trajectories, radiations and distortions as they operate through time

At the core of this Digital Score framework are three foundational theories that underpin and support the approach I have taken so far:

  1. Musicking, by Christopher Small, in which music is discussed and understood from inside the creative acts of music-making
  2. Phenomenology – that it is the “I” that is at the centre of my experience inside musicking
  3. The Medium is the Message, by Marshal McLuhan which outlines the different affects of media that coalesce and combine to make the stuff of digital scores to shape and control perceptions.

I have introduced the first 2 theories in previous posts and in The Digital Score (Vear 2019) book. However, the latter theory is important as it helps musicians understand where meaning-making occurs in digital scores and the weights, gravities, pulls, and distortions of the individual media that are combined into a digital score.

A primary principle to McLuhan’s theory is that a medium is a message that can be easily grasped by the user. It is tempting to view the message of say traditional music score as being wrapped up in the content of the page, that the lines and dots of the notation signal to the user (the performing musician) what sequence of notes and textures to play as specified in advance by the originating musician (a “composer”) using an agreed graphical language. But for McLuhan the true message of medium is not its content. He says that it is very easy to get swayed by the content of a medium when it is the medium itself that shapes and controls “the scale and form of human association and action”.

Let’s look at traditional music score from this perspective: the content of this medium is the lines and dots on the page. McLuhan proposed that the “content of any medium is always another medium”– thus, speech is the content of writing, writing is the content of print, and print itself is the content of the telegraph. McLuhan describes the content of a medium as a “juicy piece of meat carried by the burglar to distract the watchdog of the mind”. So, the content of traditional music score – the lines and dots – is another medium and message which is processed in the mind of the musician bringing forth semantic interpretations of the meaning of the symbols on the page.

So what is the message of the traditional music score as a medium? Content aside, the material page populated with its lines and dots (as opposed to a different type of page such as a newspaper) can signal to the musician a way of behaving, a way of thinking, an attitude; it can raise memories of previous experience, of professional approaches, of fear and nerves, of potentials, journeys explorations, musical experiences. In short, the presence of the score shapes and controls “the scale and form of human association and action”.

In another post, Solomiya discusses her piece artifacts of presence with the musician Dejana Sekulic. In this discussion she highlights the importance of the sensor in Dejana’s meaning-making with her piece. From McLuhan’s perspective, the sensor is a medium, and the content of this medium is data streams and connectivity to the host computer. In turn, the content of that stream is the control of parameters, which begets interactive behaviours etc. But the message of the sensor is not the same as the content. The message (I provocatively suggest) is the physical presence of the digital score system holding onto Dejana’s bow. It signals to Dejana a direct physical, material connection with the system, if not with the presence of Solomiya as the composer of the piece. The weight difference she feels through her bow, constantly highlights this connection, and even when the sensor is not operating, this presence remains; as does the sense that its contents (the data streams) could further connect them at any point, when this part of the system is activated.

Hot and Cool media

As an extension to this theory, it is worth mentioning McLuhan’s differentiation between hot media and a cool media. A distinction made by McLuhan between media such as print, photographs, radio, and movies (hot media) and media such as speech, cartoons, the telephone, and 1960’s television (cool media).

Hot media usually, but not always, provides complete involvement without considerable stimulus.  Hot media are ‘high definition’ because they are rich in sensory data. They generally emphasize one sense (for example, of sight or sound) over the others. For example, print occupies visual space, uses visual senses, but can immerse its reader. Hot media also include radio, film, and photography as it draws the mind of the user into a relationship/ connection and suggests/ states the meaning that is to be formulated through this. For this reason, the traditional music score is a relatively hot medium.

Cool media, on the other hand, are usually, but not always, those that provide little involvement with substantial stimulus. Cool media are ‘low definition’ because they provide less sensory data and consequently demand more participation or ‘completion’ by the audience. Therefore, according to McLuhan cool media includes music, television (although television has grown hotter since the 1960s as the technical picture and Smart functionality quality has improved), as well as cartoons. Dejana’s bow sensor is a very cool medium.

In 2019 I was invited to discuss my adoption of McLuhan’s media theory in my own digital score research through a case study. This chapter appears in Crossley, M (ed.) (2019) Intermedial Theatre – Principles and Practice. Here is an extract that discusses the framework for understanding the operationality and interactivity of different media with the digital score Postcards (2017). It was purposefully poetic and metaphorical in order to communicate complex ideas across the disciplines.

Poetics of Media affect

It’s operational dynamics coupled with the sensorial affect upon the listening minds of the performer and the audience are the consequential intermedial relativities of their properties and movements through space and time, and can be defined by the following poetics:

  1. Dimensional media-space (DMS) – this is the realm that the intermedia composition constructs through the relationships and connections across, within, between and emergent from the media-system library and determine the identity of each composition. In Postcards the DMS can be considered as the wholeness[1] of the piece, or the meta-narratives, dramaturgies and mediaturgies that are formed from the coalescence of the disparate elements including the live performers. The DMS is a poetic realm of potential: the potential for one media to interact or coalesce with another. As such, determining or developing the following factors are essential to defining the feel and sensation of the overall DMS and therefore the feel of the Gesamtkomposition in performance:
    1. Density – each media element will have density that can be understood in terms of its affectual power. For example, the words of Flaubert’s letters to Sands naturally has more concrete semiotic value than a sound effect from the Kyma system, because they are words. Written on the screen or spoken by La Berge will draw the mind into a pre-formed world through language. As such, it has the potential to pack more punch, to draw attention to itself and evoke more “meaning”.
    1. Mass – mass is different from density: density is about multiplicity and layers of indexical meaning, whereas mass is about pure affective weight. The word ‘Love’ compared to the word ‘sometimes’ has a different semiotic punch. The written word ‘Love’ and the spoken world ‘Love’ again have different masses, as does the sung word ‘Love’.
    1. Gravities – the gravity of a media element relates to its potential to draw attention to itself, and to effect a change in the ongoing spaciotemporal[2] signatures of affect that the performance is evoking. A distorted loud sound effect on the live voice smashing into the mix, for example, would immediately disturb the affectual balance of the ongoing performance, and draw attention to its qualities and our minds understanding of what-is-going-on.
  2. Operational dynamics – during performance media elements converge, interrupt, coalesce and relate. Time is a significant factor in this section of the poetics as it deals with relativity and relationships through experience and across time. The following factors help us define the relative relationships of densities, mass and gravities across time, and their coalescence in experience:
    1. Orbits – a media element in orbit is clearly within the realm of actual experience through performance. Its potentials within the defined DMS has become manifest and it is operational with a phenomenology of experience for the audience and performer(s) and the intermedial relationships with other media elements. For example a melodic line on the flute in combination with the words ‘Paris’ projected on the large projection screen are going to communicate something to the audience and the performer and take their minds somewhere. These elements may be coalescing into other thought-trains or generating individual responses with mind-images. If the flute melody occurred first then this might establish a specific response, modified later by the inter-orbit relativity between the emerging word ‘Paris’.
    1. Trajectories – as one media element relates to another, meaning is changed and is dependent on their trajectories over time. When a new media element emerges into the performance, it signifies the start of its trajectory. Does it converge with the existing intermedial fusion? Does it draw attention away from this fusion due to its inherent mass and gravity? Is its trajectory an outer universe orbit, remaining small and quiet, but present enough to make some small changes to perception?
    1. Attractions and repellents – When media elements are present in the dimensional space of the live, their relationships across such space, time and perception have the effect of drawing each other in, or repelling each other. The orbit, gravity, mass and densities of each elements help to define which are attracted/ repelled to what. For example, a small sound effect in Kyma and an equally small vocal utterance might be drawn together in a harmonious fusion of texture. If this Kyma sound grows in intensity – loudness, density of timbre, harmonic distortion for example – it will start to repel the small live sound and gain a significance of its own. If at this point a collage of images is also presented on the screen with qualities more alike the small live vocal sound, these will attract the live voice and repel the distorted sound. Attractions and repellents alert the mind to perceptual harmonic distance between media elements and it’s emergent affectual space
    1. Radiation and distortion – like attraction and repellents these factors are emergent from the affectual space between media. Sounding media elements can emit radiation independent of its attraction or repellent to other media. This emission can colour meaning-making of the intermedial fusion between media elements. Similarly is can distort our perception of such relationships. Radiation and distortion are coefficient with all the other parameters of this framework, and across time. They are unstable and may be quantum, in the sense that they can be both at the same time to other discreet media. Significantly they are fleeting in existence.

[1] Discussed in detail in Vear (2014) adopting a phenomenological stance that acknowledges how a the physical, computational, performative, interactive and aesthetic qualities of a work amalgamate with an understanding that ‘thought, imagination, memory and fantasy coalesce with the ‘polyphony of the senses’ and articulates our ‘sense of being in the world’ that, existentially, strengthens our ‘experience of self’ (Pallasmaa 2005, 41). As such, it is the sense of self that is taken to the worlds within each of these compositions, placing I at the centre of this experience.’ (Vear 2014)

[2] A term adopted from Elleström (2010) that ‘covers the structuring of the sensorial perception of sense-data of the material interface into experiences and conceptions of space and time’ (2010, 18)