Type: 8 - Gesamtkomposition this uses computers to synchronise multiple streams of media into a cohesive work with live performers (human or machine). It coordinates these through a central software environment that is embedded with autonomous or generative behaviour. This actively composes the work in real-time and evokes the presence of the composer in the flow of musicking.

About

Nautilus (2022) is an intermedia composition for bass flute, neural network and Unity game environment. It was created by Craig Vear, Adam Stephenson (games developer) and Carla Rees (performer). The piece was performed by Carla and a follow-on performance by Franziska Baumann. On an artistic level Nautilus is inspired by an imaginary deep-sea journey of a nautilus mollusc, as it navigates deep water trenches across the oceans. The music describes this journey with the bass flute and generative sound-design highlighting the topography of the oceans and vast openness of the depths. On a technical level, Nautilus used a Unity game engine as the main platform for the visual elements of the digital score, which included sea-bed imagery and sinking notes written on staves tied to anchors. The compositional process involved training a neural network on an initial improvisation by Carla Rees. This improvisation attempted to capture an interpretation of the artistic vision of the piece. It was then used to a) define note choices by training a neural network and b) provide short samples of sound design. In the final version, the Unity engine listens to Carla and makes judgments about what and when to generate a sound design, the behaviour and choices of the neural network was consolidated into the Unity programming as symbolic probabilities.

Further details can be found here LINK

Analysis (using the 7 modalities - LINK)

Interface Object

The music idea embedded in Nautilus is immediately communicated through the visual sense, specifically using a Unity game engine as a platform. Listening is always present in music-making, and a certain amount of supporting/ secondary material is communicated here by this digital score. The immediate phenomenon of experience evoked by this digital score is visual.

Material Affect

The Unity game engine is primarily responsible for the affectual connections with the musicians. It reaches out and draws in relationships through the following aspects, and in doing so shifts the “sense ratios” and “patterns of perception steadily and without any resistance” (McLuhan, 1964):

  • The nature of a game engine evokes an immediate sense of play, it suggests a mode of engagement that is ludic and playful. The quality of the graphics, the design of assets and feel of camera animation all reinforce this. Even though Carla and Franziska were not gamers, they are aware of gaming and its modes of engagement. Furthermore, the designers (Craig and Adam) are gamers and have been for many years (Craig for over 40 years!) and so their experiences have been embedded into their visual design of the design mechanics of this digital score.
  • The immediate experience of being drawn into this platform was one of exploration. It was an open world design, and this was perceived by both performers. This structured a way of thinking and an approach to their interpretations. This was a core intention of the designers.
  • The behaviour of the “computer layer” as commanding and ludic was also an apparent quality that shifted the immediate experiences of this digital score. The scene moved and was interactive, the logic offered materials for the performer to be inspired by the sound design was responsive to the here-and-now of the improvised interpretation.
  • Because of the development process between Carla and the team, memories of the process were embedded into her experience: the decision to make, say, the note list fixed and without a clef were part of Carla’s immediate engagement with the score, but not Franziska’s.
  • The moving camera, that appeared responsive to sound, and its general quality of movement (structured by underwater physics) also reinforced the immediate experience of this digital score.
  • The sound design enhanced this immediate visual world. It was designed to emanate from within this world, and was implemented in such a way that it prioritised the visual material by adding additional context. This, using Chion notion of the audio-visual contract – reinforced the patterns of perception rather than confusing them, and enhanced the “cultural layer” of the music-world.

Goal

The goal of this digital score is two-fold: i. To paint a sound picture of this underwater world. This is free, open and flowing and was explained in detail through the development process, in the “performers instructions” and as an artistic statement. There were also 3 sections of contrasting behaviours and 3 transitional sections. ii. To avoid, at all costs, engaging solely with the camera movement interactivity. The movement of the camera was not the goal of the score, but as a secondary concern. If the performer were to focus solely on making it move, then it would have entered into the world of game-play. But the instructions to the performer were clearly pointing to remaining in a music-play world.

Content

The primary signals that convey meaning in Nautilus were:

  • Falling notes on staves, their intensity of release and their spatial distribution, and the flashing lights that represented their anchors
  • Kelp
  • The sea-world design of sand and waves, and the floating specs moving about in the sea body
  • The quality of the camera movement, and its physics
  • The sound design anchoring (pardon the pun) the harmonic movement of the piece and its textural qualities

Language

There were three modes of interpretation present in Nautilus:

  • Iconic: the music notes on clef-less staffs offered some suggestion of specific note choices to be played. This was not didactic, and more an invitation to play, or stop playing.
  • Sensory: the kelp and particles from inside the sea-world, and the pulsing anchor lights offered interpretative suggestions about what could be played. The Kelp was interpreted as a big gesture especially when the camera went through it, the particles suggested stochastic and rhythmic clusters, and the pulsing anchors operated as rhythmical conductors. Furthermore, the tempo of the game mechanics suggested a slow feel, and the spaciousness of the first part of the piece suggested a quiet dynamic. These all changed during the different phases of the composition.
  • Arbitrary – The sound design worked in collaboration with the visual material to further control the decisions by the performers. This kept the performers within a narrow-range of decisions which served two purposes: first, conatin the improvisatory play within a fixed sound and harmic world, and second, to support the dramaturgical narrative construction of the performers. In short, a little bit of control (by the digital score) offered the musicians a degree of confidence that they were moving through the composition-world in an appropriate way, which enabled them to play more freely with the materials.

Feedback

The feedback mechanism for Nautilus is metaphorical and relies on the performers constructing and adhering to, a subjective narrative that is refreshed in realtime by the visual materials and the sound design.

Flow

The flow of Nautilus was explicitly controlled by the Unity game engine. The sequencing of experience and the ball-park territory of what could, or should, be played at any given point, was presented as options by the engine as a combination of visual materials reinforced by the sound design. Elements of games mechanics – sequencing, dynamics, tempos, feel, density etc – were structured through the duration so as to provide the performers with clear territories with which to navigate the piece. A further analysis can be found HERE