Intermedia composition for bass flute, neural network and Unity game environment (2022)

Links

Download the digital score for Nautilus (Windows and Mac) https://figshare.dmu.ac.uk/articles/software/Nautilus_digital_score_Win_Mac_/19762120

Complete Dataset can be found https://figshare.dmu.ac.uk/articles/dataset/Nautilus_digiscore_case_study_dataset_2022/20418447

Performance of Nautilus by Carla Rees https://www.youtube.com/watch?v=XK-9eXCJxCg&t=2s

Performance of Nautilus by Franziska Baumann https://www.youtube.com/watch?v=SV6TqzJkiX4&t=3s

About Nautilus

Nautilus (2022) is an intermedial composition that brought together three practitioners to explore and contribute to the project from different perspectives – composer, digital score researcher and project PI Craig Vear, bass flute player Carla Rees and Unity programmer Adam Stephenson. Each brought their own unique experience and creative practice to the piece, driving both the aesthetic and practical considerations of the work. They shared a core goal: “to create a digital score that supported and enhanced Carla’s sense of musicking to a point that felt like it was operating with her in the making of the music”. In a partner article to this one, we describe Nautilus as:

On an artistic level Nautilus is inspired by an imaginary deep-sea journey of a nautilus mollusc, as it navigates deep water trenches across the oceans. The music describes this journey with the bass flute and generative sound-design highlighting the topography of the oceans and vast openness of the depths. On a technical level, Nautilus used a Unity game engine as the main platform for the visual elements of the digital score, which included sea-bed imagery and sinking notes written on staves tied to anchors (see fig. 1). The Unity engine also listens to Carla and makes judgments about what and when to generate a sound design. 

On a technical level, this piece was created using Deep Learning processes, and in performance uses a neural net to make in-the-flow decisions about how the music is to be shaped. The compositional process started with an improvisation by Carla on the idea of the nautilus’ journey (the core aesthetic of the piece). This improvisation then became the source material for machine learning processes and the sound-design manipulation that is heard during the performance. A neural network was trained using TensorFlow methods and a dataset of transcribed jazz improvisations. At the start of each iteration of the piece random notes from Carla’s original improv are passed through this neural net that in turn outputs a notated improvisation based on the input note choices. This notation forms part of the digital score for live interpretation. Another element of the digital score is the generative sound design, which uses the audio recording of the original improvisation as its source material, and responds to the live sound as a stimulus. The final part is the game environment made in Unity which creates an immersive world for the audience and musician to inhabit through the piece.

Design Considerations

Unique research proposition: the main participant (Carla Rees) is an expert in low flutes and experimental music. Developing a digital score around a gaming environment and artificial intelligence would be a first for her. It is therefore possible to see insights into how these technical elements and the iterative devising process, shift her musicianship and offer potential transformations for her creativity.

Digital Score type:  Gesamtkomposition. This type of digital score ‘uses computers to synchronise multiple streams of media into a cohesive work with live performers (human or machine). It coordinates these through a central software environment that is embedded with autonomous or generative behaviour. This actively composes the work in real-time and evokes the presence of the composer in the flow of musicking’. (Vear 2019)

Technical infrastructure: Nautilus was created using: artificial intelligence, machine learning, and the Unity gaming engine.

Creation mode: this involved iterative and agile design processes that engaged all creative practitioners through the process.

Performance mode: the realisation of this digital score relies on improvised practices and involves experimental music language.

Accessibility/ inclusivity: this is an important consideration of the DigiScore project, and Nautilus was developed in such a way that it is accessible and friendly to the low flutes community, most of whom have no, or very little engagement with the above design considerations.

Core challenge

The challenges mentioned above centre on specific questions, but the overall challenge is to create a musicking experience that binds together, rather than is created from individual elements. The least successful version of this mixed-media approach would be to construct an experience for Carla that feels like a “bit of this”, stuck together with a “bit of that”. This unifying desire is immensely difficult as the creative musicians (coder, composer, performer), must work together to build something that has a purity and singularity of “message”, draws together and enhances the communicative value of a digital score, and has a unified aesthetic. This requires us to embrace a trans-disciplinary approach where we seek to find new, common principles and factors that contribute to a wholeness of experience, and this will normally go beyond/ distort/ transform/ enhance/ transcend our own training and ways-of-thinking.

Research Questions

  • can a games engine such as Unity be integrated into a digital score without detracting from the flow of musicking experience?
  • how can a neural network be trained with an aesthetic design to generate a series of digital scores?
  • how do we publish such a digital score so that others may engage with this composition?
  • how can narratives structures from game design be used to enhance the experience of a musician with a digital score?
  • what are the limitations of the dataset and neural net training? Does the jazz dataset shape/ distrort the aesthetic values or communicative potential of this digital score?
  • performer agency in the interpretation of the score – how can this approach develop or enhance (or restrict) that further? and what are the challenges that arise from this?

Critical Insights

  • It is truly beneficial to the creative product if the “composers” work with the musicians throughout the creation and development phases of the digital score. As evidenced in Nautilus, the digital score is embedded with a sense of playability that is felt by other performers of the work.
  • As an extension of the above, a major consideration is the musicians’, and audiences’,  immersion in platform and all its media elements. We are naturally attracted to games, and this is not limited to younger generations.
  • There is a need to get all the media elements to work as a cohesive whole. An understanding of the balance of intermediality is essential in this aspect and we would recommend that musicians do some research on it. We recommend Crossley, M., (2019) Intermedial Theatre: Principles and Practice as a start.
  • Given the openness of interpretation, there should be a degree of trust with the realising musicians, and an invitation by the digital score to share the agency of creation.
  • Traditional roles of “composer” and “performer” are blurred. Letting go of these, and operating more like a creative team was essential in Nautilus.
  • In the videogame environment, the aspects of form can now be considered from the perspective of “Journey” and “flow”. In Nautilus, these were also handled from the perspective of the dramaturgy of experience.
  • The Unity engine needs to be considered not as a gaming platform, but as a musicking platform. For example, conversations and understandings about dynamics, notation, pitch, harmony, tempo, involvement, play, narrative, design, assets, glyphs etc, were given new meanings in the context of Nautilus.

Conclusion

Through this case study we have begun to explore Unity’s potential as a novel and flexible driver for the creation of a musical score. Traditional elements of compositional design, such as structure, narrative, mood, atmosphere, are presented to a performer for interpretation and communication. They engage with the materials using musical, technical and interpretative skills and have agency in the development of their decoding of the visual cues, depending on their instrument, personal skills and musical aesthetic. An accompanying audio soundtrack also contributes to the musical direction of the work, enabling the performer to interact with sonic as well as visual cues to help them develop their individual approach to the performance. This project has provided us with a springboard for potential future development, opening up the possibility of interactivity with other performers.

Personnel

Carla Rees – Bass Flute, performer https://www.carlarees.co.uk/

Adam Stephenson – Unity developer https://www.adam-stephenson.co.uk/

Craig Vear – Python programmer, composer