Digital Syzygies: Digital Score as a shared creative platform for d/Deaf and autistic musicians

Developing a Digital Score platform for shared creativity with d/Deaf and autistic musicians, that unites creativity and collective involvement in the development of a musical idea. This Digital Score used EEG headsets to transmit musicians’ brainwaves into a set of neural-controlled musical compositions and performances.

The Issue

Traditional methods of communicating ideas in music tend towards the composer-performer model, where one musician (the composer) instigates an idea, captures it as notation, then passes the score onto other musicians to realise (the performers).

But this model assumes many “norms” and biases about the nature of the musicians involved, the music they wish to create, and the way in which they wish to work. In many cases it can exclude musicians wishing to engage in a shared creative pursuit of an idea, or wish to work remotely, or are untrained in the conservatoire model, wish to improvise, or are d/Deaf or neurodivergent or both. In all these instances, the traditional score model establishes a creative hierarchy with the composer at the top delivering the score, ending in some sort of public performance. As such it can exclude collective ideation and compartmentalise musicians’ creativity to specific tasks. Additionally, it can assume an involvement model based on “normal” definitions of musicking, hearing, creating, and communicating.

The central research question with this project was: can a Digital Score provide a platform in which a quartet of d/Deaf and autistic musicians could work creativity and challenge the traditional composer-performer model towards a way of working that was more inclusive?

The Research

The Digital Score project (DigiScore) worked in collaboration with Prof Andrew Hugill (University of Leicester) to build a practice-based case study (Digital Syzygies) that addressed the central concerns introduced above. DigiScore’s principal investigator Prof Craig Vear’s involvement was 2-fold: 1) providing the theoretical and conceptual guidance offered to Hugill, which supported his creativity and framing the conceptual development of Digital Syzygies. Second, Vear designed, developed and deployed a technical solution that facilitates the translation of brainwaves into music, supported shared creativity for the Digital Syzygies team, while embedding the definitions and concepts of DigiScore.

Digital Syzygies used the Emotiv Insight EEG Headset Brain Interface to connect four musicians who were remotely located (in Sweden, Sri Lanka/London, Brighton and Leicester). This technology was defined by Vear and Hugill as a novel solution with which to bind the quartet who are either autistic or are d/Deaf (Hugill is both), and as an inclusive mechanism with which to support a shared creative pursuit.

Over a 6-month period of experimentation, ongoing concepts and theoretical frameworks developed by the DigiScore project were used to develop a set of neural-controlled musical compositions and performances; these were entitled Digital Syzygies. Through this time Hugill and Vear advanced a conversation about the nature of diversity and inclusivity in digital musicianship, initiated by the observations of Hugill to a recorded interview between Vear and Leman ( https://www.youtube.com/watch?v=PUIbr16p39Q ), which ultimately guided the proceeding stages of this research.

To facilitate the translation of raw brainwaves from the EEG headset into the Digital Syzygies compositions Vear developed a software environment that provided a creative platform for the Digital Syzygies team to explore. It linked the brainwave output from the EEG headset to their defined aesthetic design and supported the further development of their creative ideas. This was Beta-tested by Hugill and Vear and eventually packaged as a Python library and as a standalone app. Both can be accessed at https://github.com/DigiScore/digital_syzygies

This Brainwave-to-Music app allowed the brains of the musicians to dictate the course of musical events, subject to responses measured by six EEG performance metrics: engagement, excitement, focus, interest, relaxation, stress. It also allowed individual musicians to ‘plug’ into a composition using the brain reader, and in realtime control libraries of self-created sound files and visual material into music. This could then be either recorded and shared amongst the team, released as an album, or presented in a live performance environment with an instrumentalist reacting to the soundscape and visual material.

The Outcomes

This project resulted in some transformational revelations which emerged on several levels: technical, musical and personal.

Vear’s Brainwave-to-Music app supported and framed a digital score that built connections or “syzygies” between people with neurodivergent brains and hearing differences. This app was embedded with the core principles of his DigiScore project ensuring that it supported the Digital Syzygies team’s development of the project. It was also designed to be a creative sandbox with which co-located musicians could share ideas.

This app supported Hugill to design a case study using the digital score format that enabled musicians to connect and see similarities in the way their neurodivergent brains react and interact with sounds. It also enabled those who are d/Deaf to relate to each other’s sounds and hearing peculiarities. Overall, musicking with the help of this digital score opened many new ways to making and listening to music for the musicians of Digital Syzygies. On a technical level it united d/Deaf or neurodivergent musicians in a creative space and way of working by allowing them to generate, develop and share musical ideas packaged in a novel digital and technical solution (the core definition of a digital score Vear (2019)).

For Hugill, the digital score made him unlearn everything he has previously learned in making music as an autistic musician with hearing limitations. Such things as listening to one’s instrument, listening precisely and accurately to what you’re doing, and being aware of others around you are not needed from him when he makes music using this digital score. Using the digital score also allowed him to not have to listen to sounds for long periods to make and perform music since he usually experiences pain in doing so. The digital score accessed via the brainwave reader does all the performing when it receives the sounds, Hugill can also just wear earplugs and let the brain do all the work: “I’m seeing this self-actualising brain that is making interesting music regardless of me, so it’s kind of the opposite of conventional music making” (Hugill 2022). This is inevitably a new musical skill for him which he hopes to use more in his music-making from now on. 

Composer/performer Simon Allen thinks that the digital score helps in facilitating problems of communication in making music and it’s an expansion of the techniques like indeterminacy. The composer and the performer are the same in this situation, thus it also provides some levelling between these two roles. This kind of technology can also remove bars to participation as one would not need to be trained in the Western classical tradition to make music. It also opens a conversation about where the score is, and what is the impetus for the score. Potentially many more people could come to terms with understanding this question of the musical score by interacting with a digital score interfaced by an EEG brainwave reader. 

For Anya Ustaszewski, composer/sound artist, interacting with this digital score allowed her to see the possibilities of what would be possible for people who are not even neurodivergent but come from different musical or not musical backgrounds. It opens the possibility of becoming a music creator with little prior musical training. In addition, Anya is interested to know how would people with other psychological disorders would react to music through the brainwave reader, how would the reader track their experiences? Questions she is interested to take to future research projects. 

For Elisabeth Winklander, a classically trained cellist, the project allowed her to see how other neurodivergent people like herself work in a music project focused on their similarities and differences that are harnessed in the strength-based approach: 

We have never been able to take that approach when working before, so we have an opportunity to learn how we work best while consciously exploring that part of ourselves in a context like this. (Winklander 2022)

On a personal level this digital score project had a profound effect upon Hugill:

‘’The Digital Syzygies project has led me to explore a new digital scoring system that effectively allows me to perform in public again. Since my hearing was destroyed by Meniere’s Disease in 2009, I have been unable to perform because the act of listening to music for more than a few minutes becomes painful. Using brainwave readers to compose and control the music means that I can perform without having to be able to hear, or indeed to use any observable physical movement at all. Instead, the neural environment becomes a way of communicating with other autistic and/or D/deaf musicians. By combining our neurological responses, we can create a music that expresses the syzygies (unexpected alignments) of our mental processes through sound. This system offers the potential for a transformational career impact in terms of live music-making and performance’’.

Participants
Andrew Hugill, who is both autistic and has severe hearing loss
Anya Ustaszewski, an autistic composer and musician who works with a variety of sounds and instruments and has a particular interest in immersive and abstract narrative sound experiences
Elisabeth Wiklander, a classical cellist with the London Philharmonic Orchestra who is also a Cultural Ambassador of the National Autistic Society
Simon Allen, a composer and percussionist, who experienced a slow change in his hearing up until 1993(?) which has brought 50% loss from the top down in terms of area on the audiogram, accompanied by hyperacusis, tinnitus and two hearing aids.