Type: 10 - The Living Score is characterized by the use of intelligent computation systems as a co-creational other inside the flow of musicking. There are several approaches to the implementation of computational intelligence within the digital score with each offering differing degrees of felt co-operation, presence and autonomy.

Demo https://www.youtube.com/watch?v=7dQKIpjKJu4

About

Jess+ is an intelligent digital score system for shared creativity with a mixed ensemble of able-bodied and disabled musicians. The digital score uses AI and a robotic arm to enhance the real-time dynamic engagement of the disabled musician with the able-bodied musicians allowing her to thrive in a realtime communication beyond what has so far been possible. AI and robotic technologies are used to extend the creativity of disabled musicians. In a realtime music feedback loop, a disabled musician (A) is “plugged into” a creative AI robot arm that draws graphic music notation which is performed live by other musicians (B). The creative AI and the disabled musician work as an extended system, with each feeding the other: the AI reads the real-time brainwaves and arousal data of the human (A), and (A) feels the embodied behaviour of the robot arm. The feed-back loop is closed as the live musicians (B) make sound, which in turn is heard by both the AI and humans (A+B). In this system, the AI and the robot arm is not an assistive tool, but rather co-creative other. This has flattened any hierarchy through movement and enhanced the sense of togetherness and inclusivity in musicking.

Analysis (using the 7 modalities - LINK)

Interface Object

The music idea embedded in Jess+ is immediately communicated through the visual sense, specifically through affectual, empathetic and symbolic dance movement of the robotic arm. Listening is always present in music-making, and a certain amount of supporting/ secondary material is communicated here by this digital score. The immediate phenomenon of experience evoked by this digital score is visual.

Material Affect

The sense of being part of a dance between the whole ensemble (including the AI/ robot arm) is primarily responsible for the affectual connections with the musicians. This collective inter-connectivity reaches out and draws in relationships through the following aspects, and in doing so shifts the “sense ratios” and “patterns of perception steadily and without any resistance” (McLuhan, 1964):

  • 3-D movement of the robot arm (inducing affectual, empathetic and symbolic responses)
  • agental persona of the robot arm (a sort of positive and affirming anthropomorphisation welcoming the system into the ensemble as an equal)
  • memories and trust embedded into the behaviour of the AI through long-term agile dev process
  • autonomus and ensemble behaviours chosen by the AI

Goal

The goal of this digital score is to freely interpret the movement of the robot arm as affectual, empathetic and symbolic instructions with which to respond with a sound. There are two core inspiration points that underpin such responses which form part of a background context:

  • first, is the performance practice of interpreting Cardew’s graphic score Treatise (1963-67)
  • second, the performance practice of interpreting Wolff’s For 1, 2, or 3 People (1964)

As such, the musicians must enter into a metaphorical dance with the ongoing sound they are making as a collective and being guided by the gestures of the robot arm. The goal is to allow its movements to stimulate a meaningful musical response within the here-and-now of musicking.

Content

The content of Jess+ is:

  • fixed library pre-defined movement gestures
  • the balance of behaviour symbolically contained in the AI, between predictable and unpredictable responses
  • the speed and acceleration of the robotic arm movements
  • the relationships between previous movements and perceived cohesiveness of gestures
  • inked marks on a page (if using pen-mode)

Language

There are two modes of interpretation in Jess+:

  • Iconic - through the fixed gestures developed and embedded into the AI code through agile development and informed by the two inspirations points (see goal)
  • Sensory - through empathetic and affectual responses to gestures that could be interpreted as either a musical-conductor (such as those of lead violinist as they coordinate a string quartet), or as a dance-improvisor (responding in realtime to the music, and inspiring musical reactions)

Feedback

Feedback is generated in the perceived listening behaviour of the AI and is the primary connection between humans and the AI score. This is not a didactic relationship of say an obedient dog, or jumping insect when confronted with a sound, but a balance between perceived connections and autonomy. This balance is critical in maintaining the connections of trust and engagement, and also offers elements of surprise, mischief and novelty. If this was felt to be too close-coupled then the musicians became bored by the predictability or directness of responses; too free and the bonds of togetherness were weakened and the shared sense of the dance were lost.

Flow

The human musicians’ flow in this digital score is within the inter-connected dance between them and the robotic-arm. This is maintained by the movements of the robotic arm and the balance between autonomy and responsiveness. At times the robot-arm appears to be listening and responding, but at others it appears to be doing it’s own thing in parrallel to what else is happening (like humans can do when they improvise). This latter response is a form of concurrent creativity (like kids playing together, but on individual tasks). The balance between explicit responsiveness and related concurrency is finely tuned, with the ultimate aim of cementing the bonds of togetherness and reinforcing the shared sense of inter-connectiveness of the whole extended ensemble.