Jess+: Intelligent DigiScore as a creative platform for inclusive music-making for disabled and non-disabled musicians.
The *Jess+ team at the BBC for a performance as part of the Bridging AI Divides (BRAID) event (L-R Deidre Bencsik, Craig Vear, Jess Fisher, Clare Bhabra)*
Description
Jess+ is an intelligent digital score that extended the creativity of disabled and non-disabled musicians within an inclusive music ensemble. The digital score used AI and a robotic arm to bind these musicians in a shared creative improvisation with all the musicians benefiting, practices enhanced and relationships transformed.
The project asks how a digital score generated by an AI‑driven robotic arm (Jess+) reshapes creative practice and social dynamics within an inclusive music ensemble. It explores why and how the technology opens new improvisational possibilities, alters musicians’ sense of agency, flattens hierarchical roles, and supports interdisciplinary collaboration. The study then measured the concrete musical, participatory, affective and technical effects on both disabled and non‑disabled players, and finally derives design guidelines, ethical considerations, and broader implications for inclusive artistic practice.
Questions
Core: How does a digital score created using AI and robotics stimulate new creative opportunities and relationships within the practices of an inclusive music ensemble?
Sub-Questions
1. Mechanisms & “why/how”
Q1.1: Why does Jess+ enable novel improvisational ideas for all musicians? Q1.2: How do musicians experience being “in‑the‑loop” with an autonomous system (confidence, agency, reduced anxiety)? Q1.3: In what ways does the robot flatten ensemble hierarchies and redistribute musical authority? Q1.4: How does cross‑sector collaboration (musicians+ developers+ researchers+ organisations) shape design, iteration, and acceptance of the technology?
2. Outcomes & “what effects”
Q2.1: What new musical structures, textures, or genres emerge from using the AI‑driven score? Q2.2: How does the system change the participation level of the disabled musician compared with a baseline without Jess+? Q2.3: Do non‑disabled musicians shift their role perception (from accompanist to co‑creator) when interacting with Jess+? Q2.4: What social/affective changes occur (trust, cohesion, collective efficacy) after repeated sessions? Q2.5: What technical performance characteristics (latency, reliability) are required to sustain the reported creative flow?
3. Design implications & future directions
Q3.1: What design guidelines can be derived for AI/robotic “creative accompanists” that support inclusive music‑making? Q3.2: How can the digital‑score interface be embodied (projections, haptics) to better accommodate diverse physical abilities? Q3.3: What ethical issues arise when an autonomous system takes on a “creative voice” (authorship, agency, ownership)? Q3.4: Can the Jess+ approach be generalized to other inclusive art forms (dance, theatre, visual arts) and what adaptations are needed? Q3.5: How does integrating AI/robotics influence the long‑term sustainability of inclusive ensembles (recruitment, retention, community outreach)?
System Design
Jess+ is a closed‑loop digital‑score platform that brings an AI‑driven robotic arm into the practice of a mixed‑ability ensemble, using a prior proof‑of‑concept arm‑drawing system as a shortcut that already resonated with musicians, saved development time, and made the AI tangible. The architecture is organised into four modular layers:
(1) a perceptual layer that collects and formats real‑time sound from all players together with physiological signals (EEG and skin‑conductance) from the disabled performer;
(2) an “AI Factory” of seven hour‑glass convolutional neural networks trained on the Embodied Musicking Dataset to predict a continuously updated abstract “response intensity”;
(3) a gesture‑manager that bundles these predictions into “thought trains” – 3‑ to 8‑second streams held before a random switch, with a “startled” interrupt triggered by loud sound; and
(4) a belief‑system layer that translates the selected thought‑train into a concrete robot gesture drawn from a fixed repository of pre‑defined movements (shapes and gestures inspired by Cardew’s Treatise and Wolff’s For1, 2, 3 Players), assigning randomised parameters for speed, acceleration, and endpoint.
The arm either inks the chosen gesture onto paper or moves in space, supplying an embodied, visual cue that the musicians improvise around; the performer’s physiological response feeds back into the loop, closing the interaction. This gestural “language” functions as a belief system that constrains the AI to a specific aesthetic, while still allowing musicians to add new gestures, so the robot is perceived not as a mere tool but as a conductor‑like, non‑judgmental creative partner that expands inclusive music‑making.
Methodology
The study adopted a practice‑based, qualitative case‑study approach that placed the three ensemble members’ lived experience at the centre of the inquiry, treating their reflections and the artefacts of their improvisations as the primary data.
Over a four‑month period the team ran five six‑hour workshops (plus a final, three‑stage session) in a “safe‑space” lab where musicians could freely improvise with the Jess+ system, discuss their reactions, and co‑design successive prototypes. Each workshop followed an open cycle of improvisation‑then‑reflection; between sessions the researchers iteratively refined the hardware and AI pipeline in response to participant feedback and technical needs.
The prototype evolved from a simple Dobot Magician Lite arm driven by random stream selection to a more robust system featuring a Borg data‑management architecture, EEG/EDA sensing, encoder‑decoder neural models, and ultimately a UFACTORY XArm capable of multi‑pen and “dance” gestures, with added start/end cues and a “startled” interruption mechanism.
Semi‑structured interviews were conducted after the first and fourth workshops to capture deeper personal perspectives. All sessions were video‑recorded, supplemented by detailed field notes, and interview recordings were transcribed. Researchers independently reviewed these materials after each workshop, flagging notable observations that informed the next meeting’s agenda; a final, comprehensive review of notes and transcripts was then undertaken in paired data‑analysis sessions to distil the emergent findings. This iterative, performer‑centred methodology allowed nuanced, context‑specific insights into how the AI‑robotic digital score mediated inclusive music‑making.
Findings
Over the course of the iterative, practice‑based study the research team discovered that the introduction of Jess+ benefitted every member of the inclusive ensemble in ways that far exceeded the project’s original expectations. All three musicians felt “in‑the‑loop” with the system, describing it as a non‑judgmental, attentive “voice” that listened, responded and offered musical gestures.
Jess, the disabled performer, experienced the robot as an extension of herself - a “friend” and “story‑teller” that translated her physiological and emotional state into visual artwork, giving her a new avenue for expressing feelings that her existing digital set‑up could not support. The two non‑disabled musicians also called Jess+ a friend but framed it as a “creative accompanist” that contributed its own gestures, allowing them to take risks, break habitual hierarchies and explore novel improvised ideas.
Playing with Jess+ created a “third space” that flattened mobility‑based hierarchies, reduced the anxiety of judgment, and fostered a stronger sense of togetherness. As a result, their improvisational confidence and skill grew, and they reported that the robot’s liberating, non‑human presence offered a freedom of musical and emotional expression that would take far longer to develop in purely human ensembles.
Overall impact:
- All musicians reported enhanced practice and transformed relationships; outcomes surpassed the project’s initial aims.
- Feeling “in‑the‑loop”: Each player perceived continuous, two‑way interaction with Jess+, viewing it as a responsive, non‑judgmental partner.
Jess’s perspective (disabled musician):
- Described Jess+ as an extension of herself, a “friend” and “story‑teller.”
- Valued the system for turning her physiological data and emotions into a visual representation of the music.
- Gained a new channel for expressing feelings that her prior digital tools could not convey.
Non‑disabled musicians’ perspective (Clare & Deirdre):
- Called Jess+ a “creative accompanist” that contributed artistic gestures.
- Saw the robot as a friend that enriched improvisation without imposing hierarchy.
- Creative freedom & risk‑taking: The robot’s non‑human, non‑judgmental stance encouraged musicians to experiment, leading to novel musical ideas and heightened improvisational confidence.
- Flattened hierarchy / “third space”: Jess+ created a shared creative environment that neutralized mobility‑based power imbalances and fostered a stronger sense of inclusion and togetherness.
- Reduced anxiety & judgment: Musicians reported less pressure from social expectations (“expectations and judgments”) when playing with the robot compared to human collaborators.
- Skill development: Participants noted improved improvisational skills and confidence that they could transfer to future performances and outreach activities.
- Deirdre’s synthesis (post‑performance questionnaire): The robot arm was liberating—non‑judgmental, sometimes unifying the trio, sometimes acting independently, allowing the group to start/stop, “gel” harmonically, or freely break rhythmic/harmonic conventions; it offered a quicker route to the freedom of musical and emotional expression that human collaboration often requires long‑term trust to achieve.
Dissemination
- a. Full AI and robotic code (Prof Craig Vear as main designer and developer, Dr Johann Benerradi as RA) LINK
- b. Conference papers
- i. Vear, C., Hazzard, A., Moroz, S., & Benerradi, J. (2024, May). Jess+: AI and robotics with inclusive music-making. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-17).
- ii. Vear, C., & Benerradi, J. (2024). Jess+: Designing embodied AI for interactive music-making. ArXiv. https://arxiv.org/abs/2412.06469
- iii. Workshop paper (published) GenAICHI 2024 https://generativeaiandhci.github.io/ (in print)
- iv. Vear, C (2021) Creative AI and Musicking Robots. Front. Robot. AI 8:631752. doi: 10.3389/frobt.2021.631752
- v. Miles, O., Moroz, S., Hazzard, A., Bishop, L., & Vear, C. (Eds.). (2025). Meaningful Interactions in Human-AI Musicking [Edited Proceedings]. 20th International Audio Mostly Conference (AM’25), Coimbra, Portugal. https://doi.org/10.1145/3771594.3771600
- c. Book Chapter in Digital Musicking - Vear, C., & Hazzard, A., Under Development
- d. Documented performances
Project Highlights
- Impact Statement
- Awards:
Participants
Jess Fisher Jess is a disabled musician and composer. She performs in inclusive ensembles and as a solo artist, using primary digital music tools and technologies. She worked with Anonymous on the creation of a bespoke accessible music controller CMPSR which Jess typically interfaces with a range of contemporary digital audio workstation software. In ensemble settings Jess performs the music of other composers, typically using a bespoke music notation called ‘arrow notation’ which reflects the design of a CMPSR controller. Jess does on occasion improvise, but is not as familiar or comfortable in such musical settings.
Deirdre Bencsik and Clare Bhabra Deirdre is a professional cellist and Clare a professional violinist who are both long-standing members of Sinfonia Viva, a UK based orchestra and educational organisation. Their performance practice is rooted in the classical tradition reading from standard western notation. Both Deirdre and Clare improvise in some community-based projects but both professed to not being confident improvisers. Neither employ any digital technology in their own practice, although Anonymous do sometimes collaborate with digital musicians and artists on some projects.
Research Team
Dr Adrian Hazzard Qualitative research into trustworthiness and musicking with AI.
Johann Benerradi Software development, deep learning, robot movement programming.
Dr Solomiya Moroz Embodied music cognition.
Adam Stephenson Robot movement design.
Partners
Sinfonia Viva
a British orchestra based in Derby, England https://www.sinfoniaviva.co.uk/
Orchestras Live a national producer creating inspiring orchestral experiences for communities across England https://www.orchestraslive.org.uk/
Funding
This project received additional funding from the Trustworthy Autonomous System Hub https://tas.ac.uk/ and the Faculty of Arts at University of Nottingham https://www.nottingham.ac.uk/arts/