Point Line Piano
Documentation trailer: https://vimeo.com/938043123/de174aa245?share=copy
Performance: https://vimeo.com/938033649/cf27eaadf1?share=copy
About Point Line Piano
Point Line Piano is an interactive VR project merging drawing and music, creating a vibrant, multisensory environment. Users draw points and lines in VR, activating musical notes and creating complex visuals. The project aims to stimulate multi-sensory creativity and foster a connection with abstraction. It transforms creativity by integrating composing, performing, and viewing into one experience, allowing everyone to be a composer, performer, and viewer. It supports inclusive music-making, welcoming diverse participants, and plans to become more accessible in the future.
Objectives
In this work, Jarek Kapuscinski and OpenEndedGroup’s Paul Kaiser and Marc Downie wanted to explore the potential of the VR environment. This they envisioned through VR as an immersive experience involving music, image, and interactivity, particularly exploring the fusion of image and score in VR environments. Embracing interactivity as a key property of VR, despite previous reservations, they wanted to engage viewers more deeply in the artistic experience. Particularly, their objectives were to integrate music, gesture, and image in a way that mutually informs and influences each other, aiming for a seamless fusion of artistic elements. Another objective was integration of spatial sound in VR to influence the viewer’s experience in the work. Future objectives include incorporating user feedback to adjust and refine the project’s direction, including changes to scene structure, musical grids, and user interaction, continuing to explore different formats for audience engagement and interactions through exhibitions and live performances.
Process
The journey of creating Point Line Piano began with the desire to further explore the transformative collaborative experiences from previous projects. The decision to merge image and score in VR was inspired by the immersive potential of the medium and its ability to engage the body through gesture and movement. Initial ideas drew from various inspirations such as abstract calligraphies, Chinese ideograms, and Zen script. The team embraced interactivity in VR, although they had previously avoided it due to concerns about reducing the audience to mere users. They drew inspiration from past interactive installations and sought to redefine engagement with the audience. The project evolved through real-time remote collaboration, coding, and experimentation with music and visuals. Residencies at Stanford and CCRMA provided opportunities to refine the project’s structure and explore new ideas. The team experimented with musical layering, scene progression, and spatial sound. Viewer feedback influenced decisions to simplify scenes and give users more control over the experience. The project culminated in a final presentation at ZKM, Karlsruhe, where the team observed unexpected approaches to interaction among viewers. The installation garnered attention not only as an artwork but also as a potential instrument for intermedia concerts.
Critical Insights from the Analysis
Connectivity
• The project prioritises embodied interaction, allowing users to engage with the artwork through physical gestures and movements in the virtual space.
• The project involves users in the creation of visual structures through their gestures, which is accompanied by phrases of piano music. This interactive aspect allows users to explore complex geometries and unfold intricate displays, enhancing their engagement with the artwork.
• The interaction fosters a sense of presence and agency, enhancing the user’s connection to the audio-visual environment.
• By integrating drawing, music, and visuals, Point Line Piano allows viewer to connect to the work through the multisensory experience that engages users’ auditory, visual, and tactile senses.
• Through the sensory immersion, the user forms an emotional and cognitive connection to the artwork.
• The project provides users with immediate and responsive feedback to their actions, allowing them to observe the real-time effects of their interactions on the audio-visual environment.
• This dynamic feedback loop promotes a sense of co-authorship and collaboration between the user and the artwork, enhancing the user’s connectivity and engagement.
• The freedom to explore promotes a sense of agency and autonomy, deepening the user’s connection to the artwork and fostering a rich and immersive experience.
• The project seeks to stimulate multi-sensory creativity, fostering a connection with abstraction as an embodied, self-performed experience. Users immerse themselves in the interactive artwork, discovering new ways to express themselves through drawing and music creation.
Flow
• Process of unfold: In each scene of Point Line Piano, the invitation to draw, listen, examine, and witness a process unfold remains constant, but the rules shift depending on the scene.
• Layered Music: The processes of drawing and music creation in Point Line Piano often occur simultaneously, resulting in music that exists on different layers between the immediacy of foreground melodic drawing, its echoes, and backgrounds.
• Point Line Piano offers users a high degree of exploratory freedom within the virtual space, allowing them to navigate and interact with the environment at their own pace and according to their own interests.
• Interactivity and Exploration: Point Line Piano encourages users to explore and experiment with the medium, prompting them to consider not just what to draw but also when to draw it.
• Users learn to manipulate timing and spatial elements to create unique experiences through exploration
• There is a dynamic feedback loop between the visual and musical elements of the piece. Visual gestures influence musical composition, and vice versa, creating a tightly integrated audiovisual experience.
• The structure of Point Line Piano allows for flexibility and experimentation. Scenes progress through distinct stages, inviting viewers to draw, watch, listen, and experience the evolving artwork. However, there is no fixed notion of what a scene should be, allowing for creative exploration.
• While the piece is almost entirely abstract, certain visual structures gently suggest narratives, such as museum vitrines or architectural elements like doors and windows. These suggestions add layers of meaning to the experience.
Digital Musicianship
• The project encourages users to express themselves creatively within the virtual environment, offering tools and interactions that enable personalized exploration and experimentation.
• Users’ engagement with Point Line Piano varied based on their backgrounds and experiences. Some users, particularly older ones, approached the interaction tentatively, while others explored more confidently, expressing amazement at the new way of experiencing sight and sound.
• The interaction with Point Line Piano is likened to playing an instrument, albeit a non-traditional one. Users have control over selecting notes, durations, and timing, which contributes to the overall musical and visual experience.
• There is a shift towards giving users more control over the interactive elements of the piece. Rather than precomposed sequences dominating the experience, users are encouraged to take more agency in shaping the visuals and music through their actions.
• The project aims to be inclusive, welcoming audiences with no particular digital skills. It challenges conventional notions of skill, control, talent, and spectacle typically associated with music and art creation.
• Point Line Piano may initially disarm accomplished musicians, it offers a unique experience that captivates users by engaging their ears, eyes, and hands in concert with the interactive artwork.
• Users have the opportunity to improve their performance over time by repeating scenes and experimenting with different approaches. They become more flexible and take more risks, leading to a deeper understanding of the medium and better satisfaction of their artistic tastes.
• The project introduces spatial sound exploration as an underused aspect of music composition. Users can create spatial constellations of notes around them, exploring the interaction between pitch relations and spatial dimensions.
• Point Line Piano encourages improvisation with given materials, fostering a sense of density and harmony among users.
• Exploration of Harmonic Spaces: Users can spend significant time exploring different harmonic spaces within each scene, allowing for deep compositional exploration and improvisation.
Transformations
• Point Line Piano offers a spatial and full-body experience of abstraction that cannot be found in any other medium. It opens participants’ ears and eyes to new aspects of perception.
• Inclusive Participation: The interactive nature of the project invites participation from diverse audiences regardless of age, background, impairments, or abilities. It enables them to co-create unique, joyful, and immersive experiences that can be therapeutic and uplifting.
• While currently requiring high-end technology, there are plans to adapt the project into a more accessible format once consumer hardware advancements and additional funding permit. This reflects a commitment to making the experience available to a wider audience.
• Broad Audience Engagement: The project has attracted a broad audience, including children as young as five years old, who enthusiastically engage with the experience. This indicates that the project’s appeal spans different age groups and levels of tech experience.
• Users express amazement at the project’s new way of experiencing sight and sound, suggesting that it offers a transformative experience. The project’s ability to create joyful, playful, and immersive experiences is highlighted as particularly satisfying and cathartic.
• The project explores the balance between control and freedom in interactive experiences. There is a recognition of the importance of allowing users freedom to explore and create, while also ensuring that the overall experience remains satisfying and coherent.
• While users engage with the artwork individually, Point Line Piano hopes to facilitate shared experiences within a multi-viewer VR space in the future. This communal aspect of the project could allow users to interact and collaborate with others, fostering a sense of social connection and shared creativity.
Summary
Point Line Piano is an interactive VR project that merges drawing and music, offering a vibrant and multisensory experience. Users draw points and lines in VR, activating musical notes and creating intricate visuals. The project aims to stimulate multi-sensory creativity and foster a connection with abstraction, allowing users to be composers, performers, and viewers simultaneously. The objectives of the project include integrating music, gesture, and image seamlessly, embracing interactivity in VR, and incorporating spatial sound to enhance the viewer’s experience. The process of creating Point Line Piano involved collaborative exploration, experimentation with music and visuals, and refinement through residencies and user feedback. The project prioritizes embodied interaction, allowing users to engage physically with the artwork and encouraging exploration and experimentation. Through dynamic feedback loops between visuals and music, users co-author the experience, fostering a deep connection to the artwork. Point Line Piano challenges conventional notions of skill and control in music and art creation, welcoming diverse audiences and providing inclusive and transformative experiences. The project aims to become more accessible in the future while continuing to explore different formats for audience engagement.
Personnel