Critical Insights

• for the past 6 years has been incorporating VR technologies into her artistic practice
• it came with some experimentation of how to create different sonic environments in the visual environments of modernist paintings
• Moving Sound Pictures - Konstantina takes paintings of well-known painters, the classic moderns and turns them into 3D spaces in virtual reality. The user has the opportunity to interact and make music with these 3D objects in VR
• For Konstantina making music and sounds in 3D environments has been about telling a story
• first started with experimenting with Kandinsky’s work as he was known for his synesthesia
• using VR for art mediation, what could different abstract symbols mean to one when interacting inside VR spaces?
• she thinks of paintings now as instruments where users can create their musical experiments by interacting with different objects
• the experiences the user has in these VR paintings are like what one might have with an open score
• Konstantina is also inspired by Cage and his thinking on silence and objects
• constraints in her work can be inspiring and not just limiting, she looks at the intersections where technology meets art
• using a headset is like having your head look through the window so you want to see something there
• the object becomes embodied when the user engages with it, so she has to think about the details of its behaviour when she composes
• her work in VR is like building a closed system in which there are interactive objects that are part of one instrument
• it becomes very interesting to observe what kind of embodied experiences users have inside the systems that Konstantina sets up inside the painting
• believes VR has a lot of advantages: can use it for simulation, training, building new worlds, for collaborative works – so there is a good chance that it will continue to exist
• has some questions for the future of VR: how we can use the medium as a window for new opportunities, how you can experiment with new things that normally you would not be able to
• there are still limitations, it still takes a lot of work to achieve something good in VR
• the tech itself will become lighter and more mobile as well as more affordable
• she would love to see more mixing of technologies for concerts for example between virtual, mediatized and acoustic
• things are there and we should mix them, not everything just online or virtual!

Summary

Konstantina Orlandatou is a researcher at Hamburg HfMT in digital media. She is a classically trained composer who started to compose for VR and multimedia environments when she came to study at the Hamburg HfMT. There she started incorporating different technologies into her artistic practice such as Virtual Reality. She has been always interested in visual arts since the beginning of her studies in multimedia composition as a medium to work with. She admits that there are many constraints of working in this medium, but she finds that there are also many advantages.

When she started working as a composer in multimedia, she started to experiment of how to create different sonic environments in the visuals of the paintings she was working with. The project that Konstantina is involved with is already 5 years old, it’s called the Moving Sound Pictures, where she takes paintings of well-known painters, the classical modernists and turns them into 3D spaces in virtual reality. The user has the opportunity to interact and make music with these 3D objects in VR. For Konstantina making music and sounds in 3D environments has been about telling a story. When looking at the original paintings, they are static and 2D but they tell a story, so Konstantina thought, why not jump into the painting and explore the sounds of the story that some of the objects could be made to tell in the painting.

When she started working on re-imagining classical modernists in 3D, she was very inspired by Kandinsky, who is known to have experienced synesthesia. Thus, she has started experimenting with cross-modal correspondences between sound and colour. She finds that for the user it has to do with one’s background and how the painting makes sense to you. She started working with abstract art because it is open and can mean different things to different people. Using VR for art mediation is seeing what different abstract symbols could mean. Additionally, she finds inspiration in Pop Art, Edward Hopper as well as Cage’s ideas on silence. Finding objects to play with in Konstantina’s 3D VR environments reminds one of Cage’s aesthetic. She often thinks of paintings as instruments that the user interacts with in her work. The experiences the user has in her VR paintings are similar to what one might have with an open score where one becomes a co-creator of musical experiences.

On Tools and Technology

She is always trying to understand where technology intersects with composition. Sometimes when she has constraints it becomes inspiring for her work. She often seeks the overall mood of the paintings as it is not certain how the user will interact with the sound events. Colour plays a big part in setting the mood where all the events have to work together in the painting. She had to learn Unreal and how it works to create works in VR. She prefers to work with very stable systems since her installations can sometimes run for 3 months at an art gallery. She often likes to work in Middleware to create her sounds for VR objects and sometimes uses FMod. She also likes to work with a sequencer which helps to organize her sound materials logically. For the visuals, she collaborates with a visual artist who makes all the artwork of the paintings in VR.

Her work with the musical material in VR is very focused on the objects and what it feels like to hold them inside the painting, their texture and their behaviour. The idea of music as a medium needs to be adjusted in her VR work, she feels like she is working more as a sound designer who is composing the user’s experience in the paintings. The objects become embodied when the user engages with them, so she has to think about the details of their behaviour and the kinds of experiences that the user will have with these objects. Thus, she focuses on building a closed system with interactive objects that you can think of as instrument on the whole.

It becomes very interesting to observe what kind of embodied experiences users have inside the systems that Konstantina sets up inside the painting. She finds some older users need permission to interact in the space with encouraging prompts; whereas, younger ones tend to jump into the space and explore much more. People usually come out with big smiles on their faces, they have a feeling that they co-created something – that’s a very important aspect for her. Since the beginning of January Konstantina is leading the XR lab of the new funded ligeti centre for innovation and creation for multimedia and new music. For Konstantina’s VR works, she has plans to extend the system to use haptic sensors, so one can feel the surfaces when touching painted objects in VR. Currently, they are also considering sonifying sculptures which are in the National Gallery. Generally, in her work, she aims for her installations to be experienced by everyone, from a child to an 85-year-old, thus having more access for people who have physical limitations or visual impairments through haptic feedback would be a great extension of her current work.

On the future of VR

It is hard to predict in VR what the new heights of it will be, it is becoming more and more prevalent and especially now with Metaverse, etc. She believes VR has a lot of advantages: one can use it for simulation, training, building new worlds, and collaborative works – so there is a good chance that it will continue to exist. An extension of the Moving Sound Pictures with the National Gallery will be working with institutions that also want to implement their educational programs into VR.

Future Projects

She is currently looking into implementing animated graphical scores into the VR environment, maybe also with musicians who would be playing in the VR space. She finds that animation as a medium is very powerful and popular with younger generations. She is still questioning where VR will land in the future, how we can use the medium as a window for new opportunities, how you can experiment with new things, and where you can do things that normally you would not be able to. The pandemic showed us that we can do a lot of things online and create new user experiences. Now it’s the question of extending this knowledge into musical activities that can also happen live. Konstantina finds that there are limitations because it still takes a lot of work to achieve something good in VR. However, with time, she thinks the technology part of VR will become lighter, more mobile and more affordable, thus VR could become a bigger part of our creative activities. She would love to see more mixing of technologies for concerts for example between virtual, mediatized and acoustic.