| |
C.L.A.V.E. - Collaborative Live Audio Virtual Environment
Performance Collaboration through Virtual and Physical Space
Traditionally a clave is a cylindrical hardwood stick used in a pair as a percussion instrument or a syncopated two-bar musical pattern. My project was to create a visual instrument to create new syntaxes for collaboration between musicians and also potentially their audiences.
|
Towards the end of 2004 my research into physically transforming architecture required a parrallel virtual side with which to rapidly test ways of using real world data to transform structures in real-time. These Virtual interactive 3Dimensional 'Datascapes' used Frequency, Amplitude and Rhythm as the data inputs and as the software I developed became more complex I started to wanted to find out more about how these increasing amounts of data could construct more complex and more visually fluid outcomes.
"Derived from the particulars of the real world, from data and processes of the virtual world, or from numerous techniques of capturing the real and casting it into virtual, motion capture for instance. Since time is a feature of the model, if the model is fed time-based data, the form becomes animate, the architecture liquid." Marcos Novak .
These ideas of surfaces of data moving in relation to live data, particularly Video and Audio was the focus of my experimentation. To get Video and Audio from one source I began looking into live performances of music. My first aim was to build a system which would be able to create live datascapes representative of the performance to support these events.
|
|
|
Using OpenGL inside of MaxMSP Jitter I built an application that took color and luminosity values and constructed 3 Dimensional topologies. The use of video provided a rich textured surface and came to life when live video was feed through it. I also added a second video input and was able to cross feed live and prerecorded footage to build collages of terrain. Using a number of effects and rendering options I was able to create a vast number of different visuals from few video sources.
| VJ application
I contacted the University Student Union about testing my application on a live event. The DJ Society offered me the opportunity to support them as a VJ for the visiting guest DJ Kid Lopez on a Saturday 19th February. This was perfect as there would be a large crowd for me to get feedback off. I also invited two more MediaLab Students Ben_Hanbury and Emma_Blackburn to bring down their video work, to have a play and share ideas.
I wired up my application to an Evolution UC-16 MIDI Controller so I could easily and quickly manipulate all of the variables that influence the aesthetic elements as well as all the 3D camera/object movement and rotation live
This also made it possible to give other people control of my patch and play with it giving me an opportunity to see what other people would do with the visuals. My next aim would be to see what applications this could have in situations where there are multiple performers. My hope was that these datascapes will provide a visual language to assist the performing artists work collaboratively.
|
|
|
|
|
There are different models of collaboration, many traditional models are hierarchical in structure of the master-assistant variety such as Composer and Musician, however more recent values towards equal collaboration have appeared as a result of a post-modern shift away from the modernist values of an artistic genius having totally dominant lead over production.
In popular culture the growth of Jazz spread a new process of live collaboration or ‘Jamming' where different musicians may take leads at different points but as a whole share more equal roles in leading the production of new compositions and sound production. Improvisation and communication were key facets of this model. This new approach spread through into many genres of music creating new dynamic performance and composition methodologies.
|
What is interesting to me about this is how much of the communication came from listening to each other and watching each others movements to make judgments about rhythm and expression rather than speaking to each other. Musicians use all their senses to work successful in collaboration.
|
|
|
Collaboration in Physical Space
Performances of music usually position the musicians in fixed positions within a space. This is often related to the physical issues of instruments, this therefore requires carefully arrangement of a performance in space to maximize the ability of the musicians to communicate effectively. In jamming and improvisation this is essential. The larger the collaboration becomes the harder it is to arrange the space to work effectively. Often compromises have to be made limiting musicians ability to communicate effectively to the all the other musicians, therefore limiting opportunities to collaborate.
|
|
The role of music conductors has been to act as central reference to larger groups of musicians. Having a central reference however creates a hierarchy and detracts from the very nature of Jamming. This got me thinking about the difficulties that arise from collaborations and led me to start thinking about processes of communication that could solve this problem. It would have to be simple and scaleable to groups of any size. Its key characteristics would have to be that it provides the opportunity to communicate with everyone at the same time and solve the issues of physical space. I began looking for examples of how other people have attempted to solve this problem using digital interfaces; Projects that had looked at visualizing sound controls in a collaborative interface.
|
Audiopad
http://web.media.mit.edu/~jpatten/audiopad/index.php
“Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music.”
This project was interesting to me in terms of interface because it created an object orientated environment that multiple people could tactilely interact with even if it is designed to be used by a limited number of users, it has the potential to be scaled up to create an interface where people can hand objects to each other and apply effects and create sounds. While my project was not to create sound by the interface since this will be produced in Physical space by instruments, the use of objects to affect sounds, the ideas of collaboration and swapping objects all I saw as having potential in my designs.
Next Step
Following on from my VJ visualizing topology I explored the idea of producing a virtual collaboration system. I built a virtual space for the musicians to use in combination with physical space. A place where musicians while fixed in physical space when performing would be in virtual space able to move around the other musicians communicating through observation just as small groups of musicians jamming do.
|
|
.
|
C.L.A.V.E
The age-old clave, the simplest of instruments, a primitive object that represents the foundation of musical instruments, rhythm and composition has been a long part of traditional African music. It's treated as a foundation upon which complexity is formed. For me it holds a significance in its simplicity. My use of geometry primitives in as my aesthetic is an approach which I believe in a multi-user environment can act as a foundation for dynamic compositional and collaborative complexity.
C.L.A.V.E Control GUI
I built an application with 3 major components. The video feed, the audio feed and dynamic audio effects. My demo application contains one representation of a musician. By scaling this project up you could have a 3D world which had multiple representations.
If a drummer wanted to have a look at the rhythm of the pianist and what effects settings he's got arranged and maybe check what kind of noise he's making as well as the pitch, even if the pianist has a whole orchestra creating a visual and audio barrier in the way in physical space, the visual information he needs would be easily found quickly and simply by moving around viewing in virtual space his representation.
|
|
| |
Representing musicians as objects in virtual space, means that that an environment opens up where new movements and new visual languages can be used to communicate and potentially open up new visual/audio syntaxes.
|
|
|
|
|
A video feed sits at the center of the musician's representation. I brought in my VJ 3D topology here to turn the video feed into another visual cue to the type of music the other musician is producing. The more distorted the audio gets, the more the visual topology extrudes the framing of the open box distorting the live video.
|
|
|
It's resolution changes with changes in pitch so the Triangle player will always be in high resolution sharp focus but the Timpani will be much lower in resolution. Dynamic instruments like pianos will rise and fall in resolution as different octaves are played. The Rhythm of the musician is represented as changes in resolution that occur at every beat.
|
|
|
The open box also provides added information about basic effects. Chorus changes the size of the box so as a sound becomes more expansive, the box frame expands too. Flanger changes the state of the framing box so as sound begins to wobble, the box wobbles as well.
|
The sound effects are represented as primitive objects, a sphere in the case of the application demo. This approach followed some of the ideas of Audiopads simplification of objects of interaction. The depth in these objects comes from how they can be used in a 3Dimensional space and with multiple users. I have used VST plug-in as an example of how different effects could be added easily. Musicians could interactively adjust each others effects as well as sharing effects by moving your position in virtual space closer to the chosen effect.
 |
|
Conclusion
If this had been a larger project I would like to have looked at how this software could be applied control systems for the musicians to control. It may have been a foot pedal system, voice activated, a wearable technology, even maybe a head up display with optical tracking but this was outside the scope of my project. Control devices could be built into the instruments to make hybrid instruments bring the virtual and physical closer and more integrated.
The Virtual Environment layout is simple and the addition of extra effect plug-ins is simple, so too is adding multiple individuals. I like the idea of sharing and applying sound effects to each other. Giving everyone control over each others instruments adding an extra level of collaboration.
The potential to creative new and more believable audio/visual syntaxes interests me as it has applications not just in collaboration but as part of VJing. If the visual language can relate faithfully to live audio then the audience will be absorbed further into a whole multimedia performance.
This Virtual Environment could be displayed visually as part of a musical performance giving the audience a way to investigate individual instruments more closely, observe interaction between the musicians or even become involved in the collaboration by taking control of the objects.
As I whole I am pleased with the visual qualities of my VJing software and the collaborative prototype C.L.A.V.E. Both have different purposes but share many fundamental ideas about how we can visualize space and how abstract visualizations could potentially enhance collaboration, understanding and most importantly enjoyment in a live performance.
|
|
|
