MADA2 Assessment:

SUBMISSION DETAILS:

HANDS.mov – sound & vision

This is the prototype of my project ‘Can you hear my feeling?’

It is a loop of 2 gestures repeating 3 times. I have looped it to give the general idea of multiple gestures. The final piece won’t loop like this, it will have many varied gestures speaking to one and other within the frame.

The final piece will be approximately 3-4 minutes in duration and will use colour instead of black and white gestures. I have developed my palette in RGB (see my notebook, page marked with a blue bookmark) but am yet to integrate this palette into the gestures. I intend to use the outlines as an alpha channel as well as rendering ‘filled’ white hands to use as alpha channels to make hands in solid colours. I am interested in experimenting with intersecting the colours as the gestures overlap.

After experimenting with electronic / synthesiser sounds I found that they were too ‘loaded’ with meaning and I didn’t like the way they effected the emotional response of the visuals. I have instead been collecting sounds played on acoustic instruments. I am intentionally sampling them from places that have time and space embedded into the tonal quality of the recording, rather than actually playing and recording the instruments myself. By using these recordings I am intentionally playing with their potential nostalgic effect on the listener.

I am using PROTOOLS to compose the sounds rather than a clock-time MIDI based programme so that the music has a ‘natural’ sense of time. I tried writing in a MIDI sequencer but the music didn’t seem connected to the gestures. PROTOOLS’s timeline makes it easier to arrange the sounds to the gestures, as opposed to the software time grid. There is a balance here though, as a musician, out of time music doesn’t sit well with me. So if the gestures don’t sit with the timing of the music I can send them back to AFTER FX, re-edit, then re-import them back into PROTOOLS.



Notebook – start at blue bookmark

This is a place for me to jot down ideas outside of the computer.
There are some examples of other artist’s work that relates to my project.
I have found the illustrations of the hands helpful for practice drawing. When I draw over the video footage of the gestures, if the hands are in fast motion the footage is blurred between the fields. Having an intimate knowledge of how to draw hands in multiple positions helps fill in the gaps.
Now that I have my technical methodology and workflow in place the details are written in my sketchbook. They are also on my blog.

INTERIM REPORT:

‘Can you hear my feeling?’ is progressing well. I have defined my technical methodology and workflow (see blog or sketchbook for details). Thereby enabling me to be free creatively with the materials I bring to the audio and visual timelines.

My attempts to film people’s gestures failed for two reasons. One was that the filming situation was too staged for my subjects to express the unconscious gestures I was aiming to capture. The other was the quality of the film was too low for me to see the details of the hands in fast motion. I have solved this problem by discovering a wealth of gestural footage from the world of sport. Sports footage is perfect for me as it is a heightened emotional situation and the players are often attempting to communicate with people out of ear shot. I have acquired some close-up material of players that is shot on slow motion cameras. This has the added advantage of being good quality due to the high frame rate. Making it easier to draw over the hands in FLASH. I am currently working on rotoscoping a gesture a week.

I aim for the whole composition to be around 3-4 minutes in duration. Over the next few months I will spend time researching how this will be projected with sound within the exhibition space. This is a vital element to the production of the piece. I don’t like headphones in a gallery situation as I think it is alienating to the audience. The sound is a vital part of my piece so how it is projected needs close consideration. I am currently thinking about an enclosed comfy space with chairs and darkness, perhaps projecting on more than one surface.

I am having second thoughts about my proposed performance of the piece by conducting an orchestra of visual materials, cardboard cutout grids of each frame being ‘played’ by a group changing the cards in time. I think the visual material of the gestures is very delicate and I don’t think it would be clear in such a large (and potentially randomly timed) format. I have not cast aside the idea though; I will just save it for another project.

I was surprised to discover that my synthesiers (where I tend to create my sounds) do not sit well with my gestural material. The tone colour seems too ‘loaded’ with connotations of a certain time. Digital Art as a new medium already has clichés and electronic music is one of them. I definitely do not want my piece to look and sound like club visuals. I hate to admit it but acoustic instruments seem to engage the emotional field better for this project.

I am aiming to have all of my gestures rotoscoped within the next few months at which time I will also have my sound library collated. Concurrently I will be composing vignettes of gestures in AFTER FX and PROTOOLS. The final step is working on the entire timeline as a whole.

My next step in 2007 is to start experimenting with colour. I have defined my palette but haven’t really started to explore how to use it within the piece. On researching the long history of connections between music and colour I have come to the conclusion that no specific colour links to a specific note or chord. I will be experimenting based on my own feelings.

In building the prototype of my project I have finalised the technical workflow, thereby enabling me to be free creatively with the materials I bring to the audio and visual timelines.

1. Rotoscope the gestures in FLASH (from video footage) as white splines on black. I find that rather than keyframes at regular intervals the motion is better if I keyframe at key moments in the gestures. This makes the animation of the lines more fluid. I am not tweening the keyframes as the lines are too complex when the hands twist and turn in relation to the camera view. Therefore it is more of a stop frame style and the irregular keyframes draw the emotion out of the gestures and move away from clock time.

2. Take a rendered .MOV of the white on black gesture into AFTER FX. The original football footage was shot in slomo (to get better quality of the fast moving action) so my gestures need to be timesquashed. They look best rendered with motion blur but without field rendering as it blurs the purity of the line.

3. In AFTER FX I use the timeline and layers to create my symphony of gestures. I will use the white lines of the hands as alpha channels to fill with colours from my palette. In FLASH I will render versions of the gestures as white fill shapes to use in AFTER FX to make solid coloured shapes of the hands. Where the gestures intersect I will collide the colours. I will perhaps experiment with the 3D camera to play with the position of the gesture in the frame.

4. I am constantly moving back and forth between AFTER FX and PROTOOLS rather than creating the whole visual timeline before the sound or vice versa. In PROTOOLS I am loading all of my raw sound files into the library. These come from my ever expanding collection of field recordings. I have been collecting sounds from (unrecognizable) film soundtracks as its main function is to heighten emotion in the viewer. In PROTOOLS I can compose these sounds on the timeline while playing my .MOV of gestures. I also record ‘live’ sounds from my electronic musical instruments to the .MOV. This can then be exported back to AFTER FX either as picture, sound or both for me to re-edit the gestures in response to the composed music.

I aim for the whole composition to be around 3-4 minutes in duration. Over the next few months I will spend time researching how this will be projected with sound within the exhibition space. This is a vital element to the production of the piece.

I have discovered the work ‘Gesture Co-ordinates' (2006) by Max Kazemzadeh who takes audience gestures and then translates them into digital codes that provide audio visual feedback to the viewer. As the audience moves around the space, their gestures are turned into pixel patterns on a screen which is overlaid with their projected image. This relates to my project in that human gestures are being encoded into a new form. However, in this interactive situation, the viewer cannot help but modify their gestures to ‘produce’ certain effects with the animation. I want to expose unconscious gestures and let the audience perceive their own emotional response from them as with the music I compose. There is sound in this work, however it performs the function of audio wallpaper accessory to the visually encoded material.

Another interesting Kazemzadeh work is 'Dial-logs' where "sound and image are crossed through a file conversion process to explore relational patterns of digital identity.
Quote from http://www.maxkazemzadeh.com/: Mouse and Man (Mickey and Arnold) takes the image of Mickey Mouse and Arnold Swartzenaegger and converts them to individual sound files. And conversely, the digitally recorded words "Mickey" and "Arnold" are converted from two sound files to two separate images. Those images are projected over sillohuettes of the iconic figures where drawings are made using only those digitally converted sound file artifacts that overlap the drawn contour of the figure. Clear audible relationships can be made from the sound to its respective image.

Having not actually heard the sound file it is difficult to comment on this work but I think the idea of turning sound to picture and vice versa is very interesting.