[December 19th 2005]


Screenshot from video documentation of 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

'How long does the subject linger on the edge of the volume.'
- An Interview with Marc Downie

'How long does the subject linger on the edge of the volume.' is a live interactive imagery for dance theatre by Marc Downie, Paul Kaiser and Shelley Eshkar in collaboration with the famous New York based choreographer, Trisha Brown. The piece was rewarded with an 'Award of Distinction' at the Ars Electronica festival 2005. Emil Bach Soerensen and Tatiana Lyng met with Marc Downie for a talk about the creative meeting between dance and live computing. Marc is an artist and artificial intelligence researcher and recently finished his doctorate at the MIT Media Lab. A version of the live performance, with the Trisha Brown Dance Company, can be experienced next time at Palais Garnier in Paris, January 6-8, 10, 2006.

Emil Bach Soerensen is a student of visual culture, Tatiana Lyng works as a media artist and they are both members of the Danish exhibiting group Maskinstorm.

Emil: Well, congratulations with the 'Award of distinction' that you, Paul and Shelly won for 'How long does the subject linger on the edge of the volume.' We are now standing in front of a video documenting the performance. Maybe you should introduce us to your work.
The piece was two and a half years in the making and it is sponsored by Arizona State University that built a motion capture lab in order to obtain state of the art motion capture studies. They thought tha t in order to investigate how to use this stuff to look at human motion, they would find some complicated human motions. Conveyers of complicated human motions are choreographers, so the university made the decision to create two pieces with choreographers. What I think is a bit unusual is that they decided at the outset to finance the project for three years, and the premiere was fixed in advance.

Three years was very generous, and gave us a long time to work out how to handle the data from an engineering perspective, and it also allowed us to have two and a half years in the theatre with Trisha. We met for five one-week long workshops, which was a wonderful way of working. The reason why we took this project is that we had such a long period of time.

All of my works are computed live, a technique that I have been developing over a number of years. The project, 'How long does the subject linger on the edge of the volume.', seemed to provide a frame, that would actually use live computation l egitimately in this domain.


Screenshot from video documentation of 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

Emil: Try to describe what we see here on the video documentation, for somebody that hasn't experienced the performance.
In the theatre we have a proscenium stage and standard stage light. In front of the stage there is a transparent scrim (a type of transparent screen, ed.), which the audience can't see because there is no light on it. Hidden from view are 16 infrared cameras surrounding the stage. Our visual imagery, which was computed live, was projected on the scrim. The job of the cameras is to capture the movement of the dancers and work out exactly where they are in that moment. Motion capture hardware can do an excellent job of this - quickly and accurately. This means, that I can choose locations in the audience where the imagery lines up precisely with the dancers. I can connect the physical form, which is moving, with the architectural form of the space.

Emil: Please elaborate on the relations between the position of the audience and the perceived imagery.
Everything is calculable. I know where my scrim is and where my projector is, so when I get data indicating that a dancer is at a specific position in space, I can do the mathematics to draw a point on the scrim that, from a specific seat, aligns precisely with the dancers on the stage.

Tatiana: How many projectors do you have?
Just one gigantic really expensive projector! And 16 cameras and each dancer has 12 markers on their bodies.

Emil: How does it work with the 12 markers?
There is possibly 48 points out there. Some of them are flickering in and out, so the computer has to work out, how it wants to relate the points to each other, and what relationships are most likely. And then you have to segment those down in order to say; 'well this cluster here must be a dancer if dancers are this big'. You have to figure out when dancers come close to each other that they don't merge into a single dancer. Markers are sometimes also illuminated and disappear from point of view, so the data is frustrating from an engineering point of view. They are, at the same time, extremely accurate but sometimes also hard to handle correctly.

Tatiana: I have seen other pieces capturing the movements of the dancers, but the dancers had to navigate according to markers on the floor. It appeared very structured and it wasn't really an intuitive way of navigating.
We wanted to work with Trisha Brown. We did not want to work with Trisha Brown dealing with some kind of technology, that she was not familiar with. So we tried to make the technological problems disappear as much as we could. But you are right, there are ways of constraining the problems, so that interaction can occur in a way that computers find easier to deal with. But who am I to tell Trisha how to choreograph, in order to maximise the input that the computer can handle. That was never an option that we considered!


Screenshot from video documentation of 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

Tatiana: Can the dancers see the projections on the scrim while performing?
They can't really see the projections. First of all, the imagery is front-projected, which means that all that the dancers get to see is the projector pointing at them. Secondly, the dancers are mentally somewhere else. They are concentrating on their bodies and not watching the feedback.

The piece is interactive so that Shelly, Paul and I can interact with Trisha. It is not interactive in a way that the dancers can choose how they interact with the imagery.

This it not to say, that they weren't interested in the imagery, or never saw the imagery. The dancers came out, and they even put their parts on other people, in order to get off stage and see what it looked like when they were dancing. They were extremely interested in the imagery and it got very personal. We took a solo from one of the dancers and re-projected that over her, so that she could see herself in it. Those were wonderful moments where we gave something back to the dancers. It wasn't an anonymous experience, but a tightly choreographed one.

Emil: How did you experience the creative meeting between the way Trisha Brown works with choreography and the live computing you do?
Let's look at how Trisha worked as a choreographer in this piece. For 'how long.' there is a certain amount of seed material, which the dancers know really well, and then starting from this seed she starts to improvise by setting up problems for the dancers to solve. So they work through it in an open way on the dancers bodies, and then setting the improvisation saying; 'ok that is it, do it again!'

Our goal, as programmers and producers of visual imagery, is to do the same. Can I arrive in the theatre not with a piece but with a certain amount of algorithmic material which I can then extend, stitch together and explore and can we all do this at the same time?

Tatiana: What do you think that the audience watching the performance primarily focus on, the bodies or the graphics? When we are standing here - again we are only looking at a video - my focus is very much on the graphics.
There are two things in favour of the dancers in real space. First, the eye is good at distinguishing objects in depths, and there is a depths difference between the plane that we project on and the dancers. Secondly, the eye is also really good at reading human motion. With the exception of facial gestures the thing that the human visual apparatus is set up to read is human motion. It is not set up for reading marks of visual imagery, so in live performance, with real humans moving, the separation is much more impressive than any video of it. And live you can hear the sounds made by the dancers' bodies, so a video is very unfair to the dancers. However, when my imagery goes from live with a glorious projector to a video it suffers too.


Screenshot from video documentation of 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

Emil: The eyes and the mind are good at perceiving human motion. Can you tell a bit more about that?
There are some great experiments of the 1970s - you film a person in a black room walking or running with white markers on their joints. When you show these moving points to people, it turns out that everybody can say whether it is a male or a female person. People can recognise their own motions, the movement of a friend and obviously also interpret gestures and actions. We can interpret human motion through points, because there is a whole chunk of our brain dealing with this specific task. That is why it makes sense to work with motion capture by capturing points.

Emil: You are talking about the mental perception that is going on in the mind and you investigate artificial intelligence and use the term 'imagery'. I guess 'imagery' here refers to the link between human perception and interpretation, when the audience creates images in their mind responding to the movements of the dancers. How do you work with this mental capacity?
I have two aesthetic sources for the piece. One is in an odd way Trisha's own dance diagrams - she is a visual artist too, and she draws the most incredible diagrams.

The other source is my own experience of trying to work out what is going on in Trisha's choreography, visually. Or rather, it is me being confused about it, not being able to capture it, not being able to hold on to it, before it disappears. In her choreography there is a sort of chasing after a relationship and never quite capturing it. Suddenly something disappears t hat you thought you had a hold of. In a way I am trying to build something, which portrays the experience of watching her choreography.

Emil: That is the essence of the work, isn't it?
Right, that is what I am thinking about when I make these forms. But how did I go about doing that? In the AI research that I do, I actually made things that were chasing after relationships. Very simple biologically inspired artificial intelligences that would go after relationships, seek stability and fail. So, there is a sort of sincere conclusion about AI in this piece; that the AI is somehow broken and not up for the complex task. Watching Trisha's work, there is a similar feeling of inadequacy.

Emil: You are trying to visualise perception and intelligent human behaviour. But I guess you can't stop the human imagination, so you just push it to another level. What do you think is the mental impact of actually visualising human processes?
I don't know what the effect is on the audience, I can't predict that. I am more concerned about conveying a certain intentionality and a certain effort to the graphics, a transience and a fragility to the imagery, that I think are symptoms of reception processes, than actually getting the perception right. I am not presenting a theory of visual cognition for dance or any kind of model. I try to present a whole bunch of inadequate ways to see and experience dance.


Screenshot from video documentation of 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

Tatiana: Can you tell us a bit about the agents?
Sure! I will start from the top. The triangle is an agent that opens the piece and then it returns again half way through. It is playing a fairly simple game, maybe the simplest of the relationships to the choreography. The opening triangle wants to move from stage right to stage left. They can do that by hitching a ride on the dancers. They can draw a line from the edges of the stage to a point on the body of one of dancers and then they get dragged along by that. The question for the agent is its plan for when to attach and when to disconnect its relationship to the dancers. It has to optimize when it does that in order to make progress from one side of the stage to the other. It does that by rehearsal, by being exposed to the choreography as we are making it. As you can see here (on the exhibited visual material that document how the artificial agent develops through rehearsal ed.), it gets better. After a few times through the opening sequence the agent does an increasingly good job of connecting and disconnecting from the dancers.

Tatiana: You were responsible for the programming and the generating of visuals, but did you participate in choosing the dancers and the style of the dancers, or did Trisha do that on her own?
She was responsible for the choreography, but we were collaborating. She saw our stuff as early as possible, we saw her movements as early as possible and we saw the dance together, while it was just material without choreography. So we both declared our sensibility very early on, and then met frequently, but we worked it out in our own separate spaces. You have to leave the decisions of choreography to the choreographer, because she is just so much better at it than me. Working together is about mutual respect and trust.

Tatiana: As a media artist I see my life as a long process. When I make a new piece I develop techniques and perspective s that go beyond the piece itself and point forward to later works. Did you develop something, while working on this piece, which you didn't imagine before, and which you would like to take further in future works?
We worked on this piece for two and a half years and it was fantastic because it gave me the opportunity to build tools for this piece, instead of just making the piece. A whole programming language was invented for two minutes of this work. This way of working functioned very well, it allowed this collaboration to occur, but at the same time it assured that there is way more stuff left out than put in. If you never come back to that, then it is not a sustainable way of working.

I think there is a transition from exploring, getting excited -even calling for that excitement to occur- and to the phase where you are finishing the piece. At that juncture you almost have to promise yourself another work to follow, to allow yourself the freedom to drop material that has been hard won, but doesn't belong in the work.


Digital imagery from 'How long does the subject linger on the edge of the volume.'. Courtesy of the artists.

Tatiana: Are you still interested in working with dancers?
Maybe, I have a tendency of not wanting to do the same thing again, so I don't necessarily want to work with a choreographer for while, but human motion is still thrilling. The next piece will have human motion in it, it probably won't be choreographed and there won't be a dancer. We, meaning myself, Paul and Shelly, are moving back to public artwork installations in our next commission.

Tatiana: Every art form has its own language and its own audience. I think this performance shows how media can reveal new potentials and add a new layer of experience to a dance performance. In what way do you feel that the creative alliance, between a classical art form and new media, is reflected in the reactions of the audience?
I think (I hope?) that this work has an audience that is substantially broader, or at least substantially more varied, than either modern dance or new media. Yet at the same time this piece can be seen in the context of both, and as a bridge between both. Ideally, it shows a deep connection between 'new media' and what choreographers have been doing for decades (not everything about new media is as new as some would have us believe).

Emil: A last question; 'How long does the subject linger on the edge of the volume.' doesn't seem like a usual title for an art performance. How did the title occur?
Trisha claims to have overheard the phrase being said early on in the workshops - an engineer was wondering aloud how long the dancer, (the subject) that he was supposed to be calibrating, was going to remain at the (difficult to capture) edge of the volume of space that the motion capture cameras all cover. You see, the title could have been much longer.