Free Pixel

We have been working on a new machinima/ performance interface for some months and it is about time to post more info about it.

The basic idea of the AR Second Life project is to combine Augmented Reality with the MMO client of Second Life. Augmented Reality is the technology that attempts to blend virtual elements and real world elements together - one recent (not very elaborate) example for AR is Sony’s Eye of Judgment. In our case, the combination of a MMO and AR became possible only when Linden Labs released their client last year. A smart grad student (Tobias Lang) who visited the Augmented Environment Lab here at Tech from the LMU Munich hacked his way into the client and hooked up the AR tracking software we use in our labs for Augmented Reality work. The result is that we can do a live composite of the MMO world of Second Life and a real performance space. A real actor can perform a scene together with a Second Life avatar, live.

This in itself is not a new principle. In fact, it has been done multiple times in the field of AR but there are some new twists to our version. To my knowledge AR has been a very rare in MMO worlds and we think a lot of new opportunities open up with this setup (the question has been briefly discussed over at Terra Nova). Interestingly enough, the original target was a new form of Machinima production. Another interesting detail is, that the system remains very affordable.
The system allows two main forms of AR participation:

1) players can wear a Head Mounted Display and do a typical AR experience. The problem here is that this setup is confined to the range of the trackers

2) or they set up the system with fixed cameras and play against a kind of virtual mirror where you see yourself and the surrounding avatars in a mixed reality view.

A short documentary about the technology is here. We have the HMD tracking set up in the lab but for our weekly tests we usually use the fixed camera system. In the background we have a small “stage” on which actors can perform. They see themselves next to the Second Life content on the large LCD screen and can play with the virtual elements. In the foreground you can see two of our operators as they control the client that runs the camera and the normal Second Life clients that we use to control our avatars in the virtual world.

Fiducial markers are used to calibrate the combination of the real and the virtual environment.

Then, when an actress performs in the physical space she sees herself mirrored in the LCD screen next to a virtual avatar:

The image she sees is something like that:

We have done extensive tests with Improv performers (including Atlanta’s Dad’s Garage and Tech’s own Let’s Try That!) and produced some machinima pieces with it. One of them for the 48hour Second Life film festival - go and search for “museum of the mind” at 48.tv. We got disqualified because we used real live video - too bad. Both might illustrate the new options we unlock with such a technology.
Currently, we have a constant setup of the fixed camera version installed in our gallery space at LCC at Georgia Tech, we have run demos and performances (including one for CNN.com) and we are working on new and different uses at the moment. The project remains a big experiment and we continue to play around with different uses of the platform. Obviously, we can do Machinima in real-time and deliver a composite format that was possible only in post-production so far. But the mixed reality platform also affects the way one could think about online meetings or mixed reality work spaces. Thinking about it, it makes the definition of Machinima maybe even more complicated but it certainly is an exciting new option for virtual performances.

Will try to keep some updates coming as we go along with the experiments. Some of our movies should be on the website, soon.
The project is a collaboration between the Wesley Center the Augmented Environment Lab and the Digital World and Image Group and is driven by Blair MacIntyre, Jay Bolter, Kathryn Farley, and myself (Michael Nitsche). Students active at the moment include Tobias Lang, Florian Schulz, Brian Schrader, Jarryd McCree, Jenifer Vandagriff, and Mindy Drinkuth.