There’s a lot of technology in the show. In particular, most of the set is projected, and we are using a Microsoft Kinect to track the actors on stage, and modifying the projections based on their location.

I’m working on Linux, and using OpenNI for interfacing with the Kinect. Things almost worked perfectly. In this post I will document the trials and tribulations of getting the Kinect to work for Half Real.

I often fall into Not Invented Here Syndrome, and so slowly I’m trying to get out of it. Obviously, interfacing with hardware like the Kinect is not something I really wanted to do during a 3 month theatre development. My Spatial Augmented Reality framework is built on Linux, so I basically had the option of Libfreenect or OpenNI. OpenNI appears to be more mature, and so that’s what I went with.

As you can see, I’m only really tracking the position of the actors – we aren’t using any of the gesture recognition stuff.

During development everything looked peachy. However, during production week when we started running through the whole show, a major issue popped up. It turns out there is a bug buried deep in OpenNI that eventually rears its ugly head if you have a few people running around at the same time:

Program received signal SIGSEGV, Segmentation fault.
 0x00007ffff215574d in Segmentation::checkOcclusion(int, int, int, int)

This is a big problem. See, this is a theatre show, where the entire set is projected. If the system crashes, the stage goes black. The operator has to restart and bring the projections up to the right point in the show. It turned out that in our tech previews, the software was crashing 2-3 times per show. This was simply unacceptable.

Thankfully, I was only interested in the positions of the actors. This meant I could run the tracking in a completely different process and send the data to the projection system without too much overhead. So, on the day before I finished working for the project, I had to completely rewrite how the tracking worked.

The Data We Need

As I said, we only need position. I didn’t have to send through any camera images, gesture information, etc. All I needed was:

struct KinectMessage
{
    uint8_t actor_id;
    float   quality;
    float   x;
    float   y;
    float   z;
};

The process that interfaces with the Kinect simply sent these messages over a TCP connection to the projection system for every actor on stage. TCP worked pretty well. Both processes run on the same system, and the Kinect only updates at 30fps anyway. So you know, there’s only 510 bytes per second, per actor that needed to be transferred. If I was transferring images, a better IPC technique would be required.

While True

At this point, the hard work was done. Simply wrap the tracking process in a shell script that loops forever, rerunning the process when the segfault occurs. The projectors never go to black, and the worst case is the tracking lags for a a couple of seconds. Not perfect, but infinitely better.

I guess the moral of this post is to be wary of relying on 3rd party libraries that are not particularly mature. And if you have to (you don’t have much choice if you want to talk to the Kinect), wrap it up so it can’t screw you over. TCP worked for me, because I didn’t need to transfer much data. Even if you were doing the skeleton tracking and gestures, there isn’t a lot of data to send. If you need the images from the camera, TCP may not be for you. But there are plenty of other IPC techniques that could handle that amount of data (even pipes would do it). I guess the good news is OpenNI is Open Source, so in theory someone can get around to fixing it.

Hope this helps someone.

Michael

So right now I am at the IEEE Symposium on 3D User Interfaces in Singapore. We have a couple of publications which I’ll be posting over the next few days. First up is Adaptive Color Marker for SAR Environments. In a previous study we created interactive virtual control panels by projecting onto otherwise blank designs. We used a simple orange marker to track the position of the user’s finger. However, in a SAR environment, this approach suffers from several problems:

• The tracking system can’t track the marker if we project the same colour as the marker.
• Projecting onto the marker changes it’s appearance, causing tracking to fail.
• Users could not tell when they were pressing virtual controls, because their finger occluded the projection.

We address these problems with an active colour marker. We use a colour sensor to detect what is being projected onto the marker, and change the colour of the marker to an opposite colour, so that tracking continues to work. In addition, we can use the active marker as a form of visual feedback. For example, we can change the colour to indicate a virtual button press.

I’ve added the publication to my publications page, and here’s the video of the marker in action.

Hi Everyone

The guys over at Packt Publishing have kindly asked me to review a new book on programming with OpenSceneGraph. I haven’t looked at the book yet, but will get the review done sometime this week. In the mean time, checkout the book:

http://link.packtpub.com/NwneHs

Multiple X Screens

The easiest way to do dual screen output is if you have your X server configured so each output is its own X screen. The first thing you need to do is make sure you have enough screens. Finding this out is easy enough:

int getNumScreens()
{
    osg::GraphicsContext::WindowingSystemInterface* wsi = osg::GraphicsContext::getWindowingSystemInterface();
    osg::GraphicsContext::ScreenIdentifier si;
    si.readDISPLAY();
    return wsi->getNumScreens(si);
}

You should do this before attempting to create your screens to make sure the X server is configured correctly. Otherwise OSG will throw an error when you try and create a graphics context for a screen that doesn’t exist. Setting up the viewers is fairly straight forward:

//Create main scene viewer
 
ref_ptr<osgViewer::CompositeViewer> compositeViewer = new osgViewer::CompositeViewer;
 
// create first view
osgViewer::View* v0 = new osgViewer::View();
v0->setUpViewOnSingleScreen(0);
 
// add view to the composite viewer
compositeViewer->addView(v0);
 
// do the same with the second view
osgViewer::View* v1 = new osgViewer::View();
v1->setUpViewOnSingleScreen(1);
compositeViewer->addView(1);

And thats it. You also need to set the scene data for each of your views and a couple of things I’ve missed, but that is the basic idea. The problem with this method is it creates 2 graphics contexts, which in most cases will cause a performance hit.

Single X Screen, Single Context

I looked into this method because I use TwinView on my Linux desktop boxes. Of course, TwinView means that there is only 1 XScreen that spans both monitors. Therefore, the getNumScreens function above will return 1. I have also set up our projector setup to use TwinView, so I don’t need to have one lot of code for testing on the desktop and then do something completely different when using the projectors. The other benefit of this approach is you only create one graphics context.

First thing we do is get the dimensions of the screen, which will be the combination of both screens.

// get the total resolution of the xscreen
osg::GraphicsContext::WindowingSystemInterface* wsi = osg::GraphicsContext::getWindowingSystemInterface();
unsigned width, height;
wsi->getScreenResolution(osg::GraphicsContext::ScreenIdentifier(0), width, height);

Once that is done we can create our (single) graphics context.

// create a context that spans the entire x screen
traits->x = 0;
traits->y = 0;
traits->width = width;
traits->height = height;
traits->windowDecoration = false;
traits->doubleBuffer = true;
traits->sharedContext = 0;
traits->overrideRedirect = true;
osg::ref_ptr<osg::GraphicsContext> gc = osg::GraphicsContext::createGraphicsContext(traits.get());

The important one here is overrideRedirect. Some window managers (I’m looking at you Gnome) will redirect the position of your graphics context, so it won’t appear where you want it. The overrideRedirect option is kindof new, it does not exist in the version of OSG shipping with Ubuntu 8.10. Therefore, I am running the latest stable release (2.6) compiled from source.

To get the equivalent of 2 screens to draw on, we create 2 views like before. However, we have to set their viewport manually. Here we just make v0 use the left half of the screen, and v1 use the right half. Easy enough?

//first screen
osgViewer::View* v0 = new osgViewer::View();
 
osg::ref_ptr<osg::Camera> cam = v0->getCamera();
cam->setGraphicsContext(gc.get());
cam->setViewport(0, 0, width/2, height);
compositeViewer->addView(v);
 
//second screen
osgViewer::View* v1 = new osgViewer::View();
osg::ref_ptr<osg::Camera> cam2 = v1->getCamera();
cam2->setGraphicsContext(gc.get());
cam2->setViewport(width/2, 0, width/2, height);
compositeViewer->addView(v1);

setViewport sets the viewport of the camera. The first 2 parameters are the position in the context’s window, the next 2 are the dimensions. So, each view gets width/2 for the width, and the second screen’s position is offset by half the screen width meaning it starts on the second monitor.

Conclusion

And there you have it. Two methods for dual screen using OpenSceneGraph. Looking at the code, it is fairly simple. However, after browsing the doxygen docs for OSG it was not at all obvious to me. Of course, the osg-users mailing list was a big help here. In fact, here is the thread from the mailing list

Cheers
Michael

Note that this video is 1024×768, so hit the full screen button to get more readable detail.

Notes:

The 3DBuzz C++ videos include the following code:

main() {
   cout << "Hello World" << endl;
}

This code fails to compile on newer versions of Visual C++ and GCC. All that needs to be done is to specify that main will return an int, and then actually return one:

int main() {
   cout << "Hello World" << endl;
   return 0;
}

Of course, whether this actually needed a video is debatable!