There’s a lot of technology in the show. In particular, most of the set is projected, and we are using a Microsoft Kinect to track the actors on stage, and modifying the projections based on their location.

I’m working on Linux, and using OpenNI for interfacing with the Kinect. Things almost worked perfectly. In this post I will document the trials and tribulations of getting the Kinect to work for Half Real.

I often fall into Not Invented Here Syndrome, and so slowly I’m trying to get out of it. Obviously, interfacing with hardware like the Kinect is not something I really wanted to do during a 3 month theatre development. My Spatial Augmented Reality framework is built on Linux, so I basically had the option of Libfreenect or OpenNI. OpenNI appears to be more mature, and so that’s what I went with.

As you can see, I’m only really tracking the position of the actors – we aren’t using any of the gesture recognition stuff.

During development everything looked peachy. However, during production week when we started running through the whole show, a major issue popped up. It turns out there is a bug buried deep in OpenNI that eventually rears its ugly head if you have a few people running around at the same time:

Program received signal SIGSEGV, Segmentation fault.
 0x00007ffff215574d in Segmentation::checkOcclusion(int, int, int, int)

This is a big problem. See, this is a theatre show, where the entire set is projected. If the system crashes, the stage goes black. The operator has to restart and bring the projections up to the right point in the show. It turned out that in our tech previews, the software was crashing 2-3 times per show. This was simply unacceptable.

Thankfully, I was only interested in the positions of the actors. This meant I could run the tracking in a completely different process and send the data to the projection system without too much overhead. So, on the day before I finished working for the project, I had to completely rewrite how the tracking worked.

The Data We Need

As I said, we only need position. I didn’t have to send through any camera images, gesture information, etc. All I needed was:

struct KinectMessage
{
    uint8_t actor_id;
    float   quality;
    float   x;
    float   y;
    float   z;
};

The process that interfaces with the Kinect simply sent these messages over a TCP connection to the projection system for every actor on stage. TCP worked pretty well. Both processes run on the same system, and the Kinect only updates at 30fps anyway. So you know, there’s only 510 bytes per second, per actor that needed to be transferred. If I was transferring images, a better IPC technique would be required.

While True

At this point, the hard work was done. Simply wrap the tracking process in a shell script that loops forever, rerunning the process when the segfault occurs. The projectors never go to black, and the worst case is the tracking lags for a a couple of seconds. Not perfect, but infinitely better.

I guess the moral of this post is to be wary of relying on 3rd party libraries that are not particularly mature. And if you have to (you don’t have much choice if you want to talk to the Kinect), wrap it up so it can’t screw you over. TCP worked for me, because I didn’t need to transfer much data. Even if you were doing the skeleton tracking and gestures, there isn’t a lot of data to send. If you need the images from the camera, TCP may not be for you. But there are plenty of other IPC techniques that could handle that amount of data (even pipes would do it). I guess the good news is OpenNI is Open Source, so in theory someone can get around to fixing it.

Hope this helps someone.

Michael

I’ve spent all this morning trying to talk to the Microsoft Kinect using OpenNI. As it turns out, the process is not exceptionally difficult, it’s just there doesn’t seem to be any up to date documentation on getting it all working. So, this post should fill the void. I describe how to get access to the Kinect working using Ubuntu 11.04, OpenNI, and NITE.To talk to the Kinect, there are two basic parts: OpenNI itself, and a Sensor module that is actually responsible for communicating with the hardware. Then, if you need it, there is NITE, which is another module for OpenNI that does skeletal tracking, gestures, and stuff. Depending on how you plan on using the data from the Kinect, you may not need NITE at all.

Step 1: Prerequisites

We need to install a bunch of packages for all this to work. Thankfully, the readme file included with OpenNI lists all these. However, to make life easier, this is (as of writing) what you need to install, in addition to all the development packages you (hopefully) already have.

sudo apt-get install git build-essential python libusb-1.0-0-dev freeglut3-dev

There are also some optional packages that you can install, depending on whether you want documentation, Mono bindings, etc. Note that on earlier versions the install failed if you didn’t have doxygen installed, even though it is listed as optional.

sudo apt-get install doxygen graphviz mono-complete

Step 2: OpenNI

OpenNI is a framework for working with what they are calling natural interaction devices.Anyway, this is how it is installed:

Check out from Git

OpenNI is hosted on Github, so checking it out is simple:

git clone https://github.com/OpenNI/OpenNI.git

From there, change into the Platform/Linux-x86/CreateRedist directory, and run the RedistMaker script. Note that even though the directory is named x86, this same directory builds 64 bit versions just fine. So, don’t fret if you’re on 64bit Linux.

cd OpenNI/Platform/Linux-x86/CreateRedist
./RedistMaker

The RedistMaker script will compile everything for you. You then need to change into the Redist directory and run the install script to install the software on your system.

cd ../Redist
sudo ./install.sh

Step 3: Kinect Sensor Module

OpenNI doesn’t actually provide anything for talking to the hardware, it is more just a framework for working with different sensors and devices. You need to install a Sensor module for actually doing the hardware interfacing. Think of an OpenNI sensor module as a device driver for the hardware. You’ll also note on the OpenNI website that they have a Sensor module that you can download. Don’t do this though, because that sensor module doesn’t talk to the Kinect. I love how well documented all this is, don’t you?

The sensor module you want is also on GitHub, but from a different user. So, we can check out the code. We also need to get the kinect branch, not master.

git clone https://github.com/avin2/SensorKinect
cd SensorKinect

The install process for the sensor is pretty much the same as for OpenNI itself:

cd Platform/Linux-x86/CreateRedist
./RedistMaker
cd ../Redist
sudo ./install.sh

On Ubuntu, regular users are only given read permission to unknown USB devices. The install script puts in some udev rules to fix this, but if you find that none of the samples work unless you run them as root, try unplugging and plugging the Kinect back in again, to make the new rules apply.

Step 4: Test the OpenNI Samples

At this point, you have enough installed to get data from the Kinect. The easiest way to verify this is to run one of the OpenNI samples.

cd OpenNI/Platform/Linux-x86/Bin/Release
./Sample-NiSimpleViewer

You should see a yellow-black depth image. At this point, you’re left with (optionally) installing the higher level NITE module.

Step 5: Install NITE (optional)

Firstly, you need to obtain NITE. For some reason, the link on PrimeSense’s homepage results in a 404 error. After quite a bit of searching around yesterday, I found the correct link:

http://www.openni.org/downloadfiles/opennimodules/openni-compliant-middleware-binaries/33-latest-unstable

Download either the 32 or 64 bit version, depending on your platform. As of writing they only have versions for Ubuntu 10.10, but these seem to work fine with 11.04 as well.

Extract the archive, and run the installer:

sudo ./install.sh

At some point, you will be asked for a license key. A working license key can be found just about anywhere on the Internet. I don’t think PrimeSense care, or maybe this is a non-commercial license or something. But whatever, just copy that license into the console, including the equals sign at the end, and NITE will install just fine.

Conclusion

After following these steps, you will be able to write programs that use the Microsoft Kinect through OpenNI and NITE middleware. I hope this helps someone, because I spent a lot of time screwing around this morning trying to get it all to work. Like I said, the process is pretty straight forward, it just hasn’t been written down in one place (or I suck at google).

To be honest, I am not all that impressed with 10.4. Why you ask?

Font rendering in Firefox is broken

The first thing I notice with fresh Linux distribution is both the choice of default fonts, and the default font rendering. This is no problem, Gnome’s appearance dialog lets me easily configure anti-aliasing and sub-pixel smoothing how I want. There’s just one problem: Firefox completely ignores these settings!

To be more specific, it ignores the font smoothing settings and does it’s own, dog ugly version. There is a bug report for this here. By the looks of things, there is a workaround, but I haven’t tried that yet. The workaround involves removing config files in /etc, and running a console command to reconfigure fontconfig.

This is a ridiculous oversight when Firefox is one of the flagship applications shipping with Ubuntu.

Update Manager thinks it’s the most important app on the system

This technically was introduced in an earlier version, I’m just surprised they haven’t changed the default behaviour by now. See, Update Manager pops up it’s main window if it finds updates it thinks you should install, on top of whatever you happen to be doing. This is both intrusive, and annoying. I don’t have any data, but I’m willing to bet users are more likely to dismiss the dialog instead of installing updates when it gets in their way.

This is a similar problem with Windows deciding it needs to reboot after an update, nomatter what you think to the contrary. I understand the Ubuntu devs are worried that the bright red icon in the notification area isn’t enough to get people to click. However, putting it in their face is even worse, as Windows has shown with the Cancel/Allow UAC dialogs. People don’t read dialogs! A much better solution would be to put the lovely new notification system to use. Just a subtle notification message every 10 minutes or whatever would be much more affective, without getting in everyone’s way.

Nautilus’s Address Bar is missing!

What the hell!? Stop with this Apple induced breadcrumb navigation nonsense. I want to type a path into an address bar. In previous versions of Ubuntu, there was a button to toggle between breadcrumbs and textual address bar. In 10.4, that button is gone. The only solution is to enter the following into a console:

gconftool-2 --type=Boolean --set /apps/nautilus/preferences/always_use_location_entry true

Of course, you need to enter a similar command to get back to breadcrumbs. Here’s what I really don’t understand: Ubuntu is a distribution that specifically tries to remove the need to go to the command prompt. By removing a single toggle button, they have removed the ability to enter paths directly, something that most of the people migrating to Ubuntu (Windows users) have always been able to do. Just put the button back.

My iPod doesn’t show up in Banshee

Arg! I’ve been having this problem intermittently for a while. I also admit that Banshee isn’t a program installed by default. But I don’t like Rhythmbox, and would really like Banshee to work again.

Window controls on the left

I hate this on Macs, and I hate it here too. Please give an easy option in Appearance Properties to flip the location of the buttons. Luckily, New Wave, my theme of choice still has the controls on the right.

The mish-mash 10.4 theme

I love the look of Ubuntu in 9.10. The grey was a cool getaway from the orange. 10.4 kind of builds on this, and in other ways pulls it in another direction. Now there’s added purple!. The login screen is the ugliest Ubuntu login screen ever. The default Ubuntu theme seems to be a mashup of three different themes: The cool black stuff from 9.10, the new purple stuff, and the old orange Human theme. Pick one, and refine it please.

Still need to open gconf-editor to get Computer, Home, Trash icons on desktop

If you aren’t familiar with gconf-editor, it is basically regedit for all of Gnome’s settings. Apparently in Ubuntu 10.4, you still need to use this to get icons such as Computer, Home, and Trash to appear on the desktop. This is incredibly confusing to new people (I just happened to remember what keys needed to be set). In my opinion, opening gconf-editor is worse than using the console, something the Ubuntu UI guys are dead against.

While we’re on the topic, the way you change the massive icon size on the desktop is to edit the Default Zoom Level in Nautilus’s settings. This is also incredibly unintuitive.

Is there anything good?

I like getting new versions of all the programs I use every day. Again, Ubuntu 10.4 doesn’t disappoint here. Everything is nice and fresh. There’s nothing really surprising here.

Conclusion

This is my least favourite default Ubuntu install since 6.06. I feel some of these things are so obvious it’s almost strange mentioning them. I’ll again point out that Ubuntu is my OS of choice for actually getting work done. I have a Windows virtual machine for the sole purpose of running Office and Photoshop. I use Ubuntu every day, and really want to see it get better. I speak up, because I care. So with that, please keep the flaming to a minimum, and sorry about the rant.

Michael

Note that this video is 640×480, so hit the full screen button to get more readable detail.

Notes

I cannot believe I messed up the remote path that many times! I am sorry about that. Of course, the side effect is you learnt how to both add and remove remotes. So yeah… I did that on purpose.

At this point, we can create a project or join an existing one. We can add files and commit changes. We can push our changes to remotes, and fetch changes that other people have made. This is all great. However, we haven’t touched on one of Git’s most powerful features. I am of course talking about branching and merging. This will be the priority for the next video.

Again, any feedback is much appreciated.

]]> http://www.20papercups.net/tutorials/git-vtm03-pushing-pulling/feed/ 3 http://www.20papercups.net/tutorials/git-vtm02-the-basics/ http://www.20papercups.net/tutorials/git-vtm02-the-basics/#comments Sat, 28 Mar 2009 08:59:45 +0000 Michael http://www.20papercups.net/?p=172 This video hits up the terminal and starts using git. We start by creating a git repository, which is super easy with Git. Then, we look at adding files to the repository, and commiting changes.

Note that this video is 640×480, so hit the full screen button to get more readable detail.

Notes

In this video, I specifically create a project using:

git init

. If you are working on your own projects, this is how you will work. However, a lot of the time you will be starting work on an already existing project. In this case, you don’t create a new repo using git init, instead you clone an already existing repository. The basic is this:

git clone path/to/repository

I will cover this in a later video, but I think it is important enough to mention here.

Another thing that I could have covered in more detail is the staging area. There is actually a lot more that you can do with it than I mentioned in this video. However, that will come later. For the first 3 videos (at least) I want to just get the basics down.

Note that this video is 640×480, so hit the full screen button to get more readable detail.

Notes

Git is a distributed version control system, originally written by Linus Torvalds for Linux Kernel development. It is Free and Open Source software. You can get it for Linux, Mac and Windows (either native or cygwin). There are also plugins for Eclipse, etc, so you don’t have to work from a terminal like me.

This video just tries to convey what makes working with a distributed version control system different and better. Nothing too strenuous just yet. If you are a Git guru, I am aware that I am glossing over some details in this video. This is intentional, and I hope to cover some of these details in later videos, or at least in the accompanying text. Still, I welcome any and all feedback and suggestions.

Also, I apologise in advanced for my terrible tablet handwriting skills.

Anyway, today I thought I would distract everyone from my lack of videos by pointing you to some videos made by someone else. And so I present: Martin’s Linux VTM.

These videos are aimed at the Linux beginner, but they very quickly get into great depth. I am still getting my head around some of the more advanced Bash scripting videos. There is actually two releases of videos, so here are the links:

Release 1

Release 2

Enjoy!

Michael