An Interactive Projection Mapped Graffiti Wall

This post explains how I built a projection mapped, interactive graffiti wall for If There Was A Colour Darker Than Black I’d Wear It. This was built in collaboration with Lachlan Tetlow-Stewart. First, here’s a video of the end result:

Let’s break down the project.

  • Multi projector, projection mapped display onto buildings
  • Allow audience members to send SMS messages to tag the building
  • Animate tags onto wall
  • Video elements

That’s the basics. Oh, did I mention we’re projecting out of a van, powered by generator, in rural South Australia?

3D Projection Mapping

My research involves Augmented Reality using projectors; quite convenient. Here we create a simple 3D model of the geometry to be projected onto, and create our content mapped to the 3D environment.

This differs from most projection mapping techniques, where content is produced for a specific projector viewpoint and mapped in 2D. The advantage of 3D projection mapping is content can be created independently of the projectors. Projectors can be added as necessary, or viewpoints drastically changed, without having to re-author content. In our case, it meant we could project onto the buildings every night, without having to get the van or projectors into exactly the right location each time.

Finding known points on the 3D model in the real world
Finding known points on the 3D model in the real world

The 3D models are actually incredibly simple, just enough detail to allow perspectively correct projections. It does, however, require actually measuring the buildings to make sure the 3D model matches the real world. The calibration process involves finding landmarks in the 3D model (corners) in the projector image using a crosshair. From there, maths takes over and we end up with a correctly aligned projection.

This whole project was just an elaborate way to get the phone numbers of audience members. We would be using them later in the show. We played a video telling the audience to text in their tags, which would then appear on the wall.

The computer system received texts using a GSM Modem, as described in my post about SMS and Linux. The handler put the message into a MySQL database, then an incredibly simple web API allowed the projection system to get new messages as they were received.

Alpha mask for animating in new messages

We used animated alpha masks and OpenGL blending to make the texts appear in a pleasing manner. All lengths of time (time spent visible, etc) were randomised to make everything feel a bit more natural.

The result was a compelling experience for the audience. They got something fun to occupy themselves before the show proper started, and we got their phone numbers for use later in the show!

Technical Details

The projection system was a standard desktop computer with 2x Nvidia GTX560 graphics cards, running Ubuntu 12.04. The software was OpenGL and C++, built on top of a Spatial Augmented Reality framework developed in the Wearable Computer Lab during my PhD.

Text rendering was accomplished using FTGL texture fonts. The software generated pool of fonts at different sizes, so the best fitting font could be chosen for messages of different lengths. The generation on startup is important – generating the texture map of an FTGL TextureFont is an expensive process. Changing the size of a font at runtime will give you serious performance problems.

The correct font size had to be calculated for each message. Here’s my incredibly elegant algorithm for this:

void GraffitiSpot::setText(std::string text) {
   mText = text;
   int index = 0;
   while (!willItFit()) {
      index++;
      if (index == fontPool.size()) {
         break;
      }
      else {
         mFont = fontPool[index]; 
      }
   }
}

So basically it looks through the fonts and chooses the biggest font that fits the message in the space available. What does willItFit do?

bool GraffitiSpot::willItFit() {
   string word;
   float currentLineLength = 0;
   float maxWordLength = 0;
   stringstream currentLine;
   stringstream ss;
   ss << mText;
   unsigned numLines = 0;
 
   FTPoint point(0,0);
 
   while (ss >> word) {
      float wordLength = mFont->Advance(word.c_str());
      if (wordLength > maxWordLength) {
         maxWordLength = wordLength;
      }
 
      // if we are at the start of the line, and it fits.. 
      if (currentLineLength == 0) {
         currentLine << word;
         currentLineLength += wordLength;
      }
      // midway through a line, and it won't fit
      else if (wordLength + currentLineLength > mWidth) {
         numLines++;
         point.Y(point.Y() - mFont->LineHeight() * LINE_SCALE);
         currentLine.str("");
         currentLineLength = wordLength;
         currentLine << word;
      }
      // midway through a line, and it does fit
      else {
         currentLineLength += mFont->Advance(" ") + wordLength;
         currentLine << " " << word;
      }
   }
   if (currentLineLength > 0) {
     numLines++;
   }
 
   float heightRequired = mFont->LineHeight() + ((numLines-1) * mFont->LineHeight() * LINE_SCALE);
   return heightRequired <= mHeight && maxWordLength < mWidth;
}

Each graffiti spot has a width and height that limits how much text will fit. This function simulates rendering the text and does some word wrap. FTGL doesn’t do any text wrapping so it’s up to us, and we’re using variable width graf style fonts. We use mFont->Advance() to calculate how many pixels wide a portion of text is. If a word fits on a line, we move to the next one. As soon as a word overflows the width, we drop down a line. We use mFont->LineHeight() to calculate the Y position. LINE_SCALE is just a line height adjustment because we found the default line height to have too much spacing for what we wanted.

If, after simulating rendering the entire message, we have gone beyond the bounds of the graffiti spot, we return false. The function above then tries again with a smaller font. If you wanted to be clever you would do a binary search to speed up finding the optimum font size, but in practice we never hit any performance problems.

Latex Thesis Template

Tweaking your own Latex template for a PhD dissertation is a rite of passage/time waster for most PhD candidates. There are lots of templates around on the Internet too, of varying quality.

I spent a fair bit of time procrastinating perfecting the template used for my thesis. I’ve pulled out all the unnecessary bits and put it up on GitHub. Hopefully some other poor PhD student will find it useful. This template is itself based on styles developed by Peter Hutterer, an earlier PhD student from the Wearable Computer Lab.

This template is particularly suited to students at the University of South Australia. It meets the guidelines specified by the Graduate Research Office. That said, with some adjustments this template should be useful for anybody.

Features:

  • Nice cover page
  • Author’s publications
  • Acknowledgements
  • TOC, List of Figures, Abbreviations, etc.

How to use:

  • Fork and clone the GitHub repository
  • Edit the information in thesis.tex
  • Update images/00/author_sig.png with your own signature
  • Update images/00/uni.png with your university’s logo
  • Add your publications as citations in 00-publications.tex
  • Tweak the styles as necessary
  • Write your damn thesis!

I’m happy to accept Pull Requests for improvements on this template.

Check it out on GitHub!

Happy Writing!

FTGL on OSX

tl;dr; I’ve made a GitHub repo that makes FTGL work on OSX again.

FTGL is a library that makes it super convenient to render TrueType text in OpenGL applications. You can render text as textures and geometry, making it very flexible. There’s just one problem: if you’re using MacPorts or Homebrew on OSX, FTGL doesn’t work! Here’s how to work around it.

FTGL makes use of FreeType to actually render text. In newish versions of FreeType, some of their source files have been moved around and renamed. This is a problem on OSX since, by default, we are on a case-insensitive filesystem. We now have a name clash where both FTGL and FreeType seem to have a file named ftglyph.h. All of a sudden software that uses FTGL will no longer compile because the wrong files are being included!

The fix for this is fairly straight forward. Since FTGL depends on FreeType, FTGL should be modified to remove the name clash. Unfortunately, FTGL seems to have been abandoned, and has not had any updates since 2013. In the bug report linked above I have provided a patch that renames the file and updates references to it. I’ve also created a GitHub repository with the patch applied.

This problem doesn’t show up on Linux because on a case sensitive filesystem like Ext4, the FreeType file is ftglyph.h, while the FTGL file is named FTGlyph.h. No name clash.

So there, uninstall FTGL from MacPorts or Homebrew, clone my GitHub repo, and build/install from source. FTGL will work on OSX once more.

Long term you may want to look at moving away from FTGL in your own software. It is great at what it does, but hasn’t been updated in a long time. It uses OpenGL display lists internally, so will not work on modern OpenGL specs. But at least you can now use it if you need to.

Sending & Receiving SMS on Linux

A little while ago I worked on a mixed media theatre production called If There Was A Colour Darker Than Black I’d Wear It. As part of this production I needed to build a system that could send and receive SMS messages from audience members. Today we’re looking at the technical aspects of how to do that using SMS Server Tools.

There are actually a couple of ways to obtain incoming text messages:

  • Using an SMS gateway and software API
  • Using a GSM modem plugged into the computer, and a prepaid SIM

The API route is the easiest way to go from a programming aspect. It costs money, but most gateways provide a nice API to interface with, and you’ll be able to send larger volumes of messages.

BLACK had a few specific requirements that made the gateway unsuitable.

  1. We were projecting out of a van in regional South Australia. We had terrible phone reception, and mobile data was really flakey.
  2. We were going to be sending text messages to audience members later, and needed to have the same phone number.

So, we got hold of a USB GSM modem and used a prepaid phone SIM. This allowed us to receive unlimited messages for free. However, we couldn’t send messages as quickly as we would have liked.

Modem Selection

There are quite a few GSM modems to choose from. You are looking for one with a USB interface and a removable SIM. GSM modems that use wifi to connect to computers won’t work. You need to be able to remove the SIM because most mobile data SIMs won’t allow you to send or receive SMS messages. The other big requirement is Linux drivers, and Google is really your friend here. The main thing to watch out for is manufacturers changing the chipsets in minor product revisions.

We ended up going with an old Vodafone modem using a Huawei chipset. The exact model I used is HUAWEi Mobile Connect Model E169 It shows up in Linux like this:

ID 12d1:1001 Huawei Technologies Co., Ltd. E169/E620/E800 HSDPA Modem

SMS Tools

SMS Tools is an open source software package for interfacing with GSM modems on Linux. It includes a daemon, SMSD, which receives messages. SMSD is configured to run your own scripts when messages are received, allowing you to do pretty much anything you want with them.

Installation is straight forward on Ubuntu et al:

sudo apt-get install smstools

Next you’ll need to configure the software for your modem and scripts.

Configuration File

The configuration file is a bit unwieldy, but thankfully it comes with some sane default settings. Edit the file in your favourite text editor:

sudo vim /etc/smsd.conf

Modem Configuration

First up you will need to configure your modem. The modem configuration is at the end of the config file, and the exact parameters will vary depending on what modem you have. Let’s have a look at what I needed:

[GSM1]
device = /dev/ttyUSB0
init = AT^CURC=0
incoming = yes
baudrate = 115200

device is where you specify the file descriptor for your modem. If you’re using a USB modem, this will almost allways be /dev/ttyUSB0.

init specifies AT commands needed for your modem. Some modems require initialisation commands before they start doing anything. There are two strategies here, either find the manual for your modem, or take advantage of the SMSTools Forums to find a working configuration from someone else.

incoming is there to tell SMSTools you want to use this device to receive messages.

baudrate is, well, the baud rate needed for talking to the device.

Like I said, there are many options to pick from, but this is the bare minimum I needed. Check the SMSTools website and forum for help!

Event Handler

The other big important part of the config file is the event handler. Here you can specify a script/program that is run every time a message is sent or received. From this script you can do any processing you need, and could even reply to incoming messages.

eventhandler = /home/michael/smstools/sql_insert

My script is some simple Bash which inserts a message into a database, but more on that in a moment.

Sending Messages

Sending SMS messages is super easy. Smsd looks in a folder, specified in the config file, for outgoing messages. Any files that appear in this folder get sent automatically. By default this folder is /var/spool/sms/outgoing.

An SMS file contains a phone number to send to (including country code, but with out the +) and the body of the message. For example:

To: 61412345678

This is a text message sent by smstools. Awesome!

Easy! Just put files that look like this into the folder and you’re sending messages.

Receiving Messages

Let’s have a better look at the event handler. Remember, this script is called every time a message is sent or received. The information about the message is given to your program as command line arguments:

  1. The event type. This will be either SENT, RECEIVED, FAILED, REPORT, or CALL. We’re only interested in RECEIVED here.
  2. The path to the SMS file. You read this file to do whatever you need with the message

You can use any programming language to work with the message. However, it is very easy to use formail and Bash. For example:

#!/bin/bash

#run this script only when a message was received.
if [ "$1" != "RECEIVED" ]; then exit; fi;

#Extract data from the SMS file
SENDER=`formail -zx From: < $2`
TEXT=`formail -I "" <$2 | sed -e"1d"`

From there you can do whatever you want. I put the message into a MySQL database.

Troubleshooting

That’s all you need to write programs that can send and receive SMS messages on Linux. Once you have smsd actually talking to your modem it’s pretty easy. However, in practice it’s also fragile.

The smsd log file is incredibly useful here. It lives in /var/log/smstools/smsd.log

Here are some of the errors I encountered and what to do about them:

Modem Not Registered

You’ll see an error that looks like this:

GSM1: MODEM IS NOT REGISTERED, WAITING 1 SEC. BEFORE RETRYING

This means the modem has lost reception, and is trying to re-establish a connection. Unfortunately there is nothing you can do here but wait or, using a USB extension cable, trying to find a spot with better reception.

Write To Modem Error

An error like this:

GSM1: write_to_modem: error 5: Input/output error

means the software can no longer communicate with the modem. This is usually caused by the modem being accidentally unplugged, the modem being plugged in after the system has powered up, or by an intermittent glitch in the USB driver. To fix this, do the following:

  1. Stop smsd (sudo service smstools stop)
  2. Unplug the modem
  3. Wait 10 seconds or so
  4. Plug the modem back in
  5. Start smsd (sudo service smstools start)

Cannot Open Serial Port

You may see this error:

Couldn’t open serial port /dev/ttyUSB0, error: No such file or directory

This occurs if you started the computer (and therefore smsd) before plugging in the modem. Follow the steps above to fix it.

Conclusion

So there you have it. Follow these steps and you can send and receive SMS messages on Linux, using a cheap prepaid SIM and GSM modem.

In the next post we’ll be looking at exactly what I used this setup for.