Sharing (old) knowledge#2: Starting Multitouch Dev

23 Mar

The following information was written by me one year ago on the Intelligent Multimodal Interfaces course internal forum. Since there are new students in need of fresh info, I reposted it here, make the best of it:

Here it goes: multitouch frameworks, possibilities and simulators – all of them open source + cross platform.

1) Multitouch possibilities
As far as developing MT (short for multitouch) applications you have a wide number of choices, the best way you can move around is stick to standarts (namely OSC and TUIO) because they are the main drive-forces behing MT communities (the biggest is NUI group), if you stick with frameworks or whatever that assures this standards you can find your way among new programs or easily integrate new code (or code from others!) – and new hardware too.

The most important standard is TUIO, which describes a multitouch event. Imagine that a user touches the table (surface, display, whatever) you will receive a TUIO formatted message saying:
– was it a touch or an identified object (these are called fidutial markers)
– position
– acceleration (and many things)
– unique identifier (this is the most important thing apart from the x+y)

With this you can code your application in no time (okay.. a lot of time can be used =P). Here’s the top TUIO software implementations that are availabe (full list here):

I’ve used the C++, Processing, PureData and Flash AS3. They are pretty much the same idea/concept but ported to the idiosyncrasies of each language – you are probably not familiar with either Processing nor PureData but they are programming languages designed for multimedia (Processing is java-derivate language developed by the MIT for graphical designers and PureData is mainly derieved to sound programming but it is a graphical language so there’s no code).

I recommend you base your decision in the following factors:
1 – are you comfortable in this language?
2 – what graphic back-end do you want do use?
3 – how fast should it run?

Because as you know 1) matters a lot… 2) depends on what graphics you want (OpenGL preferably uses C++ or Java, Processing or Pd) while Flash is a different possibility – there’s also a pythonMT (python multitouch framework) and you can join it with pyglet (opengl for python). The 3) is very easy, it goes like this, from fastest to slowest (based on my experience): C++, PureData or Java, Processing or Flash.

So its all up to you, for instance, in my thesis I decided to use Flash AS3 for the interface, because its easy to code (well.. should be) and gives nice graphics with few effort, because it is not an efficient framework I use a C++ core and the audio engine is in PureData (because I’m familiar and its medium-fast) – here’s a video.

2) Multitouch Trackers

These are the core components, they track the video input and determine how many touch points there are. Once again, if you use a standard tracker that can output TUIO messages, you can use all of the frameworks that I’ve mentioned.

The full list of trackers is here, I’d recommend:

  • reacTIVision: a computer vision framework for object tracking and basic multi-touch
  • Community Core Vision: an OpenFrameworks based multi-touch tracker (formerly tbeta)
  • Touchlib: the first free library for multi-touch surfaces based on FTIR and DI (currently discontinued in favor of CCV – see above)
  • gstreamer-tuio: a gstreamer plugin for blob detection, sending TUIO
  • bamboo-tuio: TUIO for Wacom Bamboo (Pen&)Touch tablets
  • Tongseng: a TUIO wrapper for Macbook multi-touch pads
  • Wiimote Whiteboard: platform-independent Wiimote IR tracker supporting TUIO
  • WiimoteTUIO: another application sending the locations of IR sources detected by a Wiimote via TUIO
  • TuioTouch: a simple web based TUIO server

In VIMMI lab we use CCV (Community Core Vision) so I’d recommend this one, and by far it is the simplest to set up and use (just run it..hehe).

3) Multitouch Simulators

As I said, not everyone has a touchtable. So there are simulators, that allow you to simulate TUIO messages with the mouse. Here in this video you can see my resizing some objects on flash, using the Simulator (from Reactivision project) – it is by far the most simple and complete one. But here’s the ones I recommend:

  • Java based standard TUIO Simulator application (platform independent)
  • SimTouch: a TUIO/FlashXML simulator using the Adobe Air runtime
  • QMTsim: Qt based multi-touch simulator alternative (Win32, Linux)

The SimTouch has a nice idea, it is transparent so you can put on top of your running application – but… sometimes it seems to have some bugs. So I’d rather use TUIO simulator from the Pompeu Fabra folks (the ones that developed the Reactable).


There’s a million more things that I could talk, and the text I’ve wiritten is already to long… so If you need something ask sorriso Maybe I can help.


My first posts in the thesis blog show a lot of the decisions and possibilities that you have for Multitouch, I have also a couple of vimeo videos trying some Flash multitouch libs, etc… and playing pong.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: