Archive | Music RSS feed for this section

Battle of the DJs: an HCI perspective of Traditional, Virtual, Hybrid and Multitouch DJing

16 Apr

To be presented at NIME 2011 (New Interfaces for Musical Expression, Oslo ). The remainder of the program is very interesting, so please feel free to look around here.

What about DJing?

How does it equates within an HCI perspective?

What forms of interaction exist with DJ gear?

How can one classify those interactions/gear?

The following paper addresses these questions, trying to create a framework of through concerning DJ interactions – a very idiosyncratic type of “music interfaces” – and proposes an evaluation of Traditional (analogue turntables, mixers and CD players), Virtual (software), Hybrid (traditional and software in synergy) and Multitouch (virtual and traditional in synergy).

The term multitouch here defines not a categorization but rather an implementation of a touch sensing surface DJing prototype. Explorations among new interfaces for DJing has already been hapenning for at least 10 years, but an evaluation from such perspective (interactions and devices) can aid researchers and developers (as well as DJs) to understand which tools fit in which style. 

Thesis Defence

30 Oct

My thesis defence is scheduled for 3th November, 12h, IST – Alameda.

midi1_s

(Abstract)

Disc-jockeys have come a long way, through technological evolutions. This path led them to the status and recognition they have achieved in our society. But as impressive as those technological evolutions are, as far as DJing is concerned, there are still few applications that support hands-on interaction over virtual DJ applications, and those are typically reduced to the traditional input devices.

Furthermore, recent proposals apply new DJ metaphors, but are not successful in maintaining coherence with traditional DJ lexicon of gestures and concepts, thus requiring an extensive learning period.

Accounting user feedback from an accompanying group of DJ experts, we devised a multitouch solution that maps physical core DJ concepts into a virtual application. In the proposed solution, gestural interaction from Traditional Setups remains coherent while inheriting advantages from the virtualization of the DJing domain. The system addresses typical requirements of contemporary DJing, as identified by our research, namely: robustness, low-latency, external control, adaptability, audio plugins and connectivity standard-compliance.

Additionally, we support task improvements, such as dynamic re-routing of music flow, seamlessly merging edit with DJ-performance mode and rearrangement of interface layout according to user needs. Our system allows DJs to exercise creativity with a natural interaction, creating scenarios that are not possible in the real world.

Comparison against Traditional, Virtual and Hybrid DJ setups showed how a multitouch DJing surface developed with DJ involvement, can suit experienced users changing to Virtual setups.

User tests are over!

14 Jul

User tests took about one entire week, with a total of 10 DJs travelling to the Taguspark facility to test the MtDjing interactive surface. The results have been analyzed and a paper is being cooked as we speak.

joao e lopes

Mush Von Namek using MtDjing prototype (photo credit: Vanessa)

Physical simulation: turntable motion#1!

5 Jul

Entering the world of physics…. is always a bit tough. Although simulating a turntable mechanism is not a very tricky mechanism to mimic (I’m currently just copying the inner works of a direct-driven motor). The AS3 physics engine is Box2Das3 which is a direct port of the Box2d original one, written in C++.

nha(this object already behaves pretty much like a moving motor that can be applied with force(s) from the user interaction)

After laying out a few drawings for understanding the forces in the equation, my first few attempts generated a quite interesting result, although there’s a lot of configuration to do to allow the torque to behave like a turntable model motor, pitch adjust, normalize velocity, compensate brake power, shutdown feature, etc… (and maybe include a platter for fine tuning the speed…)

DSC04017

DSC04018

DSC04019(click on any photo to enlarge the sketch)

Allowing DJs to share their patches!

3 Jun

Multitouch Djing: Loading Objects via XML from PedroLopes on Vimeo.

This is just showing the Objects in the interface being XML defined, that can be read at start up and create the patch for the DJ.

This will allow users of the interface to share patches, save, load, etc…

About the interface concepts

28 Apr

After having some mind struggles with the interface concepts, its now more or less defined:

Realtime Setup Creation

The concept is similar to the pacthing programming languages (PureData, Max/MSP) and pacthing interfaces such as Reactable, AudioPad, BlockJam, etc… thus the user is able to build his own DJ system has he plays. The video below is a mock up that shows the creation of a couple of objects on the interface.

Interface Concepts MockUp: connecting stuff from PedroLopes on Vimeo.

This is the mock up / draft version of the idea of dynamic pacthing. I use the reacTiVision TUIO simulator to send OSC-TUIO commands to the Interface layer, thus controlling the objects.

The objects are: movable, scalable, rotatable and linkable as you see in this mockup.

The need for more objects is rapidly satisfied by dragging and droping a new object from the canvas, the possibilities of connecting it with the current pacth are endless, thus resulting in a expressive virtual system that ultimately empowers the end user.

User Objects

The objects in MtDjing are organized as follows (see the mock up diagram below):

MockUpObjects

Interface Objects – these are the objects that the user manipulates actively within the canvas, in a multitouch fashion. The objects are all related to DJ systems, and strive to be a visual representation that carries such concepts. All of the interface objects can be manipulated by the user: moved, rotated, resized, created, altered and deleted in runtime. They are organized in classes:

Sound Sources – are the objects that output sound, such as a virtual turntable.

DSP Objects – these object-types can perform some sort of audio manipulation, thus they have at least one audio     input and one audio output. A simple example is a sliding fader, that can control the sound volume.

Audio Cables – the links between the objects can be changed in realtime by the DJ-user, by connecting objects with a virtual audio cable, that will let the audio flow.

Static Objects – are part of the interface static view, such as the “Canvas”, “Side Bar”, the “master audio output”,  etc… these cannot be moved, resized or altered by the user during runtime.

Audio Flow

The connectivity style is derived from typical DJ systems, as we see on typical DJ (actually its derived from stage setups) diagrams, the flow f the audio tend to work in the upwards direction… thus having the sound sources bellow, and the chain of processing goes up until sound is delivered into the “Sound System” (which is slang for sound output).

These diagrams (as we see below) represent the mental model that we expect users to accommodate while interacting, because this is very familiar to them we gain more easy on the interface comprehension phase.

DjsystemFlow

On newer setups, those that make use of MIDI control devices) the devices tend to be  on a lower level than the sound sources (CDs, turntables, etc..) or at least on the same level – in ergonomics terms they are always within “arm reach distance” (a very important concept that our large multitouch interface must capture). The image above ilustrates this concept on a DJ setup.

Object Handling

In multitouch interfaces there is a recursive problem when designing objects that can perform a large set of operations that can interfere with each other. In our case, our dynamic Interface Objects have the following set of operations:

  • Move (free move withing a X-Y canvas)
  • Rotate (constrained rotation in 90º multiples’ angles)
  • Resize (constrained to a maximum/minimum values defined in each object)
  • Link (constrained to the rules on input connects to output and output to input)
  • Sub-set of DJ actions (actions that the object performs: click, slide, whatever…)

So the issue is how to create such objects that are Rotatable-Dragable-Scalable and still allow you to use them (by means of direct touch manipulation)? One possible way to g is using Object Handles as proposed and tested by Nacenta in 2009 [1]. These object handles are shown in a diagram below, from the original article (available to the public).

ObjectHandles

Some issues I will test soon: I will deploy a version of my current prototype with object handles so all of these features can be testes. In proposing that my object handles are the following around the object :Link, Move, Rotate & Scale and in the inside of the object is the sub set of its DJ actions.

[1] Nacenta, M.A., Baudisch, P., Benko, H., Wilson, A. 2009. Separability of Spatial Manipulations in Multi-touch Interfaces. In Graphics Interface, Kelowna, B.C., Canada. 175-182.

Combining Objects

The objects on the interface are of a lower granular-level than the typical DJ setup allows, in the physical world one cannot add an extra channel or EQ knob to our mixer… so in order to build a setup fast, many objects must be created by the user. To speed up the process and to allow some customization the “Group Object” is added to the metaphor. This is a object that allows the creation of sub-pacthes (once again… reminiscent of PureData internal workings as a visual programming language). A Group Object is a visual rectangle area that comprises all objects within its boundaries to a new object, thus we can easily create “presets” such as a two channel mixer with this group concept.

Scratch/Turntablism Ph.D. Thesis!

9 Apr

Hooray! Kjetil Falkenberg Hansen from KTH has defended his doctoral thesis under the topic of turntablism. You migh take a closer look here (or download it here).

Abstract for the Thesis

This thesis focuses on the analysis and modeling of scratching, in other words, the DJ (disk jockey) practice of using the turntable as a musical instrument. There has been experimental use of turntables as musical instruments since their invention, but the use is now mainly ascribed to the musical genre hip-hop and the playing style known as scratching. Scratching has developed to become a skillful instrument-playing practice with complex musical output performed by DJs. The impact on popular music culture has been significant, and for many, the DJ set-up of turntables and a mixer is now a natural instrument choice for undertaking a creative music activity. Six papers are included in this thesis, where the first three approach the acoustics and performance of scratching, and the second three approach scratch modeling and the DJ interface. Additional studies included here expand on the scope of the papers.

For the acoustics and performance studies, DJs were recorded playing both demonstrations of standard performance techniques, and expressive performances on sensor-equipped instruments. Analysis of the data revealed that there are both differences and commonalities in playing strategies between musicians, and between expressive intentions. One characteristic feature of scratching is the range of standard playing techniques, but in performances it seems DJs vary the combination of playing techniques more than the rendering of these techniques. The third study describes some of the acoustic parameters of typical scratch improvisations and looks at which musical parameters are typically used for expressive performances. Extracted acoustic and performance parameters from the data show the functional ranges within which DJs normally play.

Unlike traditional musical instruments, the equipment used for scratching was not intended to be used for creating music. The interface studies focus on traditional as well as new interfaces for DJs, where parameter mappings between input gestures and output signal are described. Standard performance techniques have been modeled in software called Skipproof, based on results from the first papers. Skipproof was used for testing other types of controllers than turntables, where complex DJ gestures could be manipulated using simplified control actions, enabling even non-experts to play expressively within the stylistic boundaries of DJ scratching. The last paper describes an experiment of using an existing hardware platform, the Reactable, to help designing and prototyping the interaction between different sound models and instrument interfaces, including scratching and Skipproof.

In addition to the included papers, studies were conducted of expressivity, description of the emotional contents of scratching, DJ playing activities, and the coupling between playing techniques and sample. The physical affordances of the turntable, mixer and samples, as well as genre conventions of hip-hop, are assumed to explain some of the findings that distinguish scratching from other instrumental sounds or practices.

Prototype#3: Yet flash (better architecture) and PureData

30 Mar

Prototype Flash and PureData #3 from PedroLopes on Vimeo.

This video shows a small-tiny-atom-sized prototype that uses Pd (PureData) and Flex/ActionScript compiled on Win (XP) with only the Open Source Flex SDK.

The only reason for not deploying it in Ubuntu is that flashserver external pd object is not present. I will replace it with OSC and probably use oscx external.

The PureData side is just a small patch that mixes a loop (a funny beat 80’s sample…ta nan…) with a sine wave. To simplify auditive debugging.

Notice that all communication between Pd Flash is all done remotely via Sockets.

The Architecture has now some classes and modules, the turntable mock-up objects that you see inherit from the base InterfaceObject Class. More on this soon…

The scratch mat

16 Mar

 

An interesting project (appearing on The Fun Theory website) is this Scratch Mat:

Take a look!

 

Possibilities#2: Audio Layer

2 Mar

Our desired application is modular and layered, which means apart from the interface module (whose possibilities we’ve explored in the last post)

Here there’s probably one million ways to go, all very different in possibilities and in scope of programming (from low level languages to higher and even visual programming). In this post I explore only a few – due to my own limitations (programming skills, knowledge and time available).

The choices available

Here’s some simple info (just for those that are not familiar with most audio synthesis environments)

Name Primary Purpose(s) License Development status
ChucK
Realtime synthesis, live coding, pedagogy, acoustic research, algorithmic composition
GPL
Immature
Csound
Realtime performance, sound synthesis, algorithmic composition, acoustic research
LGPL
Mature
Impromptu
Live coding, algorithmic composition, hardware control, realtime synthesis, 2d/3d graphics programming
Proprietary
Stable
Max/MSP
Realtime synthesis, hardware control
Proprietary
Mature
nsound
Realtime synthesis, offline audio rendering, algorithmic composition, acoustic research
GPL
Mature
Pure Data
Realtime synthesis, hardware control, acoustic research
BSD-like
Stable
Reaktor
Realtime synthesis, hardware control
Proprietary
Mature
SuperCollider
Realtime synthesis, live coding, algorithmic composition, acoustic research
GPL
Mature
Table1. Comparison of Audio Engines/Environments (taken from Wiki)

Of course some of these are not OpenSource nor Free, so as far as I’m interested they are taken out (namely Reaktor from NI, Impromptu and Max/MSP – the last can be replaced with PureData, which Open Source and Free).

Choosing Free and Open Source alternatives

So our table looks like this:

Name Primary Purpose(s) License Development status
ChucK Realtime synthesis, live coding, pedagogy, acoustic research, algorithmic composition GPL Immature
Csound Realtime performance, sound synthesis, algorithmic composition, acoustic research LGPL Mature
nsound Realtime synthesis, offline audio rendering, algorithmic composition, acoustic research GPL Mature
Pure Data Realtime synthesis, hardware control, acoustic research BSD-like Stable
SuperCollider Realtime synthesis, live coding, algorithmic composition, acoustic research GPL Mature

Measuring the knowledge in each of the possible choices

So our table looks like this:

Name Knowledge Years using
ChucK none none
Csound none
none
nsound none none
Pure Data Medium User
2
SuperCollider Starter (just started)

So from this table its understandable that probably PureData or SuperCollider are the reasonable choices as far as minimizing the learning curve. Let’s take a look at what’s under the hood of each “audio environment”.

Interfacing Protocols and possibilities

See this table

This validates that our possibly selected languages (PureData and SuperCollider) have all the features we need (OSC, MIDI, HID…).

Technical Info on PureData and SuperCollider

Pure Data Mac OS X, Linux, Windows, iPod C C, C++, FAUST, Haskell, Java, Lua, Python, Q, Ruby, Scheme, others
SuperCollider Mac OS X, Linux, Windows, FreeBSD C, C++, Objective C C++ Client-server architecture; client and server can be used independently, command-line access

They both run on various Operating Systems (Ive used PureData on Linux/Win and SuperCollider on Win but I’m installing it to Linux soon).

Getting acquainted

Just a small fact, PureData is often referred as PD Or Pd) and SuperCollider as SC (or Sc#, where # denotes the version).

In depth Analysis!

a) PureData (The open source Max/Msp phenomena)

Historical notes: Pure Data (or Pd) is a graphical programming language developed by Miller Puckette in the 1990’s for the creation of interactive computer music and multimedia works. Pd is very similar in scope and design to Puckette’s original Max program (developed while he was at IRCAM), and is to some degree interoperable with Max/MSP, the commercial successor to the Max language. Both Pd and Max are arguably examples of Dataflow programming languages. In such languages, functions or “objects” are linked or “patched” together in a graphical environment which models the flow of the control and audio.

Advantages: Because it draws from Dataflow programming and is a graphical programming language, Pd is fairly easy to learn, all functions and objects are coupled into a program (called a “patch”, probably because it resembles a patch of cords and strings connecting the objects). External objects are a big potential of PureData (and luckily, due to the wide community of developers, there’s almost every flavor in the world as far as choosing a programing language to write the external objects).  Its a fast language towards audio and can respond to various types of messages (OSC, MIDI, etc..) can be coupled with sockets to make it a remote application (client<->server like).

(my) Opinion: Pd is in a very mature level, with a huge amount of effort form users and developers. It has been evolving seriously into a complete solution towards multimedia (with GEM from graphics and other packages for more esoteric purposes…). I’ve used it to various projects, it has a simple binding with the OSC or TUIO  protocol as I’ve posted in this blog earlier (see post or video below). Also in a disadvantage point of view (as far as my experience) certain OO concepts are hard to use/find in Pd graphical programming language, such as instanciating and manipulating large lists of objects (spawning..etc..).
http://vimeo.com/moogaloop.swf?clip_id=6919702&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1

First Try of PureData and TUIO (homemade multitouch) from PedroLopes on Vimeo.

First Try of PureData and TUIO (homemade multitouch) from PedroLopes [turn up the volume because its very lo

Literature concerning Pd

Pd~graz (ed.), ed (2006). bang. Pure Data. Wolke Verlag Hofheim. ISBN 978-3936000375. http://pd-graz.mur.at/label/book/bangbook.pdf.

Puckette, Miller Smith (2007). The Theory and Technique of Electronic Music. World Scientific Press, Singapore. ISBN 978-9812705419. http://www-crca.ucsd.edu/~msp/techniques.htm.

b) SuperCollider (Fast synthesis client-server engine!)

Historical notes: SuperCollider is an environment and programming language originally released in 1996 by James McCartney for real-time audio synthesis and algorithmic composition. Since then it has been evolving into a system used and further developed by both scientists and artists working with sound. It is an efficient and expressive dynamic programming language which makes it an interesting framework for acoustic research, algorithmic music and interactive programming.

Advantages: Being an interactive and dynamic language it is by far one of its main advantages. Its fairly easy to debug and create programs incrementally. The other advantage comes from the concept behind the SC itself: server-client schema.

In SC you type code in the client, thus creating synths (synths are actually DSP blocks that can generate or process sound) that are executed in the server. Client <-> Server metaphor uses OSC as the primary medium (which makes SC an interesting tool for our purposes).

The Language has certain advantages because it supports methods of reflective, conversational and literate programming,

Finally and this is important: Sc is suitable for realtime audio, probably over performing PD (I will test before I can assure it) and a number o other interfaces are supported (see this entry of SCWiki) making it easier to develop.

Disadvantages:  Well the main disadvantage (for my programming skills point of view) is that SC language is a bit tough to learn in few time (like most complex langs are).

Literature on SuperCollider

James McCartney: SuperCollider: a new real time synthesis language ICMC96

David S. Sullivan Jr., Stephan Moore, and Ichiro Fujinaga: Realtime Software Synthesis for Psychoacoustic Experiments ICMPC98

Platform Independency

All of the proposed solutions here are software independent, this is of course a mandatory detail that we must fulfill in order to achieve maximum portability/modularity.

Testing SC and Pd

http://vimeo.com/moogaloop.swf?clip_id=9732948&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1

Prototype for Flash to PureData (pd) from PedroLopes on Vimeo.

Well, as you can see above the prototype already tested Pd for the development of a small 2 channel Dj mixer, so now I have to do the same prototype with SuperCollider and analyze the results. A comparison will be posted soon!