home  previous   next 
The Seventeenth Annual Interactive Audio Conference
BBQ Group Report: Enhanced Input Method for Glass Tablet Instruments
Participants: A.K.A. "Nightmare Theater"

Ryan Harvey, Cirrus Logic

Neil Whyte, Wolfson Microelectronics
Josh McMahon, Realtek Tom White, MMA
Andrew Rumelt, Cirrus Logic Aaron Higgins, Sound Trends
  PDF download the PDF


Creating music is a contact sport. Literally, the more connection the musician has with the instrument and vice versa, the more expressive the performance. Connectivity in music involves connecting the senses, as much as possible, between the musician and the instrument. An acoustic or “real-world” musical device such as a Hammond B3, moog synthesizer, DJ mixing console, or Rhodes electric piano have one or more physical controls. For example, the instruments have one or more of the following: buttons, weighted or unweighted piano shaped keys, knobs, pitch bend and modulation wheels, and more to trigger notes, sounds, modulate the timbre, pitch, brightness, or other characteristic of the musical performance. All of the above controls provide some form of tactile physical boundaries, resistance, and in many cases, some vibration as feedback to the user. When a user plays a note on an electromechanical keyboard such as the Hohner clavinet, the user’s finger, holding down the note, feels the subtle vibrations of the string that is generating the sound. When a DJ plays a drum machine, the DJ can, without looking, move his or her hand to the grid of pads or buttons, and immediately know which buttons to press to produce a beat. If the pads are struck harder or softer, the buttons have the appropriate physical give, contributing to the physical, tactile connection between musician and musical instrument.

So, recently, we have seen musical instruments on flat-glass multi-touch devices, such as the iPad. We have also seen “alternative piano” controllers, such as the continuum, which is similar to a cushioned 2x4 with paint markings to delineate the note boundaries. Musicians, such as Jordan Rudess, have complained that these continuous and flag-glass multi-touch devices have a lack of musical intimacy. There is no feedback. There are no physical controls.

So, the Nightmare Theater development team at Project BBQ 2012, have invented the perfect solution to this lack of intimacy problem: The Wonder Glove. Similar to a fingerless bicycle glove, the Wonder Glove allows the musician to connect with tablet and continuous devices on a new level, even in a “mixed” physical instrument and tablet stage environment- in other words, the musician does not have to remove the glove to switch between tablet and conventional piano. Here, in general, is how the Wonder Glove works in harmony with existing iPad apps such as Garage Band:


Without the Wonder Glove, the Garage Band instrument begins generating sound as soon as the user touches a screen trigger, such as a piano key. The sound continues until the user releases his or her finger that triggered that key, off the glass.  Before releasing, the user can modulate the brightness by sliding the finger vertically and the pitch by sliding the finger horizontally.  There is absolutely no sense of the physical boundaries of controls.

Currently, the Garage Band instrument triggers a note no matter how hard or softly the user “taps” or touches the screen, as long as the touch or tap is on a note triggering control such as a piano key.  This is not like a real piano- a musician can touch the keys of a real piano without making a sound. The app must be rewritten to only generate a sound when a musician taps the glass above a certain velocity. Below and above that specific velocity, the Wonder Glove shall provide haptic feedback when the user is “mousing-over” or simply hovering over a note without attempting to play it. So, if the musician moves his hand with the Wonder Glove over the glass and the Garage Band instrument is a piano, the fingerlet on each finger will either click or vibrate as the finger crosses the boundary from piano key to piano key. This is similar to sliding a hand across a piano keyboard without playing it- the musician feels the “bumps” of the piano keybed. The same concept would be applied to a grid of drum machine buttons on the screen to indicate boundaries.

When a note is playing, the fingerlet on the finger which triggered the note shall vibrate, ever so slightly, to cue the user that a note has sounded, just as a real piano would transmit vibrations to the musician’s finger when a note is played. In addition, the Wonder Glove shall provide haptic feedback on the left side when a musician pushes a pitch bend lever to the left, simulating resistance, and likewise, vibration on the right side of the fingerlet when the user has displaced the pitch bend lever to the right. Other forms of feedback include when the user slides his finger vertically after triggering a piano note to increase the brightness by opening the filter- the vibrations in the fingerlet for that finger would increase. When the mod wheel is used to trigger vibrato, the vibration intensity would increase and decrease, following the envelope of the LFO. In short, all of the above would provide a much more connected musician and musical instrument.  The musician would not have to look at the iPad or other flat-screened or flat-featured musical instrument, and still have an intimate, connected feel with the instrument, causing much improved musical performance and authenticity of the on-screen flat instrument. In short, this would help solve Jordan Rudess’s request.


section 7

next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. Smart and Connected Microphones and Speakers
4. The Speakers are Talking, are the Microphones Listening? Connecting Smart Speakers & Microphones
5. Form Factors and Connectivity for Wearable Audio Devices
6. BACON™ for Your Ears: Designing a Musical Hearing Enhancer
7. Enhanced Input Method for Glass Tablet Instruments
8. Schedule & Sponsors