I. What is Interactive Audio?

Interactive Audio is sound produced by an Interactive Audio System.

So, what is an Interactive Audio System?

An audio system that is designed to have its pre-determined sonic behavior influenced in response to real-time events, and is comprised of an Interactive Audio Engine and Interactive Audio Data.

Interactive Data
Data that may include parameters and/or instructions interpreted by the interactive audio engine to control audio behavior. All types of audio, including music, dialog, and sound effects, can be classified as interactive audio data.

Interactive Engine
A set of software and/or hardware algorithms that process interactive data based on the input stimuli to render a unique audio output.

Direct vs. Indirect Input Stimuli
The input stimuli to the system can be classified into two categories. In the direct case, the user is consciously controlling the audio; in the indirect case, the user is controlling some other parameter that in turn affects the audio. For example, the MadWaves MadPlayer is a direct experience, whereas a video game player indirectly interacts with the audio.

Reactive vs. Interactive
Not all systems that respond to input stimuli can be defined as interactive audio systems. An interactive system allows changes in input behavior to modify the audio behavior, whereas a reactive system simply plays back static audio events without any adaptation to the user stimulus.

Current Examples of Interactive Audio Systems

Beatnik (www.beatnik.com)
An interactive audio system allowing pre-designed responses for sound and music cues to play in real-time response to end-user input. It gives interactive audio creators an opportunity to compile and compress different media types and design them to react in a predictable, cohesive way in response to unpredictable user and/or system input. Initially designed for Web sonification, Beatnik technology is now used primarily for sonifying wireless devices such as mobile phones. According to the company, "Beatnik offers core audio engine technology, application programming interfaces (APIs) and tools that allow the development of audio-enabled applications, services and content delivery for the mobile world."

Mixman (www.mixman.com)
This consumer software automatically aligns triggered sounds to the beat, so no matter how rhythmically impaired you are, you can create music that sounds good. It can be played from a computer keyboard or a proprietary USB controller that resembles DJ turntables.

Ableton Live (www.ableton.com)
This Mac/Win program is a performance-oriented looping and arranging tool, in contrast to "offline" editing programs such as Sony Acid and BitHeadz Phrazer. Live automatically scales sounds to fit a song's tempo and key in real time.

Runtime Mixing System
A system that can take N channels of audio input and mix them into N channels of output based on the states of each input channel at time T.

Remixing System
As interactive data, songs include musical information plus data parameters used to modify the song on the fly. The user actions act as input stimuli to modify the pre-defined audio stream.

Roland V-Synth (www.v-synth.com)
This sampling keyboard is an interactive composition and performance tool that features real-time pitch-shifting and time-scaling based on the player's actions.

Korg Karma (www.korg.com, www.karmalab.com)
This synthesizer keyboard generates idiomatically correct musical parts based on player's actions.

HyperInstruments (www.media.mit.edu/hyperins)
Musical systems developed at the MIT Media Lab that augment the expression capabilities of musicians. By using sensors to measure extra human parameters (input stimuli), the sound (the data for the system) is changed based on these parameters.

Game Audio Systems
The capability to establish a set of rules and procedures for how audio assets are played back and controlled during playback in response to unpredictably timed player input and system-generated events, all in accordance with the audio creator's creative design.

MadPlayer (www.MadWaves.com)
The MadPlayer is a handheld composition and performance device. A MadPlayer song includes musical information and data parameters used to (re-)compose a song and modify it on the fly. MadPlayer engine algorithms take user actions and pseudo-random number series as input stimuli to modify the pre-defined audio stream; thus, the music produced has been modified in response to input stimuli.

Theme Parks (environmental response through, e.g., trip wires)
A system that reacts to a visitor's location within an environment with pre-determined audio responses to triggers.  

II. And What Should It Be?

An Interactive Audio System should include the following in order to be successful:

• Provisions for data IP management.
• Provisions for IP protection of playback system.
• A simple and intuitive UI for the audio artist, especially in a direct system.
• Flexibility in the types, numbers, and sources of input stimuli.
• Flexibility in the types of output (e.g., headphones, speakers, Bluetooth, vibrations).
• Flexibility in the types of data (e.g., user customizable).
• Ability to handle variable data (e.g., generative music, physical modeling).
• Interoperability between systems (e.g., support of plug-ins, inter-application communication).
• Re-configurable DSP processing for customization of a system's capabilities.
• Two-way communication with a host system.
• Standardized connectivity and peer-to-peer capabilities (e.g., multi-players, remote system integration).
• Multiple simultaneous client support with synchronization (e.g., networked system).
• Situationally appropriate responsiveness (reaction to gameplay).
• Context and environmental aware (e.g., muting in theaters).
• Ability to comply with various rendering system standards.
• Scalability.

Examples of Potential Interactive Audio Platforms

• Video Games
• Electronic Music Instruments
• World Wide Web
• Interactive Music Player
• Computer Software
• Computer Peripherals
• Casino and Arcade Games
• Audio Books
• Telephones
• Customer Service Robots
• Mobile Devices (mobile jamming)
• Commercial Environments (Theme Parks, Restaurants)
• Theaters
• Homes
• Art Installations
• Consumer A/V Systems
• Toys
• Greeting Cards
• Automobiles
• Appliances
• Virtual Environments
• Sporting Goods
• Clothing and Jewelry
• Education

  III. Why Interactive Audio?

• It enhances the user experience.
• It empowers the user via participation and choice, facilitating the market trend toward active consumption.
• It provokes and inspires user involvement.
• It creates a unique personality for products.
• It enables users to perform new types of activities.
• It creates a participatory education experience.
• It's potentially cheaper to implement.
• It allows simplification of the system and cost reduction.
• It allows audiences to experience interactive audio outside of its original context.

For many years, audio designers in the video gaming industry have been asking for tools and components for the creation of Interactive Music. Systems have come and gone and still satisfaction does not yet seem to have been achieved. For others, the question of Why still remains. Who or where are these hordes of consumers demanding interactive music scores to enhance their games? Who on the consumer side is even asking for interactive music for any purpose? Of course one could say, "Who ever asked for the hula-hoop or the pet rock?" Yet once these products were available hordes of consumers wanted them.

Unlike with the pet rock, however, there is already a huge consumer base for interactive music. Perhaps one of the biggest indications of this came on August 1, 1981, when a new television broadcast network called MTV first aired. Suddenly it was no longer satisfactory to simply record your music for audio-only playback. To satisfy the consumer, music video versions had to be made as well.

Also in the '80s, with technology rapidly advancing, a new type of musician was born. This musician replaced the traditional instruments with tools like the turntable and the audio mixer. Rather than study traditional methods of composition, this musician used technology to alter the physical aspects of sound itself. This musician could recycle existing music and remix it with other sounds to make something new.

So much has been evolving in interactive music that for the last two years a UK-based organization called IMA (Interactive Music Awards) has been highlighting and honoring the best in interactive music on the Web and elsewhere. The IMA seeks out and solicits content for submission in a number of categories to be judged in two ways. The first is by an expert panel of judges and the second is by vote from music fans.

In addition to leisure consumption, interactive music has taken a large part in musical education as well. Loads of Web sites offer musical experiences aimed at educating people about music creation. All levels of toys offer interactive systems that allow children and adults to take part in music creation without having learned to play a traditional instrument.

With all that is going on with interactive music today, there are still many artists who can see new paths to take. For some, the necessary tools or methods do not yet exist, but it is obvious that continuation in the design of interactive systems and platforms will continue to help satisfy the consumer demand.  

IV. Conclusion and Recommendations

There is a documented trend toward interactivity in many areas. Consumers, especially young ones, are demanding dynamic, creative activities instead of passive, static ones. Remixing and downloading are extraordinarily popular. Interactive audio has the potential to feed that demand. However, if it is poorly implemented, it could stifle innovation and disappoint this important new audience. Standardizing a framework for interactive audio systems now could ensure that the next generation of audio-based applications is rewarding for everyone.

There are many examples of a growing trend toward interactive audio in society today - on computers, video game consoles, home entertainment devices, and telephones; in cars, shopping and checkout systems, theme parks, museum exhibits, education centers, and more. The most advanced interactive and adaptive audio today is found in video games. As games become more sophisticated over the coming years, so too will other instances of audio that is played adaptively in response to user stimulus. The audio will be played back more and more intelligently, both in how it plays, and in what plays. For example, if a person were to say something to a robot, the robot's responses in today's state-of-the-art would be primitive compared to a human being's. But with Artificial Intelligence (AI) growing in leaps and bounds, and interactive audio systems improving across countless platforms and applications, demand for adaptive audio to be innovative and natural will increase and evolve.

That is why it is important to start now to develop a formal understanding of the main principles of what constitutes interactive audio so we can collectively design and share customized instances of those building blocks and simplify the design and production process for all industries seeking to enhance their products with interactive audio. Interactive audio is here to stay and will continually increase its presence in all types of human experiences. It makes the greatest sense to get the foundation properly established at the outset to avoid scattered efforts and incompatibilities, and to enable the greatest level of individualization of a solid, well-defined departure point.

V. Further Information

For more information on interactive audio and the progress on related standards, the Interactive Audio Special Interest Group (IA-SIG; www.iasig.org) is an excellent resource. In particular, its IXMF (Interactive XMF) working group is in the process of completing extensions to the XMF format specifically to address many of the topics that have been covered in this report.

VI. Action Items

Complete the report within 4 weeks

• Research existing technologies (e.g., IXMF) - Rob R.
• Complete What is I.A.? section - Danny
• Complete What Should It Be? Section - Mark
• Complete Why? Section - Keith
• Complete Conclusion Section - Clint

• David will write a sidebar in his book
• Clint will evangelize to press and industry
  • Update: December 17, 2003: Mix magazine has confirmed it will publish an overview of the MadPlayer and interactive audio in Paul Lehrman's February 2004 Insider Audio column as a result of input from this group.

section 5

next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. IXMF Rollout Outline
5. What Is Interactive Audio? And What Should It Be?
6. Creating a Win-Win-Win-Win-Win-Win Scenario for DRM
7. nASCAP - Stock Market in Media Rights
8. Quality Audio Applications in the Interactive Entertainment Industry
9. Schedule & Sponsors