home  previous   next 
The Nineteenth Annual Interactive Audio Conference
BBQ Group Report: Interactive Music Creation and Adoption - The Quest Continues!!!
Participants: A.K.A. "eleven inch nails! -- 11IN"

Phil Brown, Dolby Labs Licensing Corp.

Rick Cohen, SONiVOX LLC
Karen Collins, Ehtonal Mark Estdale, Omuk
Josh McMahon, Realtek Semiconductor Jay Smith, Livid Instruments
Mikko Suvanto, Akustica, Inc. Alex Westner, iZotope, Inc.
Guy Whitmore, PopCap Games/Electronic Arts Tom White, MMA
George Sanger, Magic Leap  
Facilitator: Aaron Higgins, Native Instruments  
  PDF download the PDF

Brief statement of the problem(s) on which the group worked

  • There are barriers in the creation of interactive music.
    • The tools are relatively inaccessible. Conceptually a new arena. 
    • Stereotypical genre (EDM?, ambient) is a barrier for some consumers and producers.
    • Rock/pop artists may not be aware of these tools. 
    • Currently algorithmic tools have sound sets that are not satisfying. Maybe typical tools are used for keyboard-based music rather than guitar/bass/drums stuff. 
    • The non-recording features of DAWs may be too “techy” right now.
  • There are barriers in the distribution of interactive music
    • No monetization/distribution model
    • No standardized player (e.g. iTunes)

A brief statement of the group’s solutions to those problems

After examining workflow for content creators:

  • Make recommendations for usability improvements in tools
  • Suggest new steps added to workflow

Feasibility of rock music for interactive content:

  • Created a interactive/algorithmic rock song

After examining the end-user problem:

  • Suggest end-user monetization strategies (distribution)
  • Conceptualize end-user music player, list interactive inputs


Current State of Affairs

What are the creation tools now?

  • Ableton Live
  • Wwise
  • FMOD
  • Max/MSP,  pd
  • SSEYO (Brian Eno)
  • Koan
  • NI Reaktor
  • ProTools

Who are the creation tools targeting?   - CONTENT CREATORS.

  • DJs
  • Game composers
  • Rock bands who want to try creating interactive music

Who is the end result targeting? -CONSUMERS

  • Game content
  • Home studio prosumer
  • Casual listeners - EVERYBODY!


  • Person at the gym, audio clips in the cloud.  Replacement for Pandora.
  • Driving in the car, replacement for XM satellite radio

Approaching how we would author a dynamic rock song:

Possible standard “moves” in rock music:

  • “Horizontal resequencing” (Transition from section to section)
    • Intro
    • A section
    • B section
    • Fill to A
    • Fill to B
    • Ending
  • “Vertical layering” (mixing, mutes and solos)


  • Record phrases at multiple tempos
  • Use MIDI?


Changing harmonic content:

  • Auto accompaniment/MIDI
  • Celemony
  • Record parts in both major and minor


  • Voice recording
  • Voice synthesis
  • Change lyrics for the song based on inputs, big data, today’s news, etc.


  • FX
  • Changing grains in unexpected ways
  • One barrier today is that you need to record all possible variations to use as building blocks.   Can we automatically process some of the grains to create variations?

In the future:

  • Granular approach (blocks, phrases)
  • Generative/melodic (at the note level) is not currently available in Wwise
  • “DNA” – The algorithm learns/adapts
  • Generate the sounds in real-time

Past examples of interactive pop/rock music

  • Beatnik was doing some of this 20 years ago.
  • Guy: Remember the interactive achievements of the past? Bowie, Gabriel.
  • Rick: Todd Rundgren, No World Order (1994). 
  • http://en.wikipedia.org/wiki/No_World_Order#Interactive_release
  • Bjork’s non-linear album app was created with FMOD and Cocos2D.  FMOD is similar to Wwise (audio middleware layer).  Cocos2D is similar to Unity (game engine).
  • Harmonix’s Rock Band Network is an example of how to turn a music track into a game. Can be viewed as a fail, because it was so complicated and time consuming (expensive), and the return was not there to make the effort worthwhile.

Wwise – the tool we used to author an interactive rock song

Our process

  1. Create segments of arbitrary length.  Units can be measures, real-time, etc.
  2. Each segment opens up to a DAW-like view of tracks. Includes “properties” e.g. filters and volume and pan (incl. surround) with real-time parameter control (RTPC == MIDI).
  3. Each track includes “clips”
  4. Each track can have multiple variations.  The choice of the variations is “random”.
  5. “Playlists” are collections of segments, can be linear or random. In general, plays one segment at a time, but you can also layer things called “stingers”.
  6. It is possible to transition from playlists to segments and back
  7. Segments contain Markers, which are the transition points.  (there should be an automatic algorithm to find these marker points). Can transition to the beginning of another segment, or to the same logical offset into another segment. Can also specify a special “transition segment”
  8. Foundation level is a set of states. The player is in one state at a time. E.g. “Pause the game”
  9. More advanced programming may require special support written into the game engine
  10. Some extension is available with scripting as well
  11. Audio FX can be applied to the tracks. Wwise provides some effects, also some available from iZotope.
  12. We can use the Profiler feature to show real-time playback and how it changes
  13. Switches – one part of the music to another
  14. States – different “mixing snapshots”

Wwise feature requests/ things that are not in Wwise:

  1. Recording within the application
  2. Be able to tell which variation is playing within a segment, if the segment window is open. (It works within a playlist)
  3. Import AIF
  4. Import multiple WAVs on a Mac
  5. Wiper does not stay in sync with audio while playing
  6. Playing from anywhere within a playlist
  7. Save a project as a template.
  8. Interface more like Ableton Live?

Proposed workflow for developing interactive music:

  1. Traditional DAW:
    1. composition, recording
    2. Export MIDI and WAV
  2. (we should consider asset management integration and translation into the workflow )
    1. How about translator tool to import a Live session into Wwise?
    2. Templates for Wwise?  E.g. verse/chorus; basic templates could include randomization, switches etc.
    3. A Turbo Tax style questionnaire (wizard) for allowing musicians to import their assets into Wwise
      1. Organizing clips into segments in Wwise
      2. Tagging clips with metadata for Wwise
      3. Metadata for matching input to states and switches (see “Cues and Inputs” slide)
  3. Non linear DAW
    1. arrangement of clips
    2. variability and control
    3. switches/states/etc. (parameters)
    4. demo/playing
  4. App authoring engine
    1. Connect user inputs to states/switches
    2. UI integration
  5. Deploy the app

Defining the End User Experience

Parameters that might be defined by the content creator

  • Mood
  • Tempo
  • Instrumentation
  • Density (Instrumentation depth)
  • Dynamic Range
  • Intensity
  • Lyrical content
  • Genre
  • Sex (male / female)

Business Opportunities

  • Production music – huge opportunities that could revolutionize this industry!
  • Revolutionizing the role of music in people’s lives
    • Providing an interactive music consumption experience that is immersive enough to be the only for of entertainment needed for extended periods of time (>10 minutes)
  • Education
    • teach people to understand music structure
    • make it easier for people to use music for learning
  • Big data
    • Artists basing their creative process on what moves people audience
    • Album updates
    • Targeted advertising
  • Tiered subscription model
    • free: advertisements / jingles included in music
    • paid: no ads
  • In-app purchases of new tracks / content


  • What do you get when you buy the music? Both linear version and the parametric version?
  • Granularity of the produced music? How to handle the large amount of metadata?
  • Monetization - What do you pay for?
    • music length
    • amount
    • quality….?
  • What role does synthesis have?
    • Creating new lyrics or changing the lyrics

What kinds of metadata for user input?

  • Exercise
    • Timing (intervals)
    • Heart Rate
    • GPS / Maps
  • Daily routine inputs
    • Biometrics
      • SCR (Skin Conductance Response)
      • Blood Pressure
      • Heart Rate
      • Time of Day / Calendar (date)
  • Soundscape - The noises around you
    • Duck the sound when a phone call comes in or if somebody is speaking

Past Bar-B-Q Reports

Project Bar-B-Q 2007: Overcoming Roadblocks in the Quest for Interactive Audio

We’ve come a long way since 2007.  Lots of progress on some of the problems.  Simon Ashby from AudioKinetic (makers of Wwise) was in this group, and has implemented some of the solutions proposed in the meeting.

Other references from the 2007 paper:
IASIG Interactive XMF Workgroup (IXWG)

Bar-B-Q 2006 Group Report: Providing a High Level of Mixing Aesthetics in Interactive Audio and Games

Bar-B-Q 2005 Group Report: New Approaches for Developing Interactive Audio Production Systems

Bar-B-Q 2003 Group Report:  What is Is Interactive Audio? And What Should It Be?

Bar-B-Q 2001 Group Report: Towards Interactive XMF

Bar-B-Q 2000 Group Report: General Interactive Audio

section 6

next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. Metadata = Money
4. Reinventing the Audio Ecosystem with an Updated Smart-Connector
5. What does an Open DSP environment look like?
6. Interactive Music Creation and Adoption: The Quest Continues!!!
7. Audio opportunities in the Internet of Things
8. Schedule & Sponsors