home  previous   next 
The Sixth Annual Interactive Music Conference
brainstorming graphic

Group Report:
A Compelling Music-Making System for the Living-room Console

Participants: A.K.A. "The Compellorheads" Kristoffer Larson; Dolby Laboratories
Roger Linn; Roger Linn Design Jack Buser; Dolby Laboratories
David Battino; Batmosphere Ron Kuper; Cakewalk
Larry the O; LucasArts Chris Grigg; Beatnik
David Roach; SigmaTel George A. Sanger; The Fat Man

Facilitator: Linda Law; Fat Labs


Problem Statement

Design a compelling music-making system for the living-room console.

We asked each member to describe this killer product. We felt it should have:

  • The potential for players with minimal talent to create a song
  • An upward path to virtuosity
  • Multiplayer and multiconsole support
  • Full exploitation of the console’s standard controllers (for example, the PlayStation 2 has 13 buttons with 127 levels of pressure sensitivity)
  • An octave-shift key for buttons
  • The ability to support other, more musical input controllers
  • Force-feedback to the controller
  • A traditional synthesizer interface to create beat loops, sample libraries, and synth textures
  • Sound that is configurable by presets or by manual tweaking
  • Sample and track layering
  • Modes that let you jam with a buddy or jam with the controller
  • A setup screen where you choose a musical style
  • Multiple musical genres
  • Artificial intelligence to transform a single note into musically meaningful output.
  • Scalable screens, as complex as the hardware and software will allow
  • Visualizations that react to your input
  • A scalable design, so that it can be adjusted to whatever level of music-making is required
  • A personalizable interface, not a spreadsheet grid
  • The ability to create and exchange mods
  • Low graphics overhead, because the graphics controller will need to be co-opted for audio processing

Product Concept

E-Jam: The music-making system for the living-room console


  • Minutes to learn, a lifetime to master
  • Single person or multiperson
  • Uses existing hardware, but is expandable for alternate interfaces (e.g., keyboard, guitar, drums, motion sensors)
  • Easy accessibility through musical style templates
  • Can record sequences, saving them to flash card
  • Can record music and video out to your VCR
  • AI jamming system, à la the Korg Karma
  • Realtime input quantization
  • Can be extended with add-ons such as editors, plug-ins, alternate content, visualizations, etc.,
  • Uses standardized architecture (e.g., XMF, MIDI).
  • Scalable levels of user interface depth
  • Linking between living-room consoles
  • Uses console’s Internet connectivity
  • Inspiring visual feedback during performance
  • Audio can affect visuals
  • Realtime audio input, with optional peripheral
  • Can sing into it, with built-in pitch correction and harmonization
  • Can play guitar into it, with built-in effects
  • Sampling capability

Business Opportunities

1. Sell primary title and differentiated titles, including educational and mini games.

2. Sell plug-ins

  • Audio effects
  • Synthesizers
  • Visualizations
  • Editors

3. Sell content

  • Loops
  • Samples
  • Styles

4. Sell hardware

  • Instrument/performance interfaces, e.g., keyboard, guitar, drum, mixer, turntable
  • Other hardware interfaces, e.g., audio, MIDI, AV

System Architecture

Open plug-in interface

  • Transportable (file-embedded) read-only plug-ins
  • Copy protection
  • Automatable/realtime controllable

AI Jamming

  • Note and tempo analysis
  • AI templates (e.g., blues)
    • The templates can capture the individual style of famous musicians, allowing players to create fantasy bands.
  • “Celebrity Musician AI Agents.”

Other Features

  • Object Oriented Modularity: highly flexible, to allow plug-ins to be applied in a variety of positions in the music-processing chain.
  • Internet transfer protocol, to define how content and mods are shared among users across the Internet.

What You Get Out of the Box

  • Sample libraries
  • DLS2-compatible software synth
  • Basic effects: delay, EQ, reverb, distortion, chorus, flanger, phaser, dynamics
  • Basic subtractive synthesis
  • Basic oscillators, e.g., sine wave, sawtooth wave
  • Basic MIDI effects: quantize, pitch transpose
  • Mini games:
    • Jam mode: rhythmic, melodic, etc.
    • Learning mode
    • Competitive mode: two-player, à la Dance Dance Revolution
    • UI setup

Action Items

  • Create a standard plug-in architecture for consoles, and enforcing mechanism (we nominate Chris Grigg)
  • Convince console manufacturers to assist by easing access (Jack Buser)
  • Launch IA-SIG working group on a language for representing performer personality (Ron Kuper)
  • Write a proposal for a basic product (entire group)
  • Shop proposal (entire group)

Appendix — Items from Original Brainstorming Session

Our original theme was “Building the dream — the killer app (plus some tools)”

Items for the roundtable brainstorming session were:

  • Identify tech hurdles and tough R&D problems
  • Describe a killer app
  • Identify how to make music-making fun again
  • Apply interactive music concepts to live performance
  • Examine higher dimensional compositional structures
  • Describe compelling music applications
  • Develop an appliance action plan

We started off by trying to define a musical instrument. We identified two separate components: a box of tones and the controller. We saw a need to define the connections between the pieces. In tools, and in games, we all need access to the same box of sounds. We tried to define what the bridges are: MIDI, sync, file formats, etc.

We then discussed whether it makes sense to separate control from the sounds. Some felt that for something like a guitar or sax, the two are all wrapped up in the instrument, and that if you separate the two you lose a big part of the experience.

Another issue: How do you get the “feel” of your instrument? Why is it that a step sequencer has feel but DirectMusic doesn't? Where is the “feel” in XMF?

Roger Linn’s drum machines were cited as electronic instruments with “feel.”

What would the brainwave-transducer musical instrument be like, we wondered. Would it suck, or would it be the most compelling musical experience ever?

Dave suggested designing a whole band of instruments, considering the role of each instrument, and how it could be optimized: melody, harmony, rhythm, ambiance, and lyrics, in (possibly intersecting) spheres.

After about an hour of this kind of discussion, we begin to converge on some initial objectives. Our first mission statement was:

Identify elements that result in compelling music-making experiences, including:

  • Collaboration: human/human, human/machine, machine/machine
  • Expression
  • Creation

Larry said these topics of real-world interfaces apply much more broadly than just to musical instruments. He mentioned a head-mounted position-sensor in a game.

Inspired by Roger’s idea of synchronizing two musical devices wirelessly, we discussed the concept of a “grab sync out of the air” interface. We identified that MIDI over a local wireless network (e.g., Bluetooth) would be a very important enabling technology.

We also discussed human gestures, examining what people are trying to control, and what we are trying to communicate when we control things with our gestures.

We listed elements that result in compelling music-making experiences:

1. What do we want to see? What kinds of things do we get when we are in a compelling music-making experience?

a. Fun
b. Collaboration
c. Expression
d. Creation
e. Feel
f. Surprise
g. Interactivity
h. Repeatability
i. Ecstasy
j. Virtuosity
k. Intuitiveness
l. Tonal versatility
m. Low cost
n. Wireless
o. Portability
p. Ergonomics
q. Structure
r. Immediacy
s. Nuance
t. Connectivity
u. Scalability
v. Aesthetics (of the physical instrument) — good looks
w. Reliability
x. Durability
y. A.I., adaptive, reactive
z. Synergy
aa. Exportability
bb. Physicality – takes work
cc. Immersiveness
dd. Challenge
ee. Good sound
ff. Attainable
gg. Deep
hh. Safe
ii. Culturally significant
jj. Reductionistic
kk. Spiritual
ll. Recordable
mm. Easy to use
nn. Value

2. As applicable to #1, analysis and recommendations

  • Existing support
  • Hurdles
  • Ways to add what’s missing

We took the long list from #1, and split it along lines of three categories: construction (i.e., physical design), experiential (i.e., emotional), and technical (i.e., logical design).

Experiential elements are listed here:

Interactive, Creative, Expressive, Collaboration, Fun, Emotional Feel, Surprise, Nuance, Spirituality, Ecstasy, Path to Virtuosity, Intuitiveness, Portability, Musical Structure, Immediacy, A.I. (adaptive/reactive), Synergistic, Exportability, Physicality, Immersive, Challenging, Attainable, Sounds Good, Depth, Cultural Significance, Ease of Use, Rewarding, Value

(Note: wireless goes as existing support for a whole bunch of technical items.)

(Note: recordable goes as existing support for tasks such as capturing fader moves)

Items we took off the list: cost, safety.

We attempted to run through the list of construction and technical issues to identify existing hurdles.

Construction (physical design)


  • Knobs too close together
  • Requires additional movement
  • Shift buttons
  • Physics vs. desired playing experience
  • Multiple actions required for one output. Solution: dedicated control
  • Control: learnabilty, resolution, repeatability
  • Touch-sensitive gestural controls


  • Custom construction
  • Cost
  • Usability
  • Durability
  • Industrial design: Don’t do software without a graphic designer; don’t do hardware without an industrial designer.


  • Wires. Solution: a standardized musical wireless protocol
  • Power. Solutions: low-power chips, no wall warts
  • Amplification. Solution: personal monitoring systems
  • Size and weight. Solution: space-age plastics, wheels, small appliances
  • Standardized musical wireless protocol
  • Standardized wall-wart
  • Standardized wireless low-voltage power transmission


  • Cost. Solution: spend money
  • Product cycles
  • Poor power supply design
  • Generic replacement parts
  • Fault tolerance


  • Frequently used items break. Solution: robust design.
  • Liquids vs. electronics. Solution: waterproof cases, cup-holders
  • Planned obsolescence
  • Magnetic sensitivity
  • Extended life testing
  • Harsh-environment testing
  • Abuse testing

Technical (logical design)


  • All things proprietary: file formats, I/O
  • CPU differences
  • System performance differences
  • Wired I/O differences
  • Sync
  • API proliferation
  • Existing audio infrastructure
  • Inertia of the installed base
  • Truly cross-platform APIs
  • Performance profiles


  • USB vs. FireWire
  • 802.11 vs. Bluetooth
  • Incompatible protocols
  • Not enough connectivity
  • Latency and jitter
  • Quality of service
  • Bandwidth
  • MIDI
  • Crappy plugs
  • Multiclient conflicts
  • Inconsistent interapplication routing/communication
  • Wireless music protocol


  • Lack of widespread acceptance of multiprocessing
  • Lack of modular audio software
  • Lack of mods (modifications)
  • Never shipping updates
  • Standards
  • Dynamic bandwidth scaling or I/O port expansion


  • Crappy synchronization
  • Bad parts, poor-tolerance parts
  • Noise
  • Volatile memory
  • Lack of automation
  • Languages/messages to describe automation
  • Standards
  • Time-stamping mechanisms

Tonal Versatility

  • Memory
  • Processor
  • Algorithms
  • Synthesis technology
  • Number of controls
  • Accepting MIDI (as the only way when building a synthesizer)


  • Testability
  • Lack of disclosure
  • Cost
  • Product cycles
  • Decent OS to handle audio
  • Realtime OS

section 8

next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. Networked Audio Devices Interoperability Standard
5. Towards Interactive XMF  
6. Improve PC Audio 
7. Game Audio Network Guild
8. A Compelling Music-Making System for the Living-Room Console
9. Schedule & Sponsors