Peter Drescher; Danger

Problem Statement Summary

The group discussed designs for an interactive audio production and auditioning system that could be marketed to a wide audience in order to motivate software developers to create the necessary tools. We identified the problems faced by interactive audio developers and the hurdles that must frequently be overcome. We discussed the Phive Stages of Game Audio Development: Pre-production, Prototyping, Production, Polishing and Phinal Testing.

A Brief Statement of the Group’s Solutions to Those Problems

The need and desire for a standard, open, game audio production and auditioning platform has been discussed at BBQ numerous times in the past (by the Big Picture, Foghorn, and iXMF groups, et al), and yet there has been little progress, despite the obvious advantages such a system would have on workflow efficiency and audio production costs. Why? The answer seems clear: the interactive game audio market is not big enough to justify the cost and effort of tool development. Nonetheless, the need exists. How do we bring such a product to market?

The group came to the following realization: that by looking beyond the niche of game audio production, new (and possibly mass) markets for interactive audio development tools can be identified.

The solution proposed by the group is to suggest standards and methods for developing an "interactive audio authoring and auditioning system" that can be marketed to non-game audio producers, including:

• DJs (for performing live mixes)
• Remixers (for algorithmic and interactive control of mixes)
• Educational markets:
  • Children (such as LeapFrog)
  • Trade schools (such as Full Sail, Berklee, et al)
  • Universities (for "art of interactive audio" studies)
• Interactive audio skins for mobile devices
• User controlled audio for interactive DVDs
• And of course, various game markets:
  • The growing home brew and indie game market
  • Independent and student game audio producers
  • Interactive audio tools embedded in game audio engines
• More possible markets to come (some of which have yet to be invented)

But first, an open, standard system must be designed. In order to accurately define the technical issues such a platform must address, the group performed the following exercise:

1. Define the "Phive Stages of Game Audio Development"
2. Discuss the common obstacles inherent in each stage
3. Identify possible products that could solve these problems
4. Identify new markets that might deal with similar issues, and therefore have a need for these products

The Action Item List

1. Guy Whitmore to talk to David Battino re: contact information on Ableton Live (to gauge interest in interactive audio)
2. Roger Powell to talk with Apple Computers personnel about an interactive audio authoring and auditioning system that produces platform agnostic files (possibly iXMF)
3. A white paper / final report to be written as a group effort, lead by Linda Law
4. A press release outlining possible opportunities in interactive audio for non-game markets (Matt Tullis)
5. Peter Drescher to talk to DJs about their audio needs, and how an interactive audio system might benefit their efforts in both live performance and remixer situations
6. more to be determined …

Expanded Problem Statement

Year after year at BBQ, many of the smartest people in the interactive audio industry have discussed the need for an interactive / adaptive audio production and auditioning system. All agree that creating such a product would not be an especially daunting task technically and that "if you build it, they will come" (meaning, there already exists a customer base for the product). Nonetheless, no one has yet attempted to create the system. Why?

One answer is that proprietary systems are already in use by numerous game developers (XACT, for example). These systems are created in-house and usually only address the needs of the developers' products, even though they all basically do the same kinds of things (use of scripts, providing hooks into the game engine AI, etc.) It seems like a great deal of time, money and effort has been wasted in re-inventing the wheel, while simultaneously failing to advance the overall art form of game soundtrack scoring.

An open, non-proprietary system for producing interactive audio content can significantly improve the audio developers' workflow (and thereby reduce audio development costs) by simplifying the process composers and sound designers must utilize to audition music and sound effects in the context of gameplay. This can save countless hours of effort, free up programmer resources, and enhance overall interactive audio quality. Such a system might ultimately retire the need for in-house studios using proprietary tools, if the system can be made available and desirable to a wider market. As in the Hollywood film model, a well developed set of tools can open up competition by studios and content providers, thereby lowering costs and increasing choice.

We believe that the key to enhancing the workflow for producers of game audio is a tool base that crosses over into the pro-audio market. Once interactive audio production tools function with pro-audio development packages such as Logic or ProTools, or used with stand-alone applications such as Ableton Live or Acid, the doors may open for the creation of interactive soundtracks on a mass market scale, as has happened in the film industry. The engineering hurdles must first be addressed, but at this point, the barriers do not seem too high either technically or financially. The proof is in the work of the last 10 years of BBQ reports, and in this group's discussions.

Expanded Solution Description

While many of the problems outlined above are out of the control of audio professionals because they are rooted in current approaches and attitudes in game development, better tools and workflow could make the audio portion of game development smoother, easier, enabling better results and advancing the art of game audio. This problem was recognized by the 2003 BBQ group that worked on the iXMF rollout.

Developers, publishers and content creators are currently wasting time and money by working with multiple and incompatible audio systems. Even though there have been many advances in implementation, creative control is often still not in the hands of the audio artist, remaining the responsibility of the game programmer. This compromises quality and wastes resources, problems that are exacerbated when developing multi-platform titles. Completion of the iXMF spec in itself is not enough to solve this problem so we are providing a roadmap for its accelerated adoption by the industry.

Once the iXMF spec is completed, the difficultly is in getting the tools created to support it. Working with open standards will provide a wide enough user base to encourage the development of the tools required by interactive audio creators, and probably spark new applications as yet unforeseen.

The group came to three important realizations:

1) Many of the features required in a game audio production platform are also features useful to other markets. For example, DJ’s performing live need to be able to manipulate sections of music in real time, trigger different sections at will, lay additional sounds or beats on top of existing grooves, or adding effects manipulated in real-time. Software such as Ableton Live sets up a framework which facilitates this type of on-the-fly ordering of musical sequences, and the manipulation of audio by parameter changes made during live performance. Similarly, game audio requires the setting up of a framework of audio chunks that are then manipulated and ordered in real-time based on game variables. It would not be much of a stretch to add functionality to DJ software that exports "audio data + interactive rules" into a format that could subsequently run on a game audio engine.

2) The groundwork for the audio format that would be imported and exported is being laid by the iXMF group, which had its genesis in a previous BBQ workgroup. iXMF is a public standard structured audio file format that supports cross-platform interchange of advanced interactive soundtracks. It uses a queue-oriented model, is programming-neutral, and can be used without license agreements or royalty payments. The open file format of iXMF allows the creation of an authoring system (or competing systems in the marketplace) that can transfer data from the audio tool to the game engine, which in turn reads and plays the iXMF, adapting the music and sound effects based on game variables.

3) XML is an open standard which is currently being used successfully in other disciplines for inter-application data sharing, and could be utilized for our purposes by allowing a game audio authoring application to set up relationships and rules to control audio, and thereby facilitate real-time auditioning in gameplay context by passing data between the authoring tool and the game.

By combining these ideas, tools can be reusable from game to game without having to waste time and money on redundant efforts. By using open standards, and leveraging the capabilities of existing software tools that also require real-time manipulation of audio, new markets can be created and served.

In order to accurately define the technical issues and possible market opportunities for an interactive audio authoring and auditioning system, the group defined the phases of game audio development, discussed common problems for each phase, and suggested possible solutions. The list of solutions is intended to help define market opportunities for interactive audio systems outside of the traditional game audio development business.

1. Pre-production Phase

• Meeting with designers (akin to director / composer relationship)
• Collaboration between creative designers of product and audio personnel

  Results => audio design document, containing:
      a. Content - creative elements based on product design
      b. Implementation - technical analysis of product and platform
      c. Details of interactivity for product

  Common problems encountered:
  - Incomplete design doc
  - Incomplete or nonexistent technical specs
  - Technical, design, management, composer ignorance
  - Educating the client on interactive possibilities
  - Negotiating for physical resources (RAM, CPU, etc)
  - Negotiating for time and budget
  - Getting appropriate integration tools and specifications

2. Prototyping Phase

• Styles - establish direction of audio design with concept pieces
• Interactive proof of concept - demonstrate interactive parts of audio design
• Sign off and approval of audio design

  Results => final specification

  Common problems encountered:
  - Lack of prototyping tools for interactive audio that is able to handle multiple formats
  - Lack of game graphics for inspiration
  - "Final" spec

3. Production Phase

• Organizing assets
• Content creation
• Implementation
• Mixing

  Results => audio production assets

  Common problems encountered:
  - Lack of ability to work on audio IN PRODUCT
            Variety of game engines
            Audio content creation in tandem with game play
            Auditioning
            Editing
            Mixing
            Iteration looping
            Replay game scenarios
  - Inability to use your preferred audio production tool and components
  - Proprietary issues (some are hurdles, some are market opportunities)
  - Cross-platform differences and scaling issues
  - Tool licensing models don't exist
  - Sample bank licensing models don't exist
  - No universal plug-in standards
  - Need OEM versions of software synths / plug-ins
  - Compatibility and interoperability issues (there IS no interoperability)
  - Programmers reluctant to use black box code
  - Methodology and tools need to evolve in tandem
  - No standardized way to organize assets
            Technology
            Methodology
            Version tracking and merging
  - No mixing UI for hardware or software
  - Not getting art assets

4. Polish Phase

• Creative qualifications - reviewing and evaluating audio content in context by content creator
• Revisions - changes based on review of content in context
• Audio QA testing - testing that audio design functions as designed in the spec
• Bug fixing - fixing bugs
• Play testing
• Final mix
            Finalizing levels
            Setting effects levels
            Setting overall output levels
            Audio sweetening
            Testing in optimum listening environment with various speakers, in required formats

  Results => Sign off by producers and creative personnel

  Common problems encountered:
  - need for location cheats to test sounds in place
  - Real-time revisions while product is running
            (some do revisions in real time, but systems are unique to each product, and UI is bad)
  - need for controlled environment to listen to game audio
  - Compressed schedule
  - No standard / guidelines for volume levels
  - No specialized interactive audio mixing engineers, rooms, techniques or tools exist
  - Few audio qualified audio testers or test environments
  - No audio QA methodology
  - No automated way to test game audio

5. Phinal Testing Phase

Release candidate
Console certification
... going once, going twice...

Result => shippable product

Items from the brainstorming lists that the group thought were worth reporting

The "evolution" analogy:
There is scientific evidence in the fossil record suggesting that early amphibians did not "grow legs" so they could walk on land. Rather, when certain ocean fish moved into a newly formed environment of shallow fresh-water lagoons, stronger fins allowed them to maneuver along the muddy bottom (where all the good food was). The ability to use those thick fins to get out of the water onto land when attacked has obvious survival advantages, but is not the function the fins performed when they originally evolved.

SO, if "interactive audio tools" are the "legs", and "dry land" is the game audio industry, then perhaps we can help along development of the tools we want by first creating them for the "fresh-water lagoon" of new markets that could benefit from interactive audio capabilities. Once the tools exist, game audio developers could use them to run.

------------------------------

The "camera phone" analogy:
At first, combining a phone and a camera seemed absurd, but it solved a problem people actually had, which was "How can I conveniently carry a camera around everywhere, and send those spur-of-the-moment pictures to my friends?" While camera phones are now a very popular solution, people didn't really have the problem until the camera phone was available. Thus the product invented the need for the solution.

Perhaps something similar can be done for interactive audio authoring tools. For example, a DJ wanting to create long, evolving dance tracks might take advantage of an interactive or algorithmic audio system during a live performance. Such a tool might be the solution to a problem the DJ didn't know he had.

------------------------------

Other Reference Material

IASIG's Interactive XMF specification (not yet published)

iXMF Rollout Outline
http://www.projectbarbq.com/reports/bbq03/bbq03r4.htm

What is Interactive Audio?
http://www.projectbarbq.com/reports/bbq03/bbq03r5.htm

Towards Interactive XMF
http://www.projectbarbq.com/reports/bbq01/bbq01r5.htm

Foghorn report from BBQ 2000
http://www.projectbarbq.com/reports/bbq00/bbq00r7.htm

Big Picture report from BBQ 1999
http://www.projectbarbq.com/reports/bbq99/bbq99r4.htm

section 5

next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. Using a Multiplicity of Audio Devices in the Home PC
5. New Approaches for Developing Interactive Audio Production Systems
6. Design Features of a Mass Market Living Room PC
7. Ubiquitous Content Distribution to and within the Home
8. Improving Computer Audio and Music Production Systems User Interfaces
9. Disrupting the Current Paradigm of How Audio is Viewed and Used
10. Schedule & Sponsors