home  previous   next 
The Thirteenth Annual Interactive Audio Conference
PROJECT BAR-B-Q 2008
brainstorming graphic

Group Report: Wrap It Up -- Creating a New Middleware Marketplace for Digital Entertainment Platforms

   
Participants: A.K.A "Run DSP"

Simon Ashby, Audiokinetic

Peter Drescher, Microsoft
Lucas Gonze, XSPF Jim Rippie, Invisible Industries Consulting
Brian Schmidt, Brian Schmidt Studios Guy Whitmore Microsoft
  Facilitator: David Battino, Batmosphere
 

Problem Statement

Content creators use a wide variety of high quality DSP effects to produce digital audio in the studio. These same algorithms would be an effective and efficient way to enhance audio produced at runtime by gaming, web, and mobile applications. However, there is currently no established method for DSP vendors to sell their products to software developers. There is a demand but no effective marketplace.

Brief Statement of the Group’s Solutions to those Problems

Recent progress has been made in the console gaming industry, where some developers have integrated third-party runtime DSP algorithms into shipping products. We recommend that game developers take further steps to create a viable marketplace for these kinds of deals, with an eye towards possible future expansion into web audio applications and mobile devices. However, a solution for gaming may not necessarily transfer to other industries, as each technology faces unique business opportunities and technical barriers.

Mission Statement: To jump-start and foster an ecosystem of DSP middleware companies serving game audio, offering a wide variety of high quality yet efficient algorithms across multiple platforms.

Expanded Problem Statement

While the market for DSP plug-ins for digital audio workstations is thriving (i.e. Waves plug-ins for ProTools), the market for runtime DSP for applications (i.e. convolution reverb for games) is currently limited by technical and business hurdles:

Technical

  • Plug-In Formats: There are multiple formats for runtime DSP plug-ins, creating implementation issues, integration complications, and developer frustration.
  • Gaming Platforms:
    • The engineering effort required to port algorithms across various platforms doesn't scale very well.
    • Optimization is necessary, since CPU performance and RAM footprints are always tightly constricted.
    • The relative strengths and weaknesses of various platforms require real differences in the software.
    • Ongoing software maintenance is unavoidable, due to changes in platform functionality via SDK updates.
  • Web Applications:
    • Architectural issues due to online security and system latency can cause problems for runtime DSP.
    • Sufficient CPU may not be available to all computers connected to the Internet.
    • Various browser limitations and online upgrades make implementation and maintenance troublesome.
  • Mobile Devices:
    • The technical issues described for gaming platforms are even more pertinent for mobile devices.
    • CPU and RAM are currently too constrained to allow useful levels of DSP to be performed.
    • The enhanced audio quality must always be balanced against increased battery consumption.
    • A wide range of operating systems and feature sets makes implementation and distribution problematic.

Business

  • Market Size: The customer base for middleware DSP in games is relatively small. While each game developer or publisher is a potential purchaser of runtime DSP plug-ins, the sales potential pales in comparison with the digital audio workstation market.
  • Business Development:
    • Start up costs are steep, as is the required continuing maintenance and upgrade engineering.
    • Dev kits for multiple platforms are a significant investment.
    • Necessary legal and promotional services can be prohibitively expensive.
    • Entry into market may be too difficult for small or one-man shops; ironically, this is where many interesting, efficient, and/or useful DSP plug-ins are produced

Expanded Solution Description
 
Incorporating high quality DSP algorithms (such as reverb, EQ, compression, etc) into runtime environments not only produces better sounding content, it can provide an easy and inexpensive method for varying audio generated by any application, whether it be a AAA game running on a console, a Web application running on a laptop, or a casual game running on a cell phone.

Audio content creators are accustomed to using these tools for sound design on workstations. Providing access to these types of algorithms in the runtime environment will strengthen the desired audio experience, while enhancing realtime audio adaptation. As the processing power of each successive generation of systems and devices increases, using runtime DSP to manipulate audio on the fly, in response to gameplay or user interaction, will become a progressively indispensable technique.

It would be best if a single, open, standardized, plug-in format were used to produce runtime DSP effects, preferably one already in wide use in the professional audio industry. While there is no compelling technical reason to use one format over another, there are many incentives for choosing a spec and sticking with it, including faster and easier development, porting, and maintenance, of products.

There seems to be no technical reason why such a marketplace could not exist. In order to foster a growing ecosystem of business relationships between DSP effects producers and game developers, an initial seeding is needed to produce a self-sustaining marketplace. This process has already begun, as evidenced by license agreements between Waves and Bungie, and Wave Arts and the Microsoft Game Studio.

To encourage DSP makers and game publishers to collaborate on creating this new marketplace, the group intends to evangelize the concept, advantages, and opportunities, at various industry events (such as the Game Developer's Conference), and in various publications, as listed in the action items. If successful, we hope that three to five runtime DSP effects businesses will emerge at first, allowing developers to choose between different solutions without requiring inordinate amounts of custom integration work. Progress will be evaluated in one year, possibly at Project BBQ '09.

Another way to increase business opportunities is to widen the market to include web applications and mobile devices. As more processing is done by distributed networks using modular open source libraries, development of online DSP effects for social music collaboration, MMOs, and "cloud computing audio" become desirable. However, significant business, security, and browser resource issues, must be resolved before such a marketplace would be viable.

As mobile devices become more powerful, with greater CPU and higher audio fidelity, new opportunities for runtime DSP also become possible. Some mobile applications might include voice processing for spoofing or entertainment, increased audibility for phone calls, environmental  noise cancellation, and EQ to compensate for speaker limitations. Don't forget the "who knew" factor, whereby unpredictable convergences can produce unexpected functionality that becomes highly profitable.

Other Reference Material

Links to online resources (GDC presentation slides, online article URLs, etc) will be listed here when available.

Action Items

 

Who’s Responsible

Due Date

Description

1

Peter Drescher

11/25/2008

Complete report for publication, adding Brian's list (next)

2

Brian Schmidt

11/15/2008

Create list of technical and business hurdles that new DSP vendors should be aware of, and approaches to solving those obstacles

3

Guy Whitmore

March 2009

2009 GDC talk on this topic

4

Guy Whitmore

March 2009

Challenge Sony and other publisher/developers to strike deals such as the MGS-WaveArts DSP licensing deal.

5

Brian Schmidt

11/25/2008

Explore speaking at NAMM regarding this, targeting DSP companies

6

Jim Rippie

 

Try to place game DSP article in Mix magazine, built around interviews with platform manufacturers, game publishers, DSP makers, and middleware developers.

section 4


next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. Wrap It Up -- Creating a New Middleware Marketplace for Digital Entertainment Platforms
5. The EduMusiTainersT
6. So you want to work in game audio?
7. Smart Ambient Sound Sensor
8. New Creators and New Creative Tools – Understanding the New Ways to Make Music
9. Working as a Creative Professional in a Corporate Environment
10. "PRAGMA" Rebooted (pet rocks and game music alliance) A Project Bar-B-Q 2008 rogue group updating a Project Bar-B-Q 2004 rogue group
11. Schedule & Sponsors