- There are barriers in the creation of interactive music.
- The tools are relatively inaccessible. Conceptually a new arena.
- Stereotypical genre (EDM?, ambient) is a barrier for some consumers and producers.
- Rock/pop artists may not be aware of these tools.
- Currently algorithmic tools have sound sets that are not satisfying. Maybe typical tools are used for keyboard-based music rather than guitar/bass/drums stuff.
- The non-recording features of DAWs may be too “techy” right now.
- There are barriers in the distribution of interactive music
- No monetization/distribution model
- No standardized player (e.g. iTunes)
After examining workflow for content creators:
- Make recommendations for usability improvements in tools
- Suggest new steps added to workflow
Feasibility of rock music for interactive content:
- Created a interactive/algorithmic rock song
After examining the end-user problem:
- Suggest end-user monetization strategies (distribution)
- Conceptualize end-user music player, list interactive inputs
What are the creation tools now?
- Ableton Live
- Max/MSP, pd
- SSEYO (Brian Eno)
- NI Reaktor
Who are the creation tools targeting? - CONTENT CREATORS.
- Game composers
- Rock bands who want to try creating interactive music
Who is the end result targeting? -CONSUMERS
- Game content
- Home studio prosumer
- Casual listeners - EVERYBODY!
- Person at the gym, audio clips in the cloud. Replacement for Pandora.
- Driving in the car, replacement for XM satellite radio
Possible standard “moves” in rock music:
- “Horizontal resequencing” (Transition from section to section)
- A section
- B section
- Fill to A
- Fill to B
- “Vertical layering” (mixing, mutes and solos)
- Record phrases at multiple tempos
- Use MIDI?
Changing harmonic content:
- Auto accompaniment/MIDI
- Record parts in both major and minor
- Voice recording
- Voice synthesis
- Change lyrics for the song based on inputs, big data, today’s news, etc.
- Changing grains in unexpected ways
- One barrier today is that you need to record all possible variations to use as building blocks. Can we automatically process some of the grains to create variations?
In the future:
- Granular approach (blocks, phrases)
- Generative/melodic (at the note level) is not currently available in Wwise
- “DNA” – The algorithm learns/adapts
- Generate the sounds in real-time
- Beatnik was doing some of this 20 years ago.
- Guy: Remember the interactive achievements of the past? Bowie, Gabriel.
- Rick: Todd Rundgren, No World Order (1994).
- Bjork’s non-linear album app was created with FMOD and Cocos2D. FMOD is similar to Wwise (audio middleware layer). Cocos2D is similar to Unity (game engine).
- Harmonix’s Rock Band Network is an example of how to turn a music track into a game. Can be viewed as a fail, because it was so complicated and time consuming (expensive), and the return was not there to make the effort worthwhile.
- Create segments of arbitrary length. Units can be measures, real-time, etc.
- Each segment opens up to a DAW-like view of tracks. Includes “properties” e.g. filters and volume and pan (incl. surround) with real-time parameter control (RTPC == MIDI).
- Each track includes “clips”
- Each track can have multiple variations. The choice of the variations is “random”.
- “Playlists” are collections of segments, can be linear or random. In general, plays one segment at a time, but you can also layer things called “stingers”.
- It is possible to transition from playlists to segments and back
- Segments contain Markers, which are the transition points. (there should be an automatic algorithm to find these marker points). Can transition to the beginning of another segment, or to the same logical offset into another segment. Can also specify a special “transition segment”
- Foundation level is a set of states. The player is in one state at a time. E.g. “Pause the game”
- More advanced programming may require special support written into the game engine
- Some extension is available with scripting as well
- Audio FX can be applied to the tracks. Wwise provides some effects, also some available from iZotope.
- We can use the Profiler feature to show real-time playback and how it changes
- Switches – one part of the music to another
- States – different “mixing snapshots”
Wwise feature requests/ things that are not in Wwise:
- Recording within the application
- Be able to tell which variation is playing within a segment, if the segment window is open. (It works within a playlist)
- Import AIF
- Import multiple WAVs on a Mac
- Wiper does not stay in sync with audio while playing
- Playing from anywhere within a playlist
- Save a project as a template.
- Interface more like Ableton Live?
- Traditional DAW:
- composition, recording
- Export MIDI and WAV
- (we should consider asset management integration and translation into the workflow )
- How about translator tool to import a Live session into Wwise?
- Templates for Wwise? E.g. verse/chorus; basic templates could include randomization, switches etc.
- A Turbo Tax style questionnaire (wizard) for allowing musicians to import their assets into Wwise
- Organizing clips into segments in Wwise
- Tagging clips with metadata for Wwise
- Metadata for matching input to states and switches (see “Cues and Inputs” slide)
- Non linear DAW
- arrangement of clips
- variability and control
- switches/states/etc. (parameters)
- App authoring engine
- Connect user inputs to states/switches
- UI integration
- Deploy the app
Parameters that might be defined by the content creator
- Density (Instrumentation depth)
- Dynamic Range
- Lyrical content
- Sex (male / female)
- Production music – huge opportunities that could revolutionize this industry!
- Revolutionizing the role of music in people’s lives
- Providing an interactive music consumption experience that is immersive enough to be the only for of entertainment needed for extended periods of time (>10 minutes)
- teach people to understand music structure
- make it easier for people to use music for learning
- Big data
- Artists basing their creative process on what moves people audience
- Album updates
- Targeted advertising
- Tiered subscription model
- free: advertisements / jingles included in music
- paid: no ads
- In-app purchases of new tracks / content
- What do you get when you buy the music? Both linear version and the parametric version?
- Granularity of the produced music? How to handle the large amount of metadata?
- Monetization - What do you pay for?
- music length
- What role does synthesis have?
- Creating new lyrics or changing the lyrics
What kinds of metadata for user input?
- Timing (intervals)
- Heart Rate
- GPS / Maps
- Daily routine inputs
- SCR (Skin Conductance Response)
- Blood Pressure
- Heart Rate
- Time of Day / Calendar (date)
- Soundscape - The noises around you
- Duck the sound when a phone call comes in or if somebody is speaking
Past Bar-B-Q Reports
Project Bar-B-Q 2007: Overcoming Roadblocks in the Quest for Interactive Audio
We’ve come a long way since 2007. Lots of progress on some of the problems. Simon Ashby from AudioKinetic (makers of Wwise) was in this group, and has implemented some of the solutions proposed in the meeting.
Other references from the 2007 paper:
IASIG Interactive XMF Workgroup (IXWG)
Bar-B-Q 2006 Group Report: Providing a High Level of Mixing Aesthetics in Interactive Audio and Games
Bar-B-Q 2005 Group Report: New Approaches for Developing Interactive Audio Production Systems
Bar-B-Q 2003 Group Report: What is Is Interactive Audio? And What Should It Be?
Bar-B-Q 2001 Group Report: Towards Interactive XMF
Bar-B-Q 2000 Group Report: General Interactive Audio