home  previous   next 
The Twenty-first Annual Interactive Audio Conference
PROJECT BAR-B-Q 2016
BBQ Group Report: R.I/O.T: The Next Great Interactive Group Listening Experience!
   
Participants: A.K.A. "SONIC R.I/O.T"
Brad Lambert, Cirrus David Dully, Dolby
Matt Windt, Skullcandy Peter Otto , UCSD
Matt Johnston, Microsoft Bobby Lombardi , WD/Sandisk
Brett Shipes, Magic Leap Phillip Nicol, Dolby
   
Facilitator: David Battino, Batmosphere™  
  PDF download the PDF
 

Problem Statement

Currently the opportunities for groups to participate in an interactive listening experience are limited and shrinking due to the increasing popularity of headphones and other isolating audio delivery methods.

Solution:

We recognized that feedback between listeners and the artist/event is what defines the “next level” listening experience. SONIC RIOT worked collaboratively to design an abstract system model that supports an ongoing feedback and processing loop. The system creates a generative experience based on real-time collective group input. As the group collectively participates, they receive real-time positive feedback that reinforces their participation. The system supports and expanded definition of “group” by extending participation options beyond the immediate physical location.

This system combines the best parts of the individual listening experience (such as audio quality and personalization) with the excitement of the group listening experience (such as social participation, surprise, and eventfulness) and opens numerous commercial and creative opportunities. The system lets you feel like you’re a part of something bigger than yourself — and actually be part of it.

RIOT: Realtime I/O Togetherness

System overview:

Key components

  • Artist/Event (seed) – “oscillator” kicks off the loop with an event
  • Producer Processor (X2) – Determines what about the event gets passed to the listeners or seed
  • Listeners (Local and/or remote) – processes output from Producer and generates input to sensors. Also manages cross-talk between listeners.
  • Data filter/gate/aggregator – aggregator of listener output, including curation.

Optional:

  • Personalization filters (pre or post listener) – modifier that allows for individual parameter control including security/privacy.

Implementation/sample scenario 1: “THE WAVE”

Overview: A stadium system stimulates and enhances collective crowd participation during “the wave”

Setting: A stadium sporting event with 20,000-plus people

Seed: The wave starts either organically or as prompted by cues from the stadium system (producer).

Platform overview:

  • Hardware similar to current stadium sound designer gear that is open to sensor input and controlled by stadium sound designer.
  • Stadium sound system has been upgraded and optimized for a more distributed ambient sound experience, and microphones that can capture participant ambient noise.

Sensors:

  • activity
  • energy
  • location
  • amplitude
  • heart rate
  • temperature
  • C02
  • respiratory rate
  • Butt-In-Seat Sensor™

Rendering:

  • Near-field speaker systems in seating
  • Mic arrays that can capture participant noise level
  • Far-field speakers
  • Transducers in the seats/floorboards
  • 3.5mm headphone jack

Inter-Listener Operations

  • Peer pressure
  • Section-to-section grieving/trash talk
  • Exciter/transducer activation as warning prompt/trigger when wave approaches

Curator/Producer/Processor role

  • Technical sound design
  • System design
  • Failsafe operator
  • Synthesizer
  • Sampler
  • Audio engine
  • Authoring/UI

Filter/Gate – receives/parses/filters all user data

  • Pattern recognition (avoid false positives)
    • Location
    • Direction
    • Amplitude
  • Noise
    • Rogue movements
    • Non-wave-related butt activity
  • Event recognition for override
    • Game events
    • PSAs
    • Moments of silence / national anthem
  • Sectional identification based on location
  • Parameter scrubbing for handoff to seed

Sequence:

  1. Once the wave starts with a small subset of the audience standing up and waving arms, the sensors identify this behavior as wave=true.
  2. Wave activity location is determined by Butt-In-Seat Sensor™, and aggregated as input to SEED.
  3. Pattern recognition identifies largest concentration of participation activity and directs primary audio signal to that location.
  4. Audio is processed and delivered back to the participants.
  5. Wave activity continues and builds due to positive audio feedback from stadium audio system.
  6. If WAVE=TRUE repeat!

Modifiers:
As more participants join in, the quality of the sound changes:

  • Amplitude
  • Spatial position
  • Sonic density
  • Pitch shifted
  • Harmonic density
  • Band shift
  • Rise/drop
  • Percussive/rhythmic build
  • Mic signal fed back in
    • Participants
    • Field activity (crack of the bat during a big moment)
    • Focus spotlight on wave peak

Parameters:

  • Number of participants
  • Speed of wave motion
  • Length of consistent wave activity
  • Wave density location / point of maximum displacement
  • Ambient audio volume
  • Game scenario:
    • “Roulette wheel” with light

Other possible scenarios:

  • Bird watching
  • Walking Tour (headphones)
    • When one person goes rogue, the system can help get everyone back together
    • Sensors can determine energy level of group
    • Orientation/IMU scenarios
    • Museum tours
    • Planetarium “space walk” tour
      • Sound localized to attract focus to the right space
    • Voice notes to attract others to an interesting feature in the environment
      • GeoLoc audio notes once you get to the interesting feature
    • Ratings crowd source interesting features
    • User-generated tours, geographical music compositions
    • Botanical garden tours that can be annotated by experts and also “scored”
  • 21st Century personal radio for sporting events
  • NASCAR
  • Custom crowd walla based on social graph
  • Collective party music DJ
  • Co-located / distributed VR silent disco dancing
    • Avatar/3D scanned models show co-present dancers
    • Virtual speaker system that surrounds dancers
    • Private directed conversations (cone of silence)
    • Synchronized dancing levels up the music by adding layers/textures
    • Group energy level rises and adds audio/visual effects
    • Different dancing “arenas” or stages
    • Anomalous behavior triggers events (bad dancer / sitting down)
  • Listening to the northern lights/space/meteor showers (headphones)
    • Everyone lays down in a field
    • One person can narrate
    • Group breathing exercises (meditation)
    • Sensors analyze the star field and sonify the stellar activity
    • Dynamically mixed soundscapes based on group focus / other inputs
    • Beacon navigation to help orient your focus
    • Pillows
  • Listening to underwater volcanic/hard to attain sonic activity
  • Motorcycle convoy

Potential stakeholders:

  • Artists/creators
  • Consumers
  • Manufacturers / hardware
  • Organizers
  • Broad audience

Possible challenges/issues

  • Will leagues allow it? – mitigation: beamformed audio delivery
  • Privacy
  • Cost
  • Users gaming the system (problem or feature?)

Extensibility options:

  • Open source platform for plugin development

Competitive/similar systems

  • Plastikman concert performance system
  • Stadium audio feedback systems
  • Taylor Swift bracelets
  • World of Color hats

section 6


next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. The Future of Voice Interfaces
4. Audio Sensor Opportunities: Market Requirements and Technology Challenges for the next Decade
5. Always Be Closing (This isnít marketing after all)
6. R.I/O.T: The Next Great Interactive Group Listening Experience!
7. The Need for a New Wireless Audio Network Standard
8. Creating Immersive Music with Audio Objects
9. Schedule & Sponsors