home  previous   next 
The Nineteenth Annual Interactive Audio Conference
BBQ Group Report: Audio opportunities in the Internet of Things
Participants: A.K.A. "Talk to the P.I.C.K.L.E. (Personal IoT Connected Knowledgable Life Enhancement)"

Yipeng Liu, Cadence

Scott McNeese, Cirrus Logic
Ilias Tsigkogiannis, Microsoft Dan Bogard, Conexant
Konstantine Merkher, Ceva Tomer Elbaz, Waves
Frederic Caldwell, Audience Sridhar Pilli, Knowles Electronics
Andrew Rumelt, Cirrus Logic  
Facilitator: Doug Peeler  
  PDF download the PDF

Brief statement of the problem(s) on which the group worked

  1. What are some of the compelling applications of audio in wearables and IoT?
  2. What are some of the acoustic barriers/challenges to meet these experiences?

IoT is quickly emerging as a fast growing set of segments creating new opportunities for novel user experiences that will benefit from the inclusion of audio capabilities. The number of compelling applications are legion with a wide array of sensors and interfaces for input and output.

IoT is a heavily segmented field, with many potential opportunities for audio applications.  The team felt that maximum value for the report would be achieved by focusing on a particular area.  This report focuses, primarily, on the smart home but we also considered two other applications, a smart watch and accessibility / personal enhancement.

A brief statement of the group’s solutions to those problems

Many of the elements required to create a compelling solution already exist; however, they have not been put together in a manner that is easy to use or attractive to the general population.  Furthermore, there are many hurdles (e.g. fragmentation in standards, privacy concerns, discomfort with conversing with machines) that are slowing down the ability of vendors to create compelling products.  As such, there are many approaches for vendors to address these problems and capitalize on the emerging market opportunities.

Expanded problem statement

There is a lot of energy surrounding the smart home, with many different players entering the market with their own products and platforms.  There has been a robust proliferation of different standards and platforms, which is creating difficulties for vendors to come up with a product that can be widely applicable.  Standards include the AllSeen Alliance, OIC, and other similar platforms.  Protocols are also lacking a standard bearer, resulting in multiple approaches lacking interoperability.

Other potential challenges include possible user objections to conversing with machines for various reasons, including general unease, cultural bias, or simply lack of comprehension due to language or accent barriers.  Furthermore, the cost of ownership is potentially prohibitive.

One of the bigger concerns that will need to be addressed by the industry at large will be privacy.  As more of the user’s information is collected, there is likely to be growing angst over what happens to their personal data.  The right type of value exchange can help users overcome their apprehensions; however, this will need to be managed with some delicacy.

Finally, there are several technical issues that will need to be considered:

  • Acoustics & Algorithms
    • Handoff/transitions between devices (mic arrays as an example)
    • Direction /source of response in user queries
    • De-reverberation, echo cancellation for far field voice and speech
    • Dynamic environment (moved furniture, party, etc.)
  • Architectural and Technical challenges
    • Distributed or centralized control?
      • Device to device connectivity collaboration (does not really exist today)
    • Latency/response time
    • Ability to readily connect to Cloud/internet
    • Power:  hard wired or self powered?
    • Installation/tuning
    • Determining the most appropriate and available sensor/device for a task (inputs and outputs)
    • Opportunities for analytics
    • Self diagnostics / self maintenance - (don’t to have to call pro out to check and repair. Want system to be robust, know and adapt to problems, notify when service is required)

Compelling audio-related applications within Smart Homes

There are many different ways in which a user will be able to interact with their smart homes.  It is our shared vision that within the next decade, the following use cases can be enabled, all with the intent of improving and enhancing the daily life of the user.

  • Speech interaction (human to machine)
    • entertainment
      • media playback
      • media recording
    • productivity
    • games
    • social networking
  • voice communication (human to human)
  • sound detection of movement / “presence”

Further benefits can be found in the form of:

  • Monetization of content
    • e.g. When a user sneezes, the action is detected, resulting in something serving up an ad for Claritin
  • Cost reduction for user interaction
    • Display replacement (cost, power)

Specific examples of audio use cases within the smart home include:

  • Speech control:
    • A/V controls (sound and video)
    • Scene creation (e.g. “sunrise” setting on lighting)
    • Environmental control (temperature, fan, HVAC)
    • Security
    • Distributed listening, with microphones in many “things”
    • Roaming ability - room to room (can continue phone conversation, etc.)
    • Device “talks” back
    • Local and cloud based feature definition
      • Preserves privacy
      • Improves reliability
    • Personalization
      • Personal micro-climate, lighting
      • Personalization of talk-back voice (text to speech)
      • Permissions / parental control (e.g. kids can’t change temp)
    • Multiple control inputs (manual fallback in case speech does not work properly)
  • Potential characteristics of a Smart Home
    • Smart controls and affordances
      • Environment / HVAC
      • Light
      • Fans
      • Temp / humidity
    • Smart Appliances (white goods)
      • Inventory
      • Maintenance
    • Smarter Security
      • Locks
      • Doors
      • Alarms
      • Monitoring
    • Data collection and analytics
      • Personal
      • Cloud
  • Potential Interfaces within a Smart Home
    • Inputs
      • Speech
      • Switches
      • Touch
      • 3D Gestures
      • App on mobile device
      • Video
      • Sensors in general
    • Outputs
      • Displays
      • Speakers/buzzers
      • Lights (IR, color, etc.)
  • Sensors that might be commonly found in a Smart Home
    • Microphone (more below)
    • Temperature
    • Humidity
    • Moisture
    • Camera
    • Ambient light/color
    • Motion/occupancy
    • CO2/CO/VOC
    • Smoke detector
    • Radon
    • Scent
    • Radio (Wi-Fi, Bluetooth, BLE, ZigBee, Z-wave, etc.)
    • RFID (tracking the location of dumb things)
    • Mass spectrometer
  • Microphone uses
    • Voice
      • Emotion/mood
      • urgent response (“help I’ve fallen and I can’t get up”)
    • Active listening
      • Security
      • Intrusion detection
      • Presence detection, location
      • Crying baby
      • Detect breaking glass
      • Detect fall (person collapses and can’t respond)
      • Structural shift, damage
      • Occupancy
    • Passive listening
      • Sound levels
      • Termite, squirrel, other infestation
      • Vibration detection (earthquake, etc)
      • Context
    • Indoor Positioning
      • Location using sound (e.g. sonar, echolocation, ultrasonics, time of flight, etc.)
      • FLP (fused location provider) contribution

Other reference material

  1. 2013 BBQ - Using Sensor Data to Improve the User Experience of Audio Applications
  2. 2012 BBQ - Form Factors and Connectivity for Wearable Audio Devices (AKA Clothing Optional)

section 7

next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. Metadata = Money
4. Reinventing the Audio Ecosystem with an Updated Smart-Connector
5. What does an Open DSP environment look like?
6. Interactive Music Creation and Adoption: The Quest Continues!!!
7. Audio opportunities in the Internet of Things
8. Schedule & Sponsors