Larry Przywara, Tensilica, Inc.

Problem Statement

The audio user experience is often compromised by the surrounding environment, the user, and the context. To overcome the multitude of scenarios, we believe that fusing audio and many non-audio sensors can significantly improve the user experience of audio applications.

Example: Your child is presenting on stage, you’re in the audience in the back row with your camera. You zoom into your child and also want to capture the audio as he says his lines. For this there are several functions which could be added to the system. One example is that you pick up the signals from the microphone he’s wearing, by tapping into the environment resources (house sound system). Another example is to use an audio zoom function linked to the camera which combines microphone beamforming with shake compensation, position compensation, etc.

Sensors are getting widely deployed on smart devices. For example on smartphones today, accelerometer, gyroscope, GPS, proximity sensor, microphone arrays, speaker, front and back facing cameras have become common. Despite the presence of the sensors, mobile phone sensing is still in its infancy.

We want to raise industry awareness of end user benefits by combining multiple sensors domains. In this report we limit the scope towards benefits for audio related applications/use cases and their relationship to sensor data. On top we analyzed bandwidth requirements for sensors to enable these benefits.

The report is organized in following sections:

• Sensors
• Multilayer approach
• Audio applications
• Challenges
• Conclusions

I. Sensors

Below figure shows an overview of different sensors. Some of these are widely deployed into mobile phones, smartphones and wearable today, such as an accelerometer, gyroscope, GPS, proximity sensor, microphone arrays, speaker, front and back facing cameras.

The output of the sensor layer is a raw data signal with structured properties, such as information about the current data, sampling frequency, number of dimensions and the size of each dimension. Most sensors will yield one dimensional data, for example, audio signals, temperature. There are also sensors providing multi-dimensional data, for example an accelerometer.

A. Sensor Details

The physical limitations and range of measurement for the sensors bound what can be measured. The system architect also needs to know power and processing requirements.

     1. Mechanical

          a. Acceleration

          b. Rotation

          c. Atmospheric Pressure

          d. Sound pressure

          e. Ultrasonic wave pressure

          f. Gasflow

          g. Speaker

          h. Temperature

     2. Electromagnetic

          a. Ambient light

          b. Infrared light

          c. Magnetism

          d. GPS

          e. Camera

     3. Human

          a. Blood pressure

          b. Hand Grip

          c. Skin conductivity

          d. Fingerprint detection

     4. Connectivity

          a. Bluetooth

          b. WLAN

II. Multilayer Approach

The raw data from the sensors are typically interpreted in a multi-layer approach in order to make higher level, context-aware decisions. Reasons against always streaming the raw sensor data are:

• Privacy: sending raw data to the cloud
• Bandwidth
• Energy consumption: e.g., application processor processing high-bandwidth raw data
• CPU usage

In this report, we follow a three-layered architecture:

• Functions: the raw sensor data, which sensors are available in the system;
• Features: compressed summaries or cues interpreted from (multiple) raw sensor data, what can we learn from the sensors?
• Applications or User benefits: This is the decision level: how to combine features into a tangible benefit towards a user;

A. Examples

Example 1: You are picking up a call in your office, and you need to have a conversation with a group of people.  You are walking to a meeting room where more people are present, you put down your phone and your phone goes into a desktop conferencing mode.

• Functions: Ultrasonic microphone, accelerometer, gyro, magnetometer, proximity, grip detect
• Feature: Local device mode (Near Field vs. Hands Free vs. Far Field)
• Application: Automatic switching from earpiece to speakerphone mode during a call

Example 2: You are on a conference call, and one participant is causing manipulation noise (rubbing his phone on the table, touching buttons, etc.) adding all kinds of background junk to the call. These clicks can be very annoying for the far-end listener. We can resolve this using audio content only, but it gets much easier if we can also use input from the accelerometer, etc., to more robustly detect these manipulation sounds.

• Functions: Microphone array, accelerometer, gyro
• Feature: Manipulation noise detection
• Application: Improved noise suppression during voice calling

Example 3: Make a noise map of a city. For this an application would probably want to measure sound levels when the phone is out of the pocket or bag. To make a robust, in-pocket detection, multiple sensors could be combined:

• Functions: GPS, Microphone, Accelerometer, ambient light sensor
• Feature: Sound level, Pocket detection
• Application: noise map of a city

B. Architecture

Although it is not the goal of this working group to focus on the details of the system architecture, a high-level proposal can be investigated. The architecture split up of Functions, Features and Applications are done in a smart way which could enable the architecture as pictured below.

The sensors (Functions) are connected to a sensor hub or a sensor specific DSP/CPU core in the application. To minimize the data transfer the Features routines will need to run locally in this sensor hub or specific DSP/CPU core. Application functions will run on the main application processor and can call the feature routines to get key information (the application will call the specific Features routines in the hub. These Features routines will give back simple information based on the specific sensory called out by the Feature routine).

For a detailed list of the Functions, Features and Applications please refer to the tables in this document.

Advantages of proposed architecture:

• Reduced data transfer between Hub and Application processor
• Optimized for power management
• Standardization possible to support distributed systems (Ubiquitous Network)
• Independent for main operating system used

III. Audio Applications

Redundant and complimentary sensors can be fused and integrated in order to enhance system reliability and accuracy. Multi-sensor fusion can bring benefits in a wide range of applications, such as, robotics, military and biomedical.

In this section, we analyzed how fusing audio and non-audio sensors using the multi-layer approach can benefit the experience of audio applications. In a first step, we list typical challenges in audio use cases. In a next step, sensors functions are mapped towards resolving these hallenges.

A. Use Cases and Challenges

For this analysis, audio applications were classified according to use cases:

• Two-way communication (human-human)
• One-way communication (human-machine)
• Multimedia recording
• Multimedia playback
• Objective audio
• Idle case

1. Two-way communication (human - human)

Two-way communication happens in adverse acoustic conditions:

• Noisy environments
• Echo: sound of the speaker is captured by the microphone resulting in echo for the far-end talker;
• Room acoustics: reverberation and reflections of the audio signals
• Varying signal levels;
• Unknown user handling: strange device positions, covering microphones, position of speaker towards ear, pressure of speaker on the ear, etc.

These conditions are improved by active voice processing techniques, e.g., acoustic echo cancelling (AEC) and multi-microphone noise suppression (NS) algorithms.

2. One-way communication (human - machine)

One-way communication happens in similar adverse conditions as two-way communication. Solutions however can be distinct as automatic speech recognition engines do not necessarily react the same as human listeners.

3. Multimedia recording

4. Multimedia playback

Multimedia playback can happen in a variety of environments from quiet, to consistent noise (airplane) to varying noise across a variety of output devices – headphones/earbuds, internal and external speakers.

5. Objective audio (gaming)

6. Idle case

The idle case is when the device is in the low power always on state waiting for a wake up event.  Lowest power is ideal so only those sensors absolutely needed are left on.  Always listening can be accomplished by the combination of a sound/speech detector and a full voice trigger that’s initiated after the sound/speech threshold is tripped.  For proximity detection ultrasonic detection and the accelerometer can be utilized and is sufficient for always on.

B. Exploiting Sensor Fusion

1. From Functions to Features

In a first step, we map the functions towards features (or cues) which we need to enhance above discussed challenges. Multiple sensors can be used to provide a reliable or more accurate feature. This is sensor data fusion.

2. From Features to User Benefits

In a second step, we map the features towards an improvement in user experience. From this table we see a second level of sensor fusion as multiple features are combined: ‘feature fusion’.

IV. Challenges

The system is rather complex and the infra structure is currently not available to make optimum use of all the sensors and systems available in a room or even in a 1 box system.
There are also challenges in standardization

• To make use of this system in a most effective way it is important to standardize on:
  • the Features and their interfacing command structure
  • interfaces / bus  to the sensors but also to the Application processor
• Ultrasonic:
  • Standardize on identification
    (different frequencies for different devices in the room, adding meta data, ….;
  • who is pinging;
  • ultrasonic pollution
• Software architecture

V. Conclusions

• Fusing of the sensor data can improve the user experience by increasing the contextual awareness of the device.
• Audio (microphone and speakers) can be considered a sensor as well.
• Most valuable sensors to fuse with common audio processing seem to be:
• Accelerometer
• Ultrasonic-microphones
• Proximity detector
• Speaker as sensor
• A layered architecture is needed 
• Multiple feature routines are running simultaneously on the sensor hub
• Higher-level functions run on the application processor
• The sensor hub can have it’s own OS (lower power & performance)
• Standardization is Required: 
• In the software API
• In the sensor bus
• Minimization  of ultrasonic pollution
• Identification (authentication) of the ultrasonic source

section 6

next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. Ubiquitous Networked Audio
4. HD Audio Capture in Consumer Devices
5. Enabling More Profound Human Expression with Modern Musical Instruments
6. Using Sensor Data to Improve the User Experience of Audio Applications
7. When is Hardware Offloading Preferable, Now and in the Future?
8. Schedule & Sponsors