Problem Statement

Define a framework to evaluate how closely a render system produces an experience equivalent to a live (aka real) one. Additionally, define a methodology to reduce the dependence on human test subjects.

Outline:

1. Scope
   1. As called out in the problem statement, the team’s focus was using the framework of an Audio Turing Test to evaluate a render / reproduction system. At first, we focused on the reproduction of a single talker. The target is mostly around single outputs (human voice, triangle, musical instrument, etc.). Machine Learning (ML) should be used to optimize the output system under test.
2. Definitions

3. Assumptions / Caveats
   1. Idealized Source Capture (located at source location)
   2. Target listener shall also have a listener capture / microphone system (one mic per ear for example)
   3. The device under test is not intended to be an Artificial Intelligence (AI) using Natural Language Understanding (NLU) or Speech Synthesis but a reproduction system of a live source
   4. Assumption is that test being run can leverage users of the appropriate training
   5. The capture system is assumed to be ideal and a practical solution to the capture system is outside the scope of this paper
   6. Render system is expected to be targeting a specific use case or product
4. Source Capture System
   1. The source capture system shall capture all needed acoustic characteristics of the source. Examples are: frequency response, directivity (which implies multi microphones), level, etc. Additional information such as distance from source will be captured in metadata which will be given to the render system
   2. To build an ideal system many aspects some of which are:
      1. Number of microphones and their placement and application (noise floor, directivity, response)
      2. Source / environment interaction (material properties of walls, distance to walls/corners, RT60, etc.)
      3. The freedom of motion given to the source and therefore metadata needed to faithfully reproduce
   3. Utterance Processing Example for Audio Turing Test v1.0

The below framework is an example of how one could do data collection for an Audio Turing Test. It is not intended as an endorsement of a given brand.

Define the Input of the Recording System

• Four-channel Broadcast Wave Format (BWF)
  • Ch1
    • Name:  Primary Source
    • Mic:  Neumann TLM103, S/N TBD
  • Ch2
    • Primary Reference
    • Mic:  Earthworks M30, S/N 9800
  • Ch3
    • Mouth Reference
    • Mic:  Earthworks M30 , S/N 9805
  • Ch4
    • Ear Reference
    • Mic: Earthworks M30, S/N 9806
• Each channel:
  • Suggestion: 96 kHz sample rate
    • Requirement: at least 44.1 kHz
  • 24-bit word length
• For each file:
  • iXML data contains details need for cataloging or post processing

Processing steps

1. Extract each channel from 4-channel file
2. For Primary Source
   1. Correct frequency response: gives Equalized Primary Source (EPS)
      1. Use a measured curve captured in situ and in an anechoic chamber
   2. Apply high-pass filter to EPS representative to output render
      1. Examples:
         1. Artificial mouths
            1. HMS II.3
            2. B&K 4128
            3. B&K 5128
            4. B&K 4227 or equivalent
   3. Check Utterance Level / Crest Factor
   4. Normalize levels of HPF-EPS
      1. Currently to –26 dBFS using P.56 ASL
   5. Convert to 48 kHz sampling rate, 24-bit word length (for archive)
      1. Suggestion: MATLAB resample at high-quality (e.g. filter length > 128, not default) should be unity gain.  If sample rate conversion is not unity gain, do Normalization after SRC
         1. Recommendation: investigate resample software and artifacts that the system may introduce to the playback system
   6. Check SNR to validate and reduce reproduction artifacts
   7. Measure speech production level
      1. Power analysis on the 96-kHz/24-bit format
      2. Calibrate/equalize Mouth Reference
         1. use calibration value from iXML for the Mouth Reference mic to calibrate to absolute dB SPL
         2. use the calibration curve for the corresponding S/N for each M30 mic to build an equalization filter for the mic’s calibration curve
      3. For each talker, apply calibration and equalization filter to the Mouth Reference channel, to each file, to get Equalized Mouth Reference (EMR)
      4. Measure the speech power of EMR using P.56 ASL. This is an estimate of the speech production level, in dB SPL ASL, for that talker on that sentence
      5. Save in a database
   8. Measure mouth-to-ear transfer function
      1. Calibrate/Equalize Ear Reference
         1. use calibration value from iXML for the Ear Reference mic to calibrate to absolute dB SPL
         2. use the calibration curve for the corresponding S/N for M30 mic to build an equalization filter for the mic’s calibration curve
      2. For each talker, apply calibration and equalization filter to the Ear Reference channel, to each file, to get Equalized Ear Reference (EER)
      3. Compute Transfer function using EER and EMR for that talker on that utterance
      4. Save transfer function in a database
   9. Trim ‘silence’ and avoid background artifacts that would impact reproduction
      1. Fade in and fade out file to avoid discontinuity
   10. File naming convention
       1. File name that is human readable
   11. Other data for database, from iXML for each file – Optional
       1. Region, Language
       2. Text of utterance
          1. Compare this text to that produced by ASR in off-line check.
       3. Talker Gender
       4. Talker Age Code
       5. Mic S/Ns & Cal values
          1. Cross-check cal values from Cal recordings, also taken for each Day of recording
       6. Date of recording

5. Transformation System - the transfer function / system takes the capture system into the required format for the render system
   1. If the Transformation System isn’t ideal, it will be part of the Device Under Test in the Audio Turing Test
   2. As the input system is ideally defined, we cannot prescribe an exact transformation system but we can articulate the system requirements
   3. The owner of the render system is responsible to take the necessary inputs to meet their output render system
      1. If N mics and the system only needs M (M < N) inputs which then maps to P output
   4. The Transformation System consumes the capture system metadata and translates and generates new metadata according to the needs of the render system
   5. The Transformation System should be part of the Machine Learning Loop to tune the system to reduce the error from the render system

Transformation System Training
This diagram illustrates how ML can be employed to build a scoring system that determines the subjective quality of a transformation system. During the training session, the transformation system is fed a list of sets of control parameters, one at a time. For each parameter set, the expert listener assigns a score such as “0% (terrible)” or “100% (perfect). The listener’s score, plus the parameters that were fed into the transformation system, plus extracted audio features from the captured sound, provide the training for an ML based scoring system.

Transformation System Refinement
This diagram illustrates how once an ML-based scoring system has been created, it can be used to continue to refine the transformation system in an unsupervised learning process. Control parameters to the transformation system are varied at random using Monte Carlo methods, and the scoring system is used to accept or reject that random parameter changes.

6. Render System – is the output system that is under test for believability. Takes the transformation system as an input.
   1. Different types of render system will have their own requirements
      1. e.g. smart frames or AR will need to have a specific set of setup requirements from that of a speaker system located at the original source location, may have to simulate the room.
7. Listener Capture System – sufficient number of microphones to capture what is near the user ears (shall be acoustically transparent to not impact the test)
   1. Render system or Tuning algorithm may require a specific setup
   2. head reference point (acoustic center of head), earDrum Reference Point (ERP), etc.
8. Physical Setup (Person Prep / Training)
   1. Known location could be taken into account or remove location knowledge from source (blindfold)
9. Scoring / Metrics
   1. We are specifically trying to evaluate an Audio Turing Test. Any influence of vision or other senses shall be removed as necessary
   2. [Reference][2] Olive, Sean. (2004). A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model  http://www.aes.org/e-lib/browse.cfm?elib=12847
      1. Weighted components for loudspeaker attributes
      2. Positive correlation
      3. Abs average deviation 200-400Hz
      4. Narrow band deviation (100-12kHz octave at a time)
      5. Low frequency extension (–6dB)
      6. LFQ - Abs deviation from -6dB to 300Hz
10. Human Scoring
    1. Uses all scoring parameters to help automate the process to reduce “cost” of actual test listeners
    2. A-B-X Testing [3]  (https://en.wikipedia.org/wiki/ABX_test)
    3. Believability
    4. In the setup of the Audio Turing Test you could re-run against a Mean Opinion Score (MOS) type of score to help tune the system
    5. Additional feedback on the accuracy of the user can be attained through skin resistance and / or EEG sensor.
    6. Should presented in a way to minimize “test” pressure
11. Classifier Training
    
    Classifier Training - Listener Evaluation
    
    
    1. Could use the results of the test to train a neural network to be able to detect what’s realistic (train a classifier to learn from people what’s better or worse.)
    2. Could then use the trained ML to figure out how to vary parameters of the transfer function to improve it
12. Metrics / Electrical Characteristics
    1. Machine learned automatic improvements
13. Usage and Applications
    1. Device types that were considered:
       1. Single Speaker
       2. Multi-Channel Speaker
       3. HMD (Headphone, Headset, Earbuds, AR/VR Frames, etc.)
       4. Consumer Electronics (Phones, Tablets, etc.)
    2. Specific Use Cases:
       1. Reproduction System for Voice Assistance:
          1. Prove the render system is a sufficient reproduction of a known human voice.
       2. Reproduction of Musical Instruments
          1. Single Instrument - Triangle, Violin, Trombone, etc.
          2. Multiple Instruments - String Quartet, Guitar and Voice
          3. Rock Band & Large Orchestra – Can be done but scale and room size may be impractical in most applications
       3. Naturally Occurring - size and location will need to be considered
          1. Foley type events: Rustling of leaves
          2. Streams and Brooks
          3. Animals, Birds
          4. Wind
14. Clumped Topics / Additional Considerations
    1. Deep Fake - How do you know if that sound is real?
       1. Once Synthesis was taken out of scope we didn’t look into this too closely
    2. Establishing a set of concrete speaker performance measurements
       1. The render system not passing the Audio Turing Test and the parameters that are tuned to improve performance (either from human or ML system) will be used to understand the concrete speaker performance measurement needed for reproduction
    3. Clarity on what makes audio feel Human / Natural
       1. The feedback loop from the ML system would be expected to inform us as to what is required to be perceived as real
    4. A human perspective on audio
       1. We are utilizing other senses for validation for the Audio Turing Result but didn’t look into what does it mean to be human or organic edge computing
15. Conclusions (necessary?)
    1. Under a bounded setup this is able to be accomplished today. Doing an A-B-X test of a simple acoustic source versus a reproduction system is possible today with this guidance
    2. How to scale to complicated or large sources / rooms will need to be considered
    3. Using ML has a potential to automate the system, both in tuning and in scoring
16. Next Steps
    1. Validation of Audio Turing and the feedback to the reproductive system should be done
    2. A mock-up of the ML that would help with the tuning or scoring systems
    3. Understand the source and room impacts and how to reduce the complexity (and therefore cost) of the system 
17. References

[1] Action Item - Utterance Processing Guidelines for Audio Turing Test v0p1 - Dave Berol
[2] Olive, Sean. (2004). A Multiple Regression Model for Predicting Loudspeaker Preference Using Objective Measurements: Part II - Development of the Model.  http://www.aes.org/e-lib/browse.cfm?elib=12847
[3] (https://en.wikipedia.org/wiki/ABX_test)

section 6

next section

select a section:
1. Introduction
2. Workgroup Reports Overview
3. #KidsHearingMatters: Designing a Safer In-Ear Listening Experience (for Kids of All Ages)
4. A generalized haptic format standard leveraging existing MIDI infrastructure
5. Using Machine Learning to improve the experience in consumer audio devices
6. Audio Turing Test
7. The Future of In-Ear Voice Assistants (Assistance)
8. Developing a set of audio requirements for the next evolution of Bluetooth
9. Headphones Unlimited: Current Problems and Future Opportunities
10. Is this the real world? Or is this Fantasy?
11. Schedule & Sponsors