Chris Grigg, Beatnik

Table of Contents

Executive Summary
The Next Generation Audio Device
Imagining the Personal Audio Network
Surveying the Pros and Cons
The Big But
Finale
Appendix A—Other Resources
Appendix B

Executive Summary

This report represents our attempt to consider social, cultural and technical developments molding the next major evolution of audio products and communications technologies.

Recent technological innovations frequently share a common denominator: they allow people to interact anywhere, anytime with portable electronic devices. Even so, the scope of that interaction is surprisingly limited to little more than text messaging and sharing photos and brief video snippets from not-quite-smart-enough mobile phones. Incremental changes will no doubt appear next year, and the next. The Ear Pods group wanted to look further out, beyond the next-generation of products, to the nature of technical innovation as the 21st century matures. We found ourselves believing in the inevitability of certain developments and their self-evident qualities.

There are social and technological forces already in motion that will virtually guarantee the creation of an evolving breed of miniaturized audio products with new communication protocols, location-based capabilities, audio enhancement processing, and completely new media playback and communal audio technologies. Riding alongside these developments are a host of social, legal, and moral complexities, none of which will be trivially solved, but our task allowed us to acknowledge these hurdles and focus on the technical possibilities.

Audio and telecommunications industries have an unprecedented opportunity to influence and accelerate the introduction of next generation of business and social communications via “personal audio networks.”

The Next Generation Audio Device

The group had only 24 hours to brainstorm about a wide range of topics related to the future of a connected audio lifestyle, but even in this limited time, a few essential characteristics presented themselves. What’s a Personal Audio Network and why would you want it? Let’s trace back to the immediate past, our present, and project forward.

Where We Were

Not long ago (the deep 1980s), personal communication devices that could reach beyond the traditional world of short-range radio (e.g., “walkie-talkies” or Citizens Band) were rare, expensive, and huge. These “cellular phones” would not play recorded music—they were large and had one function: communicating by radio with the Plain Old Telephone System. Still walking in the park and making a phone call seemed revolutionary, even if the limited battery life didn’t fully cut the tether of the phone cord.

At approximately the same time, new forms of mass produced digital recordings appeared, and with them, devices that would play the media. These music playing devices had no communication, and were limited to playing one piece of media at a time—to provide yourself with a range of choices, you had to carry lots of individual recordings.

Where We Are

Skip ahead to a world where a product advertised as offering “a thousand songs, in your pocket” (an Apple ad from 2006) seems like it’s skimping on storage.

Just as today we reach back two decades to remember communication and audio devices with profound limitations in size, battery life, and (for audio) a limited ability to carry a variety of instantaneously selectable playable music, today’s products will eventually reveal their startling limitations. They’ve added the capacity to hear a huge selection of music, and in some cases they’ve even made it possible to acquire more without marching back to an ordinary personal computer and reattaching a cable. But the devices themselves are as personally isolating as the noise-reducing headphones frequently plugged into them. The devices separate us from each other—we become solitary islands of sound.

Today’s mobile audio products afford instant gratification in a potentially wide range of musical choices, but the experiences isolate—individuals carry their own players, with their own head- or earphones, listening to individually selected audio in sequester, an imaginary audio environment inside the head. Even new audio products and technologies that purport to “network” people with one another are really only more effective ways of transferring audio files from one device to another, often with frustrating, arbitrary-sounding limitations on how and how often this audio may be played back. They can’t connect people with an actual living audio experience.

But this paints perhaps too dreary a picture. Many of today’s most exciting consumer electronic products are portable, battery-operated, and afford more possibilities than we could have imagined a generation ago for making one’s audible life mobile and immediately gratifying. Yet equally huge changes are on the horizon.

Where We Will Be

Imagine if the experience of putting on your headphones made it possible to connect more effectively and more meaningfully with every audio device you used, whether in your pocket, at home, in the car, or at the office. Imagine connecting with other people around you, and enhancing your ability to control your audio environment as your professional and personal needs require. You’re beginning to imagine the Personal Audio Network (PAN).

With the right enabling products, wirelessly connected both to the world-at-large (the internet) and potentially to every friendly device in the immediate vicinity for two- or multi-way communication (through PANs), people will have new ways to hear what they wish and socialize with those around them.

We’ll wear audio delivery devices on our ears the way we wear contact lenses on our eyes.

Imagining the Personal Audio Network

What are Personal Audio Networks (PAN)?

In more technical terms, a PAN is a wireless audio mesh network—it provides a bridge between each of your devices and, even more importantly, the devices of other people around you.

In the future we feel is inevitable, the PAN becomes an extension of the personal device you carry, providing a totally connected resource: telecommunication, wireless networking, and audio.

A PAN offers these possibilities for audio experienced through an iHear:

• Record and playback of audio environment
• Integrates the function of multiple audio devices (e.g. phone, conferencing, MP3 player, earpod, car’s navigation system, audible product data sheet or museum objects)
• Controlled access to multiple audio streams near you
• Treat telephony as conversation, diminishing the difference
• Streaming from your server or local hub
• Meta tagged streams to inform the user and allow smart devices to automate desired features based on embedded information

In its most limited definition, a PAN Device enables your participation in the potential audio network that surrounds You could be forgiven for assuming that we’re referring to upcoming generations of mobile phones; just as today’s phone bears little resemblance to the (at the time) first blocky, bricklike cellular phone of the early 1980’s, a PAN Device will seem even more advanced.

PAN Devices will free your hands by moving more of your activity to voice commands and audible cues—you’ll use screens and buttons when they’re most necessary.

What’s an iHear?

“iHear” became our pet name for a PAN audio device that we feel most fully realizes the potential of a truly networked lifestyle. Early generations of an iHear will suffer the usual indignities of a product just slightly too far ahead of its time:

As we expect it to be, a PAN Device is always on, always ready to connect, and always at the service of the user’s preferences and situational needs. It plays audio into your ears and hears your speech.

Through ubiquitous headsets and microphones wirelessly connected to personal audio networks and devices with advanced audio processing, we will all control how, when, and what we hear. This will allow us to enhance hearing like corrective lenses improve our vision, filter noise or audio like sunglasses, and extend audio perception like x-ray specs.

We called it iHear.

Figure 1 A First Generation iHear

We assume that devices will continue to shrink physically and grow in capability, but it was pretty fun brainstorming that first prototype.

An iHear serves as your personal hub to your own PAN, presumably in an environment with multiple devices, some general purpose and some highly specific, and with many users listening and speaking on the hub of their PAN’s. Your PAN Device connects to other devices according to your preferences and the preferences of those nearby; those seeking privacy or just a few quiet moments to themselves don’t appear on the network. (Note: we discussed and decided that the question of whether “all audio devices will consolidate into one” is tangential to our notion of a PAN and various devices all participating in a shared network.)

The iHear also allows you to go fully hands-free in more situations. A new and richly defined ability to recognize your speech commands and give you audible cues and alerts for important messages and events let’s you keep your eyes on the world around you more of the time.

Figure 2 The Future iHear

Examining the Use Cases

Discussion of PAN Devices and iHear gets a little abstract without rooting it in some actual requirements and a brief overview of use cases.

The group explored a range of scenarios to guide our discussion of capabilities.

Walking home from work

1. listening to music
2. call comes in, music fades out then pauses, hears ring tone
3. takes a call, hangs up, returns to music
4. emergency vehicle beacon/siren, music pauses then returns
5. enters home, transferring to home network
6. joins multiplayer game
7. listens TV in bed

Attending a large party

1. enter house where there’s a large party, Jim is given a choice of playlists
2. some are dancing by sharing the same music
3. someone says Jim’s name from across the room, his music fades
4. last sentence is replayed, “I wonder when Jim will get here”
5. Jim records, “George, I’m already here”
6. George is sent your message
7. George is notified a new message has arrived, waits for Linda to finish her sentence
8. George asks Linda to hold on, he listens to the message spatialized to Jim’s location in the room
9. George links Jim into his conversation with Linda
10. others are shown status indicators that three of you are busy

Trade Show

1. you begin broadcasting your interests, and people you’d like to meet (by name or title)
2. you stop at a booth due to an audio alert, of a product that matches your interests
3. walk up to the product and point to it, you hear a recording of a data sheet
4. you get a buddy alert that George and David are near you
5. you get an audio alert of a highly ranked person matching your interest is walking past you
6. you learn his name, and he speaks German, using babelfish you say his name and start a conversation

Conferencing

1. you call into a team meeting from Starbucks
2. you hear a podcast of the agenda, and then log in
3. using spatialzation you hear the chair front and center, and colleagues from around the table
4. you reach the counter, mute your conference input, order coffee, return to conference
5. you replay the last 20 seconds, at an accelerated rate to catch up to real time
6. chairman notes action items, you receive as a meta tagged message

Other use cases

• dating
• military
• advertising
• international travel
• public performance (volume and frequency adjusted, without audience noise)
• super power hearing
• multi player gaming

Reviewing the Product Requirements

Based on many of these use cases, and others that might easily jump to mind, requirements for the universe of personal devices that populate a PAN would include:

• Environmental discovery
• Raw/lossless data
• Low latency
• Simultaneous streams
• Meta tagging (side channel)
• Multi-point security
• Hubs carry access policies and preferences that can be transferred
• Low power, long battery life
• Speakers and microphones for each ear
• Gesture based device control
• Voice command
• Status indicators of headset for social awareness (mic on/off, available/busy, pulsing to the beat, etc.)
• Industry standards
• Emergency overrides (fire trucks, honking cars, trucks backing up, etc.)

Surveying the Pros and Cons

The group hopes that the benefits of an always-connected, always available verbal interaction and recording personal device network (that processes and enhances audio) is clear:

• Anti-noise
• Podcasts
• music
• phone
• games
• mobile video
• themes for my environment
• audio Tivo
• audio mesh
• better than your natural ear
• concerts, theater, and live events
• product info and advertising
• museum exhibits
• babelfish
• replacing SMS/IM with short recordings
• excluding others (staying hip)
• toothing (meta world, google, myspace, buddy lists, recognizing others)
• Audio alerts (e.g. proximity alerts from buddy lists, data net listening for keywords such as your name)
• cell phones become internet devices
• everyone’s mic is always on, I can tap in to your stream (to have a conversation with others at the cocktail party from across the room)
• combined with heads up displays
• taking voice notes with your recordings
• talking to yourself as a witness/observer
• hearing cop’s radar
• Jewelry/fashion statement (e.g. Swarovski and Philips already shipping, and possibly solar-powered audio hats “hubcap”)

The Big But

We find ourselves excited by the possibilities an iHear could afford us. Even so, we acknowledge that developing the basic technologies and supporting business models entail some obvious and entangling legal and social implications, including:

• privacy concerns
• public safety concerns
• intellectual property issues
• physical and mental health concerns of a “perpetually connected” lifestyle and the products themselves
• marginalization for those who “opt-out” or cannot participate financially

Each of these issues could supply graduate students with an entire research project, and they were certainly beyond the scope of our two-day discussion.

Finale

In a world where new opportunities will appear at every turn to interact with those around us, our iHear “contact lenses for the ears” will be meshed in Personal Audio Networks. Each of us will use PAN devices not just for their basic communication and entertainment functions (getting and sending text and video, etc.), but iHears will allows us to use audio to advantage, both for its own sake as artistic experiences and as a basic enhancement to daily life in all our interactions.

Audio and telecommunications industries have an unprecedented opportunity to influence and accelerate the introduction of next generation of business and social communications via “personal audio networks” and the products that enable them.

Appendix A

Other Resources

Appendix B

Peter Drescher, BBQ Brother and Ear Pod teammate, delivered the following address at the Audio Engineer Society’s 2007 Convention’s “Game Audio for Broadband Phones” workshop, which he chaired. It appears here under a new title, but otherwise unedited.

A Day in the Life of Joe, Personal Audio Network Pioneer

First thing, of course, is coffee, and as Joe enjoys his morning brew, he unplugs his mobile device from the charger, puts on his bluetooth headset, and checks the news, weather, and sports reports, before getting his eMail. It's such a gorgeous morning, he does it all from his front porch, since he's got broadband connectivity everywhere he goes. He even watches his favorite video blogger rant about President Obama's re-election.

During the commute to work, Joe checks the items in his online catalog, and notices that the new Spiderman game is available. A few button presses later, he's web-slinging his way uptown, and enjoying the way the "thwip" sound seems to shoot out and away from his mobile device. But then the game pauses, and the Darth Vader theme song plays incongruously, with a screen indicating an incoming call from his boss ... Joe sends it to voice mail, and will to listen to it later.

While the game is paused, he selects the "gameplay music" menu item, which takes him to a submenu of his iTunes playlists. He notices a "recommended songs" option, and clicks it out of curiosity. An iTunes screen appears displaying a number of playlists, that can be used as background music during the levels. The first one, of course, is the official movie soundtrack album, remixed for gameplay. Then there's popular DJ mixes of songs from the movie, some user compiled playlists of heavy metal and industrial goth, there's even some music written specifically for the game by somebody called "pdx" (he must live in Portland, or something).
 
Joe, being a purist, wirelessly downloads the movie score, and a few minutes later, he's webbing up bad guys while grooving to the Danny Elfman theme. But only for a few more minutes, because now he's at the office, and logged into the corporate network. He works at his computer, listening to some deep house grooves, and talks on the phone, switching back and forth easily, by tapping his headset in a specific way.

A friend stops by to gossip, so Joe turns his music off, and turns on the external mic. His friend does the same thing with his headset, because he wants to show Joe the outrageous YouTube video everybody's been talking about. The friend pulls out his phone, taps it a few times, and plays the video. The audio is streamed to both headsets, and about halfway through, Joe can clearly hear his friend say "Here it comes!" An office worker passing by is startled when Joe and his friend suddenly, and without apparent reason, laugh simultaneously.

After work, Joe gets a MySpace alert on his phone, telling him about a party his friends are going to. He uses the phone's built-in GPS locator to navigate to the venue, passing by a group of kids, sitting on a stoop, all wearing matching headsets, all nodding in unison, to a pounding beat only they can hear.

When he gets to the party, he finds a group of people involved with a multi-player tournament with some folks in Saskatchewan, and others in South Korea. Each player is looking at his own device, but they (mostly) share the same audio experience. Joe joins the game, and when he makes a really good move, he shouts, "yeah!" ... and opposing players all over the planet moan in dismay.
 
Later, he chats up a cute girl by noticing the illuminated quicksilver headset she's wearing. It reminds him of the in-ear monitors stage musicians wear, and she show him how they glow and pulsate in response to the music she's listening to. Joe tunes his own headset to her frequency, with her permission, of course, and together they dance to the music. Before he leaves, he takes her picture, enters her email and number into his phone's address book, and assigns her a ringtone of the song they were dancing to.

When he finally gets home, he watches a little late-night TV (streaming the sound to his headset, of course) before removing the earbuds to go to sleep. As he plugs the phone into the charger, he realizes he didn't take his headset off once, all day…

section 5

next section

select a section:
1. Introduction  2. Speakers  3. Executive Summary  
4. Th Unfnshd Smph... Fixing Broken PC Audio
5. iHear the Future
6. Overcoming Roadblocks in the Quest for Interactive Audio
7. Call for a Highly Distributed Metadata
8. Game Producer's Guide to Audio
9. Next Generation Hand/Glove Controller
10. The Computer as a Musical Instrument
11. Schedule & Sponsors