Fast Forward: Q&A With Dolby's Chief Scientist, Poppy Crum

Virtual and augmented reality were hot topics at SXSW this twelvemonth, simply the conversations extended across headsets. To be truly immersive, experiences need to incorporate all 5 senses. Sight, sound, affect, smell, and even gustatory modality.

Fast Forward Bug Art No company knows more about technology and human senses than Dolby, which has pioneered everything from surround sound to HDR imaging. I was fortunate enough to sit downwards with Dolby Laboratories' Chief Scientist, Poppy Crum, at the show.

Crum is also an offshoot professor at Stanford Academy in the Middle for Estimator Research in Music and Acoustics and the Plan in Symbolic Systems. Crum was at SXSW as part of the IEEE's Tech for Humanity Series. She understands audio and a whole lot more.

Costa: Poppy, thanks so much for joining the states today.

Crum: Cheers for having me. It'due south great to be hither.

So nosotros're going to talk well-nigh hearables, nosotros're going to talk about augmented reality, we're going to talk most maybe a little virtual reality, peradventure we'll talk nearly those painless migraines that the two of us experience from time to fourth dimension. First of all, your part at Dolby. What does a work day wait like for you? What do yous practice when you get to the office?

We have a big team of computational neuroscientists and people that are experts in sensory perception. If you look back at the history of Dolby and get dorsum, even 50 years, at the cadre of the company, it has always been an understanding of human experience. I think information technology helped differentiate how we retrieve well-nigh building engineering science.

And then on a daily footing, the people on my team and the people I work with, we wait across technologies. We're not just sound anymore, it's actually about a holistic sense. We have labs, and there are a lot of experiments that get on throughout the mean solar day.

Our new building has up to 100 labs, just nosotros've got some amazing biophysical labs. My groundwork is as a neuro physiologist—the same with a lot of people on our teams. And there'south human physiology that's happening on a daily basis to recollect about new technologies, and there's some very seminal work happening on thinking near how we experience information that'southward multisensory and actually looking towards how are nosotros going to consume content in the hereafter that is and then rich and what that'southward going to hateful for how it affects our bodies, how it affects how we engage with others and our senses.

1 of the things that you've talked well-nigh at the show and that we've covered a lot more than on PCMag is this hearables segment. Hearables is not a word that I think a lot of our audience is familiar with. When y'all hear the word hearables, what does that encompass to you? What does information technology mean?

Right now, I think information technology's a term that'southward still beingness defined. I similar to take it as a very large subset of products and possibilities. It'southward a wireless device or in some cases, even wired. Considering Amazon Echo I will call a hearable and that's plugged in, just it's a device that has microphones, or sensors, just it's capturing information from the environment, using that data in some style to enhance your experience. Also there are a lot of companies correct now that are thinking in the article of clothing space, a device that'south wireless, worn on your body, but there are also static hearables if we look at Google Domicile and Amazon Echo, and those are transformational.

{{ziffimage id="524402" A hearable doesn't necessarily mean that it's about only enhancing the sounds around you. It might be taking information about the sounds around you and using that to somehow requite you an enhanced experience in the world. You could think virtually it'due south capturing ... the ear, it turns out, is a great place to option upward biophysical information so that you can capture a lot of information there. You can imagine, plain, analytics capturing information just about the sounds effectually you, your conversations, using that as a way to enhance your day or optimize you lot.

In that location'due south also a lot of concerns with this. One matter we see that I retrieve is worth calling out, and I remember it's what's going to transform and aid define what this space are changes in regulations.

Right now, hearing aids are hearables. Hearing aids are an augmented reality device, but you lot have this class of consumer devices that have the adequacy to help mitigate hearing loss, have the capability to be an augmented reality device for someone with normal hearing, have the ability to even exist a gaming device. You're getting the crossover of these fields the medical device that's falling into this larger class and y'all're going to take a consumer device class that is clearly crossing these boundaries and doing a lot of similar processing for us.

In terms of hearing aids, people think of hearing aids equally in one case yous outset to lose your hearing, you can and then get a hearing help that will then give you lot some of your hearing back, just there are lots of interesting things that could happen if people started augmenting their hearing with ... they have normal hearing, only they desire to have something more than than normal hearing.

Absolutely. I'm e'er a large believer in not creating this arbitrary boundary where we say, "Now I have hearing loss." Our hearing, because of many elements in the world and sound, even aspirin is an ototoxic that you have to exist-

Is that right?

Absolutely. The combination of elements that you put in our hearing starts degrading in our early 20s and probably even earlier with some of the loud sounds that people listen to.

Especially S past Southwest.

Yes. Hearables have so many different capabilities you can have. Whether it's having control over streaming your content straight to your device - wirelessly - control of elements, the spatialization of information. Things we talk about --. there'due south been a big push in taking information, augmenting our visual sense with Google Glass or some of these devices from other companies, and actually - what nosotros want -- the sonic component of that [data] can be very critical to assuasive u.s. to get past what we might call a chapters limitation. And to take information from our world and really stand for it every bit a sound.

It seems to me that the thing that's made people think differently well-nigh audio and vocalization controls and vox interactions, but as well privacy, are devices like Google Abode and Amazon Echo, which are really the first mainstream voice interfaces that we've had for this digital world and it comes with all these different consequences, which I think we're just starting to sort out. Where do you come across that market heading?

That'southward a great question. I have to say, I think these devices are transformational in so many means, and I am a strong user and a strong sharer of them, partly to besides understand and look at where they're going to evolve to and how I use them in my life on a daily footing. The thought that people are willing to accept microphones e'er on, that's a large bound. What can we practice with that?

Right now, voice is a wonderful affair. Information technology'south giving people control; it'south getting them onboard to have this device equally an assistant in their lives. Ten years downwardly the road, I don't want to have to control more than things in my life. I want my devices to ... I'm going to trust my data more than I'g going to trust me to know what I demand in some cases and I desire the devices to be...

Anticipatory.

Admittedly. I want them to be proactive and capturing a lot of data nigh the sounds around me, whether I've been cough and modulate my temperature. Or just to brand my appointments for me when I need to or to be able also just to brand our lives moving forrad without having to control all of the devices.

Yous could imagine that the Echo could detect whether you lot've been sniffling. It could notice whether or not you sound like you lot have a sore throat, or if you've been coughing, and and then lets you know that it sounds like y'all have a common cold.

To become alee and schedule your dr.'south appointment for you lot before you [know you need it]. That sounds like a little far reaching to want that, but at the same fourth dimension I call up nosotros're going to become there. I retrieve the step of having voice control, integrating these into our lives volition make u.s.a. comfy with information technology.

In the futurity, the only time nosotros'll be uncomfortable is when things don't work and when it'southward gone and when it's absent-minded.

I had an feel with an Echo that allow me realize how much it had transcended the interactions that I was having and could aid people across different demographics. But yes, I think people take thought about these devices a lot for accessibility, which is wonderful. Similar the access that it can provide to both very young children or dissimilar historic period ranges or unlike people that...

People that have disabilities or shortcomings or gaps that could exist filled in using technology.

Absolutely. In my case, I had a relative that was in the hospital and he passed away a few weeks ago. I had bought him, right when Echo came out; I bought him an Echo as an accessibility device. Hither's the transformation. I took information technology to the hospital, and devices like this are phenomenal for a hospital setting. The privacy issues practise get important to consider, only nosotros use it predominately for music playing in that setting.

But at this signal and time, my relative was not very vocal, had not been speaking, and nosotros'd been playing music on the Echo that we thought was what he would want to hear, Bach and very calming music and honestly, some of the last words he said, and I'm not kidding, which I call up, were, "Alexa, play Al Green." He wanted to her Al Green, Sly and The Family Stone and this device gave him that accessibility. It was empowering and at a time where it was very powerful.

At that place are a lot of technologies at play at that place. Yous've got the fact that Al Green is available and that there's this vast music library that'due south a voice command away and then you've got the phonation command itself, which makes it possible for him to request it personally. So in that location's a lot going on at that place.

I think when you lot bring upward the privacy issue, that's another question is that before Alexa is making our doctor's appointments, I doubtable that some pharmaceutical company will be offer to requite u.s. a common cold medicine tabular array or our allergies are acting upward and offer a Zyrtec, and I call back that's almost an intermediate step we have to get through where who'south going to command all that data that is being captured and that we're giving abroad in this audio format.

Nosotros have to comprehend it. If we don't remember about the regulation side and nosotros don't call back about how we're going to make people comfortable with providing y'all, even more than, data than nosotros currently are. I recollect Amazon correct now says, "We are only listening when you say Alexa," just these devices to do what they actually can exercise you accept to be listening all the time.

There'southward a large trend right now of companies, insurance companies, whether it's motorcar insurance or wellness insurance offering consumers a bargain or a way of having lower rates if they allow their information to be tracked--if they give away their data. I think it'southward very powerful. I think it will be function of our future, there'southward no question about that, but that hereafter of what the ramifications of sharing that information, that hasn't fifty-fifty been defined even so, and it'south hard to predict. So, we really have to think about what that future looks like.

Also, I call up there's a permanence issue where people are, "I don't listen if a give my healthcare company or my insurance provider the number of steps I do every calendar week," but that information doesn't become anywhere, and the steps y'all do this calendar week will be searchable and indexed 30 years from now, and that idea of digital permanence which we alive with today, which really, in human history, never really happened before. When you add the fact that you could accept a microphone on in your kitchen all the time, all that information will not go anywhere. Amazon volition always have information technology.

That future is unwritten, and we don't know the ramifications. We haven't defined these regulations, just those tin can fifty-fifty change in the future because many things can change.

And hither'south another thing. I think culturally, the Eu has tried to enact much more than consumer protective regulatory legislation correct now, but it'south not clear what it means because the information exists. We need to ensure interoperability when you're traveling. Nosotros need clinch security for minor IoT devices. That'due south a actually critical thing. I think groups like NIST are very active to solve this.

When y'all expect at what Dolby does and all the technologies that they're working on, y'all get-go to run into mutual themes, and one of them is that the visitor is actually trying to requite humans super sensory perception, superhuman powers. It sounds over the top. Information technology sounds hyperbolic, just there'southward a bunch of examples of people getting superhuman powers using applied science. Can nosotros just talk about a couple of those?

Admittedly. So I come from a background equally a neurophysiologist, who'due south thinking about how nosotros integrate these things to technology and there are a couple of things that are important. When nosotros think about what augmented reality is or what technology can do for united states today ... when I beginning joined Dolby, maybe we were in the process of working on Dolby Vision and Dolby Vision is a high dynamic range and wider color gamut imaging technology. Merely to get an idea, the typical display you would take bought nearly iii years agone would have been 300 to 400 candelas per foursquare meter. The moon, the natural moon is about ane to two thousand, sunlight on black pavement, you're getting upward to 15,000 candelas per square meter.

Then display engineering science was very far away from what our actual sensory system could handle. And in the process of development, nosotros were working with some content and devices that allowed united states of america to become up to 20,000 candelas per square meter in the brightness that was produced-

And this isn't necessarily contrast, this isn't resolution, this is the brightness.

Aye, from a perception and sensory perspective side, for a lot of viewing distances, we're pretty maxed out in what our sensory systems, your perceptual system can handle regarding resolution. But regarding effulgence and colour gamut, nosotros were no where near. So it'south pretty exciting to be, "okay there's this area that nosotros tin raise the feel, and we tin lead in this field." That's pretty exciting.

{{ziffimage id="524403" So in doing these studies, I call back what made me realize how powerful this could be, and likewise redefine what I thought immersive technology should be nigh was this. Nosotros were watching some burn down content and the candelas. I think we were playing at almost four to six thousand candelas per foursquare meter, and I was watching information technology, and I felt my face react to but the burn. This man was whipping a flame at me, so there's this big specular, really vivid flame and I thought, "Oh." The display itself must be producing the heat. Then I talked to 1 of the fundamental developers and asked him. He was like, "Information technology should be constant." So I get a thermal photographic camera and used thermal imaging to track the screen. It was totally constant. Simply we were able to bear witness consequent changes on people'south faces that were content dependent.

So just based on the luminance that was reaching the retina, my encephalon says, "Okay, I've never experienced flame that was this bright that wasn't real, so probabilistically my encephalon is going to behave with that fire as if it is real."

So now when we think about immersive technologies and creating superpowers and creating experiences, we tin be thinking about them engaging our physiological system in a natural mode, an accurate fashion, and then nosotros can raise them. Nosotros have the ability to go across what you might experience in the natural world, and that becomes pretty powerful.

Yes, it's a great instance. We've also talked a piddling bit about how in PCMag we've tested a agglomeration of ER systems. There are very promising technologies in the works there. We ever focus on what it looks similar on the screen, on the resolution, only the audio component to virtual reality and augmented reality is just as of import equally the visual component and helps sell the experience. Can you talk a little chip nearly what you're finding there and how much that adds to the experience?

A great trend beyond, non only across our visitor, but others, is that recognition that yous tin can't study one sense without the other. If we're looking towards hereafter devices and holistic devices in AR and VR. Right at present, all of the main VR devices are visual, only a hearing aid, every bit far as I'thou concerned, fifty-fifty though they might not want to be called this, is an augmented reality device. It'south trying to augment from the base physiological organisation, the state of it. Cochlear implants, are one of the first implanted biophysical devices.

If you think about what do we desire in augmented reality? We desire to enhance our experience of the world. We want it not to be something that interferes and gives us too much data that we tin can't do the things we used to do. Right now, we're in the trend of really cool experiences, only nosotros oasis't enhanced and touched in the way that I'k looking forward to making the mundane activities in our lives amend. And allowing us to exist more engaged with people because those things are enhanced. I'll give an instance of that in a sec, simply really, we want to be connected. We want to exist engaged with our lives. We want to exist able to have in more information. We want to take heightened experiences and we want to exist able to have command over how rich these experiences are.

So, if we take all the data that nosotros can capture or ... more than data is not e'er better, only if nosotros are trying to raise our visual ... I've seen this happen in the military, I've seen it happen with technology where I desire to create an AR, and I take information that I'm trying to enhance, and I put it all in my visual system. I put it all in glasses, or I put it in places where I have to look at it. Well, our brain can't process that much information. We can't take it in at the aforementioned fourth dimension.

So that'south where sound ... we accept a chapters limitation in one sense, only guess what? We have other senses. And of a sudden we open up up this whole store that we can go and actually use to enhance our experiences, and guess what? This is cliché, but regarding creating a physiological experience, what practise I exercise when I'm watching a scary picture show and scared? I turn the audio off. So not simply is that a really visceral office, information technology'due south a whole opportunity to heighten and enrich our lifestyle in terms of how much information we can have in, and how much information we can process. Only having voices separated or sound elements spatially disparate, we can process them at the same time instead of having our brains and the information be completely useless. It prevents the states from having all of our encephalon and cognitive chapters taken up by doing that one thing. It actually opens upwardly our situational awareness and what nosotros can do.

The audio component and all these VR things, as we start testing them, that'southward something we're putting into every review, is that information technology looks like this, it sounds similar this, and and so the two unremarkably should add up to something greater than the sum of the parts.

In that location are then many different things nosotros can practise with sound in AR or VR. Apparently creating a spatialized experience is important, but information technology is a central role of it. Dolby Atmos is an example of an object-based audio commitment and rendering. So when things are authored in Atmos and the sound elements alive with a information package, a information stream attached to it. Then, you can imagine, that tin can be very powerful. That can tell you where a sound should be. Regardless of how you're recreating that audio, it has that location. It has a width, it has a loudness, but it tin also GPS coordinates. It can too have all sorts of data. We can have emotional content in the future about how it's supposed to affect you if we know something virtually the interaction that's happening.

These are the types of hereafter things that could happen. If you think nearly an AR device, it can provide a lot of information. We currently use maps to look down at our telephone or in a auto. These things tin can exist done in a lab very easily to create a sonified version that'due south giving you tracking information and telling you, not verbally, only giving you pings that tell you where sounds are, where you should be going. The problem is that nosotros have to make certain it'due south robust and consistent across users and that's a more complicated problem.

Mappings a corking example that it is still screen based more often than not and when Google Glass came out, yous could see all this data, this little tiny little window in your visual display. What I really wanted was for it to tell me in my ear whether I should turn right or left.

Yep.

And you're right in that location, right adjacent to the ear, y'all could just requite me the vocalisation control. I idea it was a better audio system and audio organisation than the visual system regarding wear solutions.

Absolutely. And again, well, non again, I think you've hitting on the master point, which is these are all tools, and right now, today, we take tools that we can use in amazing ways, including neuro control and things, merely the question is not let'south ... it'southward picking the correct tools for the data, for the engineering, for the user experience that you're trying to target. You need to contextually optimize for and personalize for your individual and for what they're trying to reach, and that's been a big gap in what we're building correct now. Amazing technology and it'due south going to just get richer. But it's going to move around and I think we have to embrace what nosotros're trying to get to and think almost how, every bit a homo arrangement, we can best feel that.

One of my favorite things to do is to look at other species. Yous can wait at whether it's a frog or a wing, or a bat, species that have these amazing superpowers to practise things that go way across what their physical bodies and a fibroid look at their brain might tell you. Understanding how they solve that problem and more, why their evolution solved that problem for them, gives united states insights. We have to retrieve that manner. Nosotros have to think more about what are the environmental pressures on our systems, and how is engineering going to best help us solve that?

And delivering more and more information probably isn't e'er the right solution.

Rarely, rarely. I mean, what makes us successful a the human is what our brains throw away. I'm e'er talking about illusions, I beloved illusions. They're fun, only they're also this beautiful insight to when our brains become things wrong. If we experience data exactly as it is in the world, you actually end up with a pathological status that yous're non functional in the world. And what our brain is constantly doing is helping us weight certain information and not even notice different information. If we noticed all the luminance changes created by low-cal, we would never see holistic objects. We need to part, and we need to know what to activeness on, and that's what'due south powerful.

Let me ask y'all some questions I ask all my guests. Number ane, what is the thing that is keeping yous upward at nighttime? What engineering science trend are you most concerned virtually going frontwards?

Ii things. Ane I think nosotros've already touched on, there's and so much ability in creating these sort of superpowers, and what we can do with AR and VR. My hope is that nosotros look at all the technologies we're building as tools, and we call up nigh using the right tools in the simplest and the most, I say robust, but information technology's consequent across all our users to become to the experience that's intended rather than the i that's the most sexy and shiny.

The other one though is--it'southward really near the data. There's so much ability to having microphones e'er listening that will move our technology frontwards, move our lives forwards. Just nosotros have to exist comfy with what's happening to that data. We have to be comfortable with regulations whether it's self-driving cars or hearables, or just the security in IOT devices. If we don't have strong regulations and standards and understanding and, in many cases, transparency to the algorithms driving some of these probabilistic decisions and the AI backside some of these devices, I think it volition filibuster and agree back innovation. It could delay the capabilities of the engineering to accept major impact in a positive management for so many people. I want to see that happen. So I think it's going to be about standards and regulations to aid brand that come up to life.

On the positive side, what are y'all most excited about? What do yous think is going to alter the world and gets yous excited about going to work every day?

In that location is a trend and a shift that we don't recall about i sense. We're thinking about technology holistically. We're thinking about the touch it has on sight and sound together Nosotros think of our unabridged physiological organization and, I think, how we feel, our emotions. The things people intendance about now and the way nosotros describe the impact of technology is much richer regarding what it means for man connection, and what information technology means for only the power it can accept on our experiences. That'south something that I definitely think is a new direction.

So, regarding a product, a service, a applied science gadget that you dear, that changed your life. Is there annihilation that stands out that yous're like, "This is the matter that makes my life better every twenty-four hour period?"

Static hearable devices. I use an Echo every day, and I've seen it used across such a broad generation gap and use cases. I'chiliad really excited where technologies like that are going and right now, it'southward really in its early on stages of the impact it'south going to have.

So if people are listening to this talk and they're interested in your work, how can they find you online, interact with you lot?

You can discover me online at LinkedIn, Poppy Crum. You can also discover me at on Twitter @poppycrum.

For more than Fast Forrad with Dan Costa, subscribe to the podcast. On iOS, download Apple'south Podcasts app, search for "Fast Forward" and subscribe. On Android, download the Stitcher Radio for Podcasts app via Google Play.