How to get audio from my Persee? Part 1.

Posted on September 2018

There have been several requests recently regarding the audio capability of Orbbec sensors. It seems many developers are not able to utilize the microphones at all in their applications with the information available on the forums. This article demonstrates that Orbbec's sensors do have audio capability despite the fact that they don't seem to be included in any of the SDK releases yet. I'll share a handy workaround and a few tests I did on its practicality.

First of all, let's state here that indeed the game developer's choice, Orbbec Astra SDK 2.0.9 Beta3d for Unity does not include any specific support for accessing the audio stream of the device. For those of use used to the good old Kinect SDK this is slightly disappointing, of course. Some of Kinect's voice commands were pretty handy supplement to gesture recognition in many applications. What Orbbec offers is basically (still) just the outstanding quality of the sensor device, but bare bones SDK for application development. So, we'll have to go around the problem by using the device channel and operating system directly. This is of course simple stuff for those who work a lot with device interfaces and perhaps less obvious for application developers.

As a good starting point for this short exploration, let's pick any of the free open-source audio recording programs available on GitHub. My pick is Antony Corbett's OnlyR for a single reason that it does not do much else than list all accessible audio devices, record a raw audio stream and encode it to mp3, all done conveniently in C# and WPF. What else would we really need?

So, let's get onto it! Download the sources or clone the repository and open it with Visual Studio (I'm using the 2017 Community version here). Run the program. First we can just without any sensor connected to the PC, just go with the default audio device you have.

Select the Settings icon and see that your default device shows as one of the Recording device options (here, Realtek microphone of my laptop).

Click back and start recording. Telling some new joke to your PC might be a good idea as a demo!

Then, let's plug in any of the sensor versions that work on Windows (i.e. not Persee). In my case Orbbec Astra Pro. Select the sensor as the Recording device and record something again.

There is a folder-looking button you can click to reveal the recorded audio clips.

After testing a bit I noticed something useful. There is quite a bit of noise in the Astra Pro's recording whereas my old laptop produces essentially no background noise. I add here a few samples for comparison to show what I mean: this is from my embedded Realtek mike and this one is from Astra Pro. Interesting difference, don't you think? I recorded both voices from the same relative distance of about 50cm to the mike with the only background noise in the room being my ceiling fan.

I'm not an expert in processing raw audio, speech recognition or anything, but let's say if you were hoping to get some speech recognition software to work together with Orbbec sensors, you might want to think twice. You may need additional signal processing and noise cancellation before anything works.

Be specific on your use case and think carefully how your sensor must be positioned in relation to the user. See what I mean? Compare this recording from about 4 meters away with the previous one. It doesn't sound like you can do much with the off-the-shelf offering Orbbec gives us if your application aims should include voice recognition.

Orbbec tends not to release much information about their future releases and roadmaps, so we'll see if there will be proper audio support in the SDK or we'd need to make our own workarounds every time. Here the solution works only on Windows in the usual WPF/.NET environment, but a resourceful programmer should not have much trouble getting the same piece of Corbett's code working as part of another program. And for the record, OnlyR works perfectly if you run it while your Unity game is on at the same time, the operating system takes care of the separate processes just nicely.

According to the specs Astra, Astra Pro and Persee have identical microphones, so I wouldn't expect any differences between the devices.

Hope this little test helps you to get started with Orbbec and audio streams. The next article will dig into the Unity and Android world and show how it is done.

By the way, if you utilize OnlyR in your applications, please make sure you read the license and give due credit to the developer!