The best event for Persee is no event!

Posted on March 2018

For quite some time now I've been updating my old Kinect game installations. Astra Pro is fabulous and porting old apps from Kinect to Astra is almost too easy.

Now I've been attempting to turn the games into something that run smoothly on Persee. Although there has been much promise of a bright new future, things are not at all that rosy.

In this article I share some of tips and tricks for those who:

Know what they're doing with Kinect and Unity (i.e. this is no entry level tutorial)
Hope to get an old Kinect game ported to Persee
Do not want to use any of the old middleware solutions to keep things clean and lean
Already got (or hopefully are only about to get) frustrated by technical limitations

Here's the long story short. I got the Orbbec Astra SDK 2.0.8 Beta2 package for Unity 5.5.4 released in February and I'm using Unity version 5.5.4f1. My game app is similar to the classic Summer Fishing Game:

Video view shows in the background
The foreground is filled with animated and static overlay props
Joint overlays on hands provide interactivity upon collisions with the targets of the game

Nothing major.

From beginning I had a feeling I'm attempting something that those brilliant Orbbec guys haven't intended to support in the first place. Persee is not exactly supposed to be a stand-alone gaming platform, of course! But for my purposes it is perfect as long as I get my old content running on it... even if I have to simplify a lot of things. After all, who would deny that my installations get a lot simpler if I can skip having a laptop on site? Not to mention cheaper also.

For the code parts, I've already overwritten everything about the old middleware, so the Unity project has no dependencies to anything else than the above-mentioned Orbbec's SDK. It compiles to Android platform just nicely. Also the sensor itself is the same, Persee with its mini-computer is no Astra Pro with laptop. Of course there are big limitations! Now, the problems only reveal themselves once you've got your game to run on Astra Pro without a slightest problem and your APK is installed on Persee:

Persee supports only 10fps for video image, which looks too sticky on a 42'' (or larger) screen when people move a lot. So I must change the user representation completely.
I've tried background removal, but it won't run essentially faster either. I only get about 15-20fps and the image and all animations freeze every few seconds. This won't do either.
I've attempted all the available image resolutions and concluded that any combination of the better video image resolution settings doesn't seem to run fast enough. And the ones that run fast enough have simply too few pixels.

Then what? Hacking time, of course! After spending (too many!) hours of trying this, trying that, I've started to question my assumptions. What if it is the SDK itself that has the biggest bottleneck?

At this point I immediately started to suspect that passing the data within the application level in Unity is sub-optimal. So, the remainder of this article shows a simple, ugly and ahh so efficient way to get things running on Persee at the speed you'd like to have.

At the first glance I thought the nicest and most elegant part of the API is its event-driven nature. The events are supposed to be bound to the scripts that receive the sensor data in Unity Editor like in the previous Skeleton Renderer's case. But we'll skip that on purpose. After commenting out the serializable events they don't appear in the Editor at all.

So, let's first disable all the events! Of course breaking the SDK is a bad idea in general, but in case of yanking up the performance anything is legal... in my mind, at least. Basically, we only need to modify the AstraController.cs script.

Now, instead of event binding we'll use static variables. My solution is ugly as hell, but more efficient.

Let's add a couple of variables to save the day:

LatestBodies for accessing the Body array when the data is there
emptyBodies for empty data in read-only mode when there is no data

Then add simple handling for getting the data from the sensor to our static variable. That CopyBodyData() method seems to take quite a bit of effort to process, so be sure it will never be called more than once anywhere else in the app. This being the case, the best place for it is in AstraController.cs and nowhere else.

Now, obviously all scripts that used to have a OnNewFrame() function will use Unity Engine's Update() instead to access the data. As the whole app will run only at 30fps at best, there's hardly any need for controlling and synchronizing anything more than what you see here.

Finally, you can toggle off all unnecessary streams to reduce computational effort. Don't be shy. Either comment out those Start() calls or even better add public booleans to toggle their usage on/off.

You wouldn't believe but framerate jumps up those missing 10fps just like that! Ugly but efficient.

After going through all of the above, and spending hours in testing, hacking and benchmarking, here are my concluding remarks for general guidance:

Forget using any display option that uses the live video stream on Persee, re-design your app for this part
When performance problems occur, the first thing to do is to override the use of the events in AstraController
Never ever call CopyBodyData() or similar functions more than once, they're best used within AstraController