Hands down the biggest problem all forms of VR have right now is showing someone who isn't in the headset what is going on. Putting a 2D image on a screen that shows me what you see out of one eye doesn't do much for me. It doesn't give me the emotional feel, no sense that you as the player are actually in the world you are viewing. It's a real problem, and a lot of very smart people have found clever ways to solve this in some cases with depth-sensing cameras and green screens. The results are a little mixed, and difficult to reproduce in every game, but still very cool.
Microsoft has its own solution. Originally, this solution was a way to show everyone what was happening inside of Hololens. It has been adapted for occlusive Mixed Reality headsets, and the results are ridiculously cool. Unfortunately, it's not something any of us are ever going to be able to do on our own.
This big, complicated box of wires here has three different very important parts. At the top, you have Hololens to act as a sort of virtual camera. Wherever the Hololens is in the real world, that's where the camera will be recording from in the virtual world. Under that you have a big DSLR camera, which is capturing a high quality video of the person in the real world. In this situation, Alex Kipman.
Under that camera you have a Kinect sensor. That sensor is grabbing a depth map around Alex and editing out everything but him. This means there's no need for a green screen, because the Kinect is doing all of the heavy lifting here. All of this video and real-time editing comes together to create what appears to be real-world Alex standing inside the virtual world.
The end result is just plain cool looking. Alex is represented as a silhouette in the virtual world so you can see as much of his "house" and the things he is doing in Mixed Reality. This capture method creates presence, it helps you feel closer to how Alex is feeling in the headset. You can see him physically reach out and interact with the virtual world. If he were playing a scary game, you'd know exactly why he jumped instead of just seeing a shaky camera on his face. It's a much more complete picture.
It's also incredibly challenging from a technical perspective. Not only is this rig prohibitively expensive and incredibly complicated, it requires a great deal of precision to hold the rig steady and perfect lighting to Kinect can do its job effectively. It is unlikely we'll see a way to recreate this effect in the home anytime soon, but it is one of several tools available to developers and Holographic Academy attendees at Microsoft's Reactor space in San Francisco.
