Insights

Virtual Block Stacking: Creating and Interacting with Holograms

Blog Holo Tower V1

By Id Raja

Software Engineer

This post was co-authored by Nerdery Solutions Architect Michael Falkner.

In hosting Night at The Nerdery during Techweek Chicago, our visitors got a Willy-Wonka-like experience – a world of pure imagination. But instead of Oompa Loompas, Nerdery guests interacted with some very cool tech creations – all created by curious Nerds who’ve pushed boundaries on passion projects to show what’s possible during Techweek. Here is one of these creations, and subsequent posts will reveal others.

The inspiration for HoloTower, a Nerd-created block-stacking innovation, came from exploring the basic capabilities of Microsoft HoloLens and our attempt to distill the capabilities in a fun, easy-to-learn way. The HoloLens allows a user to see, hear and interact with holograms within a natural environment such as a living room or office space.

The HoloLens is different from other virtual reality headsets because it allows a user to view holographic objects without completely occluding the user’s environment. Other virtual reality headsets like the Rift require a user to put on a headset that totally blocks their vision, while the HoloLens can show virtual objects without blocking out reality.

Simulated Environment Types

Technology moves at a fast pace and different ‘realities’ are continually evolving and changing. The types of simulated environments include:

  • Virtual Reality (VR) utilizes a headset that goes over the user’s eyes and blocks the physical world. VR often requires you to be tethered to a computer due to the processing power required to produce the images you see in the headset.

  • Augmented Reality (AR) modifies VR with a display that does not block the user’s vision but instead overlays information onto the environment through a display, like a mobile phone. AR has been gaining momentum, with the popularity of Pokémon GO, and more recently from Apple’s announcement that it would support AR applications in its release of iOS 11.

  • Mixed Reality (MR) combines elements of VR and AR together. The user wears a heads-up display like VR while adding two- and three-dimensional elements into the user’s environment.

The HoloLens by Microsoft is the first major step in MR. It has its own self-contained computer — so, no need to tether — as well as speakers and a headset which are the key to displaying holograms in the physical space in front of you. But the HoloLens capabilities aren’t just limited to its ability to display objects, it’s also constantly learning about the space around you. Without the spatial mapping and understanding capabilities of the HoloLens, a hologram subject to gravity would fall forever since it would never encounter an obstacle, like the floor for instance. The HoloLens builds virtual objects that correspond to actual objects that the holographic objects interact with, making collisions with walls and furnitures possible.

Hologram collision example

For example, these holographic blocks are piled up on the floor because the HoloLens told the hologram where the floor was.

Holographic blocks on floor

At its core, creating and interacting with virtual objects utilizes a lot of gaming technology since video games have been doing this for years. A widely-used tool in the HoloLens community is the Unity3D Engine. It uses a robust 3D interface which gives you a visual concept of the X, Y, and Z axes, but also has seamless integration with Microsoft Visual Studio for scripting in C# and deployment to the HoloLens. Unity also has a remote connection app within HoloLens which allows you to test changes on the fly without the need to deploy to the device.

Since HoloTower only requires a single Hologram type — the brick — we used Blender to build the brick and apply the visual texture to it. Within Unity we built the tower of bricks, and Unity was able to handle the physics of stacking the bricks. But remember, these are virtual bricks in a virtual world, which means bricks often overreacted to gravity by falling or colliding with one another and falling over without being touched. We had to tweak the engine to make the bricks respond naturally by adjusting gravity, mass, and friction. Once we were able to get the tower stable we used the HoloLens toolkit to help us interact with the bricks.

Jenga Tower in Unity Game Editor

The primary functions of HoloLens are Gaze, Gesture and Voice.

Gaze: Unlike a computer, you do not have a mouse and cursor to select an object. The HoloLens relies on your head and eye position when selecting an object. We created a cursor to give users a visual representation of where they are looking. We coded in a response from an object when it was being gazed at by having it change color. This gave the user feedback that they were currently looking at a specific brick and it was ready for a gesture.

Gesture: Similar to gaze, there is not a mouse for clicking and dragging. The HoloLens understands specific gestures when interacting with an object, like the "tap" and "pinch". When a brick turned red, it's selected when you tap or pinch. Once it's selected, you hold your gesture to move it and stop the gesture to release it.

Voice: We kept voice controls simple. When a user wants to reset their HoloTower board they would simply say “reset” out loud, and the tower would reset to its original state.

Person using HoloLens to play holographic jenga

The final result? You're able to gaze at a brick, use the pinch gesture to select and drag it, then place your brick at the top of the tower and hope it doesn’t topple. The physics in Unity makes other bricks move if you collided with them. Once the tower topples, you simply used the voice command to say “reset” out loud and reset the tower.

You can see the final product in action here.

Application

The ability to augment the real world with holographic objects offers a lot of benefit for businesses. From space planning, to hands-on training before touching expensive equipment, to informing doctors during surgery. The possibilities are endless and can save businesses money by allowing them to make informed decisions before making major investments.

Space Planning: When decorating a room or opening a new store, one of the hardest things to do is conceptualize how furniture will fill the room and look. Mixed Reality allows users to arrange virtual objects without the fuss of ordering and returning product.

Training: Today, many training applications happen in 2D on a flat computer monitor with a mouse and keyboard. Imagine having a guided tour on the equipment using Mixed Reality. A user can look at a machine and the HoloLens will provide an overlay explaining each lever and button and give the user direction on how to perform specific tasks with visual cues.

Medical: Much like training, medical MR can enhance a doctor’s understanding of a patient and difficult procedures. There was even one case where a doctor was able to overlay the patient’s spine through the HoloLens on their body. This allows doctors to explore surgical options with their patients without having to do any sort of exploratory surgery.

Doctor Using Hololens to see patients diagram overlaid onto patient

Image source: Novarad OpenSight

Ready to start a conversation about how Mixed Reality can help push your business forward? Contact us.

Published on 08.28.17