The Testing Tool for Magic Leap

Smart environment simulation for on-device XR testing        Internship project by Yinghua at Unity Labs, 2018        Built with Unity, Photoshop, Sketchup, Blender, etc.

An on-device tool for creators to test their apps in other environments by augmenting the current physical environments with simulated environmental objects.


Let’s say you created a mixed reality app that can adapt to any space, bedroom, living room, office, etc. After building it, how are you going to test its environmental adaptability?

Screen Shot 2018-09-17 at 10.13.10 AM.png

Current Solutions

  • The direct way is to find multiple physical environments with a variety of sizes, layouts, and furniture and test with the device in each. However, the process of finding can be laborious and sometimes costly.

  • The indirect way is to preview your app with simulators on desktops. Project MARS by Unity Labs revealed its in-editor room simulator at Unite Berlin. Magic Leap has its own independent simulator called Magic Leap Remote. HoloLens has an emulator, but it doesn’t allow you to load virtual rooms, so does Unity’s in-editor Holographic Emulation feature launched back in 2016. Such simulators can help speed up the iteration process significantly, but on-device testing is eventually needed.


It is valid to assume that there is a demand for an easier on-device testing process, which doesn’t require creators to find different physical environments. I made the assumption initially based on my own experience with creating HoloLens apps.


Screen Shot 2018-09-17 at 10.05.27 AM.png

To validate the need for an on-device testing tool, I first did a case study on 19 HoloLens apps and then reached out to about 40 AR industry leads via LinkedIn and emails and successfully interviewed 16 of them via video chat and Google Form.

Key Findings

Case Study

Among the 19 apps, 8 apps achieved an exceptionally high rating. They share one thing in common - their mixed reality contents are environmentally adaptive, which build convincing and engaging experiences for users.


One of the main findings of the interviews and surveys are matching with my assumption. “Quickly testing and adjusting” is one of the two most needed features for on-device authoring. The other one is capturing world data.

Screen Shot 2018-09-17 at 9.51.40 AM.png

Project Proposal

After validating the assumption, I brainstormed 8 ideal features for an on-device testing tool and chose to design and implement one of the features: letting creators test their apps in other environments by augmenting the current physical space with simulated environmental objects.

Design Inspiration

The keys are to add or remove available surfaces and to increase or reduce surface area/heights. I brainstormed 3 ways to make it happen and analyzed their pros and cons.

Screen Shot 2018-09-17 at 11.16.47 AM.png
  1. The first idea is to give users a list of furnitures to place. The inspiration is AR furniture shopping apps. However, the shortcoming is that you have to re-scale furniture’s size to make it fit into various spaces, which might end up with an unrealistic scale.

  2. The second idea is to let users virtually change the shape of their real-world furnitures by dragging the edge of any furnitures and extend it with virtual parts. The inspiration for this idea is Transformer. However, it is very technically challenging to keep the virtual part’s materials, textures and colors the same with the original furniture.

  3. The third idea is to let users add or cover surfaces using blocks. The inspiration is the Australian animal Wombat, who produces cubic poops and use them to mark their territory. Although I like the flexibility of cubic building blocks, the idea itself might not fit for adults.

With the three ideas’ pros and cons in mind, the final design is to let creators use volume cubes to define their chosen empty space for virtual environment projects. Based on the size the volume cube they place, different environment object options will show up. This design combined all of the merits of the three ideas and avoided all of the shortcomings.

UI Design

After the brainstorm, I designed UI using illustrator, present the mixed reality interaction using Sketchup, and organized the user journey into a storyboard using Realtimeboard. I learned this new way to present mixed reality app design from Greg, one of the talented designers at Unity Labs. I was always frustrated with designing and presenting mixed reality app design using 2D tools, such as Sketch or illustrator. With Sketchup, I finally got to present the app design in a 3D way, which is extremely helpful. I invited our team and the design team at the lab to critique the design. I received a lot of useful comments and suggestions on the Realtimeboard.

Screen Shot 2018-09-17 at 11.17.14 AM.png

How it works

Creators place virtual volume cubes in their space, and based on the cube’s size, different environmental object options show up. You can add a simulated table in your room by placing a big volume cube on the floor. Or you can cover part of your real-world table by placing a small volume cube on it which can turn to a simulated table plant. To remove a virtual object, gaze at the object to select and pull the trigger.

User Journey & Persona

Mojia, an art director, would like to test their XR app for Magic Leap. Engineers import the testing tool as an asset package into their unity project and deploy the app on Magic Leap. Mojia wear Magic Leap and launch their app. She first tests their app directly in the current physical environment. Then she activates the testing tool to add simulated environment objects to create a new environment. Then she re-play their app in the newly mixed environment and make adjustments directly on-device, which will be recorded back to their Unity project.

Screen Shot 2018-09-17 at 10.17.44 AM.png


With all of the feedback and support, I dived into the implementation of the testing tool for Magic Leap. I created a customized volume cube using Blender and Photoshop, disassemble some Magic Kit apps for parts that I need, and customized the code on top of them.

Screen Shot 2018-09-17 at 11.17.20 AM.png


After the initial prototyping, I tested it around the lab and iterated the tool with fixed shaders, pivot points, etc.

The Testing Tool 1.0

The Testing Tool 2.0

The Testing Tool 1.0