The Testing Tool for Magic Leap
Smart environment simulation for on-device XR testing | Internship project by Yinghua at Unity Labs | Built with Unity, Photoshop, Sketchup, Blender, etc.
An on-device tool for creators to test their apps in other environments by augmenting the current physical environments with simulated environmental objects.
Let’s say you created a mixed reality app that can adapt to any space, bedroom, living room, office, etc. After building it, how are you going to test its environmental adaptability?
The direct way is to find multiple physical environments with a variety of sizes, layouts, and furniture and test with the device in each. However, the process of finding can be laborious and sometimes costly.
The indirect way is to preview your app with simulators on desktops. Project MARS by Unity Labs revealed its in-editor room simulator at Unite Berlin. Magic Leap has its own independent simulator called Magic Leap Remote. HoloLens has an emulator, but it doesn’t allow you to load virtual rooms, so does Unity’s in-editor Holographic Emulation feature launched back in 2016. Such simulators can help speed up the iteration process significantly, but on-device testing is eventually needed.
It is valid to assume that there is a demand for an easier on-device testing process, which doesn’t require creators to find different physical environments. I made the assumption initially based on my own experience with creating HoloLens apps.
To validate the need for an on-device testing tool, I first did a case study on 19 HoloLens apps and then reached out to about 40 AR industry leads via LinkedIn and emails and successfully interviewed 16 of them via video chat and Google Form.
Among the 19 apps, 8 apps achieved an exceptionally high rating. They share one thing in common - their mixed reality contents are environmentally adaptive, which build convincing and engaging experiences for users.
One of the main findings of the interviews and surveys are matching with my assumption. “Quickly testing and adjusting” is one of the two most needed features for on-device authoring. The other one is capturing world data.
After validating the assumption, I brainstormed 8 ideal features for an on-device testing tool and chose to design and implement one of the features: letting creators test their apps in other environments by augmenting the current physical space with simulated environmental objects.
The keys are to add or remove available surfaces and to increase or reduce surface area/heights. I brainstormed 3 ways to make it happen and analyzed their pros and cons.
The first idea is to give users a list of furnitures to place. The inspiration is AR furniture shopping apps. However, the shortcoming is that you have to re-scale furniture’s size to make it fit into various spaces, which might end up with an unrealistic scale.
The second idea is to let users virtually change the shape of their real-world furnitures by dragging the edge of any furnitures and extend it with virtual parts. The inspiration for this idea is Transformer. However, it is very technically challenging to keep the virtual part’s materials, textures and colors the same with the original furniture.
The third idea is to let users add or cover surfaces using blocks. The inspiration is the Australian animal Wombat, who produces cubic poops and use them to mark their territory. Although I like the flexibility of cubic building blocks, the idea itself might not fit for adults.
With the three ideas’ pros and cons in mind, the final design is to let creators use volume cubes to define their chosen empty space for virtual environment projects. Based on the size the volume cube they place, different environment object options will show up. This design combined all of the merits of the three ideas and avoided all of the shortcomings.
After the brainstorm, I designed UI using illustrator, present the mixed reality interaction using Sketchup, and organized the user journey into a storyboard using Realtimeboard. I learned this new way to present mixed reality app design from Greg, one of the talented designers at Unity Labs. I was always frustrated with designing and presenting mixed reality app design using 2D tools, such as Sketch or illustrator. With Sketchup, I finally got to present the app design in a 3D way, which is extremely helpful. I invited our team and the design team at the lab to critique the design. I received a lot of useful comments and suggestions on the Realtimeboard.
How it works
Creators place virtual volume cubes in their space, and based on the cube’s size, different environmental object options show up. You can add a simulated table in your room by placing a big volume cube on the floor. Or you can cover part of your real-world table by placing a small volume cube on it which can turn to a simulated table plant. To remove a virtual object, gaze at the object to select and pull the trigger.
User Journey & Persona
Mojia, an art director, would like to test their XR app for Magic Leap. Engineers import the testing tool as an asset package into their unity project and deploy the app on Magic Leap. Mojia wear Magic Leap and launch their app. She first tests their app directly in the current physical environment. Then she activates the testing tool to add simulated environment objects to create a new environment. Then she re-play their app in the newly mixed environment and make adjustments directly on-device, which will be recorded back to their Unity project.
With all of the feedback and support, I dived into the implementation of the testing tool for Magic Leap. I created a customized volume cube using Blender and Photoshop, disassemble some Magic Kit apps for parts that I need, and customized the code on top of them.
After the initial prototyping, I tested it around the lab and iterated the tool with fixed shaders, pivot points, etc.
The Testing Tool 1.0
The Testing Tool 2.0
MORE PROJECTS BY YINGHUA
XR Fitness App - YOGAR - Gamified yoga with HoloLens
XR Social App - Viewing Party - Augmented TV-viewing with HoloLens
Extension UI - FlipWord - Learn a language while browsing
3D Modeling - 3D animation and industrial design