Experiment using phones to interact with 3D models across monitors, projectors, and headsets.
Context Switches Bad, Flow Good
The user experience of making 3D models on the desktop and experiencing them in VR is filled with friction. This experiment in 2014 to tackle the practical problems of using VR in a 3D design process.
For VR to help people create and analyze 3D models, we need to minimize the context switch cost from our primary computer interface, the desktop, to a VR experience of our 3D work. Focus and flow are key to creative work. Tightening the design/feedback loop encourages design exploration, improves productivity, and keeps creators focused on the right problems.
Seamless Desktop to VR Transition
We need a seamless transition from making a change in our 3D modeling program to interacting with the result on a headset or projection system. Ideally we tweek our 3D model on the desktop, then simply look left at our 1 to 1 scale projection system or bring a strapless headset up to our eyes.
This experiment puts an app on mobile phones so we can quickly start interacting with the projection system or review the 3D model with Google Cardboard.
Seamless 3D Model Iteration
The desktop content creation application, be it Maya, Revit or Soldworks, is the single source of truth. Moving 3D models from the desktop to the VR system is painful, but there are supported workflows. Support for tweaking the 3D data in VR, and bringing it back into the authoring application is practically non-existent. So the VR system must emulate the keyboard/mouse and allow for editing of 3D model in the original authoring application. Folks can then load the tweaked 3D model in VR without putting down the VR interface.
This experiment speeds up design/review loops by making the mobile phone a remote control for the desktop’s keyboard and mouse. So designers can make 3D model adjustments without leaving VR and incurring the desktop/VR context switch cost.
Display and Tracking System
The monitors and projectors act like windows on the virtual world. Arrays of screens, or using big projections can give a good sense of immersion without the friction of putting on a headset. Nice that you can collaborate face to face with someone too. The size and positions of the displays are calibrated with the mobile phone.
To interact with the 3D data in VR, the system has an app for mobile phones. The phone acts as a 6DOF virtual reality controller with the augmented reality tracking library Vuforia.
Phones or tablets can act as mobile windows on the virtual world. The positions of phones are coordinated with the computer screens. Folks can quickly experience the virtual world in real scale 3D by activating Google Cardboard on a phone.