Deforming Reality: exploring the boundary between physical and virtual using deep learning
News
Catherine Taylor, CDE
There are currently no solutions at market that can rapidly generate a virtual reality 'prop' from a generic object, and then render it into an interactive virtual environment, outside of a studio. A portable solution such as this would enable creation of deployable immersive experiences where users could interact with virtual representations of physical objects in real time, opening up new possibilities for applications of virtual reality technologies in entertainment, but also in sports, health and engineering sectors. Catherine's project combines novel alogrithmic software for tracking deformable objects, interactive stereoscopic graphics for virtual reality, and an innovative configuration of existing hardware, to create the Marshmallow Laser Feast (MLF) DOVE system. The project objective is to create turn-key tools for repeatably developing unique immersive experiences and training environments. The DOVE system will enable MLF to create mixed reality experiences such as live productions, serialised apps & VR products/experiences to underpin significant business growth and new job creation opportunities.Written by Rebekah Hole and published on Tuesday 8th of December 2020
Centre for Digital Entertainment EngD student Catherine Taylor presented her research with Marshmallow Laser Feast at Beyond Conference.
Supervised by Darren Cosker, Catherine’s research explores creating immersive and tactile experiences in VR.
Abstract
Rather than using the traditional means of interaction, such as controllers, we transport physical objects into virtual worlds using a deep learning tracking solution. The behaviour of the physical object controls a corresponding virtual object, allowing novel tangible interactions. We design a custom neural network for the task and present results with several rigid and non-rigid objects. Our novel solution for interactions in VR combines physical and virtual elements and can be used to create immersive theatre pieces and performances.
