(In person)

Title: Object Digitization, Manipulation and Rendering for Immersive Experiences

Abstract: The digitization, manipulation and rendering of objects requires a great deal of skill and time, substantially limiting the available immersive experiences. In our work, we build tools to simplify this process and make it intuitive. To move the needle here, there are two key questions that need to be considered: 1) how to digitize and insert objects into existing scenes and 2) how does one interactively manipulate objects within volumetric environments? We’d like to solve both in 3D. To answer the first question, we show an efficient method to neural object capture and rendering from online images. Given images with the same object in different environments and lighting conditions, our method estimates material properties making it seamless to insert neural objects into scenes. To answer the second question we present a new volumetric representation, dubbed “playable environments,” which allows one to play objects inside scenes with intuitive controls, camera and style changes, making scene manipulation akin to playing a game.

Bio: Sergey Tulyakov is a Principal Research Scientist heading the Creative Vision team at Snap Inc. His work focuses on creating methods for manipulating the world via computer vision and machine learning. This includes human and object understanding, photorealistic manipulation and animation, video synthesis, prediction and retargeting. He pioneered the unsupervised image animation domain with MonkeyNet and First Order Motion Model that sparked a number of startups in the domain. His work on Interactive Video Stylization received the Best in Show Award at SIGGRAPH Real-Time Live! 2020. He has published 30+ top conference papers, journals and patents resulting in multiple innovative products, including Snapchat Pet Tracking, OurBaby, Real-time Neural Lenses (gender swap, baby face, aging lens, face animation) and many others. Before joining Snap Inc., Sergey was with Carnegie Mellon University, Microsoft, NVIDIA. He holds a PhD degree from the University of Trento, Italy.