Fourth Workshop on Computer Vision for AR/VR

Fernando De la Torre (Facebook, CMU)

Talk: Human Understanding for HoloLens (Watch)

Abstract: Mixed reality devices such as HoloLens offer new and exciting holographic experiences and capabilities. Underpinning these experiences is a rich understanding of the user, powered by state of the art computer vision algorithms. In this talk we'll explore some of the challenges we faced in bringing those algorithms into HoloLens 2, as well as taking a glimpse at some of the exciting opportunities for computer vision to impact future devices and enable transformative experiences such remote presence.

Bio: Jamie Shotton is Partner Director of Science at Microsoft. He leads the Mixed Reality & AI Labs in Cambridge and Belgrade, where his team incubates transformative new technologies and experiences from early stage research to shipping product. He studied Computer Science at the University of Cambridge, where he remained for his PhD in computer vision and machine learning, before joining Microsoft in 2008. His research focuses at the intersection of computer vision, AI, machine learning, and graphics, with particular emphasis on systems that understand the motion, shape, and appearance of people in 3D. He has explored applications of this work for mixed reality, virtual presence, human-computer interaction, gaming, and healthcare. He has shipped foundational features in multiple products including body tracking for Kinect and the hand- and eye-tracking that enable HoloLens 2’s instinctual interaction model. He has received multiple Best Paper and Best Demo awards at top-tier academic conferences. His work on Kinect was awarded the Royal Academy of Engineering’s gold medal MacRobert Award in 2011, and he shares Microsoft’s Outstanding Technical Achievement Award for 2012 with the Kinect engineering team. In 2014 he received the PAMI Young Researcher Award, and in 2015 the MIT Technology Review Innovator Under 35 Award.

Talk: Augmenting Reality to Support People with Low Vision in Daily Tasks (Watch)

Abstract: How can advances in computer vision and augmented reality help people with visual impairments? In my research, I study the experiences of people with visual impairments and design applications to help them overcome challenges. I’ll present two augmented reality applications that my students and I designed for people with low vision, who have a visual impairment that falls short of blindness. The first application help the user find product in a grocery store and the second helps her navigate through the built-environment. Both applications show how computer vision and AR can be leveraged to augment the user’s perception and support fundamental visual tasks.

Bio: Shiri Azenkot is an Assistant Professor at the Jacobs Technion-Cornell Institute at Cornell Tech and in the Information Science field at Cornell University. She is also on the faculty at Technion – Israel Institute of Technology. She is broadly interested in human-computer interaction and accessibility. Professor Azenkot’s research focuses on enabling people with disabilities to have equal access to information via mobile and wearable devices. She received a Ph.D. in Computer Science from the University of Washington in 2014, where she was awarded the Graduate School Medal, an NSF Graduate Research Fellowship, and an AT&T Labs Graduate Fellowship. She also holds a B.A. in Computer Science from Pomona College.

Talk: Slow Glass (Watch)

Abstract: Wouldn’t it be fascinating to be in the same room as Abraham Lincoln, visit Thomas Edison in his laboratory, or step onto the streets of New York a hundred years ago? We explore this thought experiment, by tracing ideas from science fiction through newly available data sources that may facilitate this goal.

Bio: Steve Seitz is Robert E. Dinning Professor in the Allen School at the University of Washington. He is also a Director on Google's Daydream team, where he leads teleportation efforts including Google Jump and Cardboard Camera. He received his B.A. in computer science and mathematics at the University of California, Berkeley in 1991 and his Ph.D. in computer sciences at the University of Wisconsin in 1997. Following his doctoral work, he did a postdoc at Microsoft Research, and then a couple years as Assistant Professor in the Robotics Institute at Carnegie Mellon University. He joined the faculty at the University of Washington in July 2000. His co-authored papers have won the David Marr Prize (twice) at ICCV, and the CVPR 2015 best paper award. He received an NSF Career Award, and ONR Young Investigator Award, an Alfred P. Sloan Fellowship, and is an IEEE Fellow and an ACM Fellow. His work on Photo Tourism (joint with Noah Snavely and Rick Szeliski) formed the basis of Microsoft's Photosynth technology. Professor Seitz is interested in problems in 3D computer vision and computer graphics, and their application to virtual and augmented reality.

Talk: Computer Vision For Remote AR (Watch)

Abstract: Remote AR unlocks a new set of experiences and use cases for customers. One use case in particular is remote collaboration where an expert in a different location can help an onsite customer with a task in the customer environment. Streem focuses precisely on this, and enable improved customer experiences with AR-supported video tools. This talk will dive into how Remote AR is used to unlock a new type of customer engagement, and the Computer Vision research problems that support these experiences.

Bio: Flora is the Head of CV/AR at Streem. She specialises in AI applied to Computer Graphics and Vision problems faced in AR/VR. Her team at Streem is making the mobile phone's camera more intelligent, by building AI agents that can understand images/videos and augment them with relevant interactive virtual content. She joined Streem, after it acquires her startup Selerio, which was spun out of her PhD work at Cambridge University. At Cambridge, Flora research focused on 3D shape retrieval using different query types such as 3D models, images/sketches and range scans. This work was awarded the 2013 Google Doctoral Fellowship in Computer Graphics and published in various top-tier venues, including ICCV and SIGGRAPH Asia. She served on several international program committees such as ICLR and Eurographics. Notably she was Paper Chair of the 2019 Black in AI workshop, co-located with NeurIPS. She was recently named among the Rework Top 30 UK Women in AI and appeared on Computer Weekly Most Influential women in UK Tech longlist.

Moderators: Andrew Rabinovich (Magic Leap), Serge Belongie (Cornell)

Panelists: Jamie Shotton, Shiri Azenkot, Steve Seitz, Flora Ponjou Tasse

(Watch)

Playback of video submissions, see papers for full submissions. Moderator: Harald Haraldsson (Cornell Tech)

Watch all videos here (YouTube playlist)

The Virtual Poster Sessions will take place on Discord, Friday June 19, 2020. Moderator: Harald Haraldsson (Cornell Tech)

See here for details.

Talk: Computer Vision for the Future of Social Presence (Watch)

Bio: Michael Abrash is the Chief Scientist of Facebook Reality Labs, a research laboratory that brings together a world-class R&D team of scientists, developers and engineers to build the future of connection within virtual and augmented reality. He was graphics lead for the first two versions of Windows NT, teamed with John Carmack on Quake, worked on the first two versions of Microsoft’s Xbox, and helped develop virtual reality at Valve. He is also the author of several books, including Michael Abrash’s Programming Black Book.

Talk: GHUM, Interactions, and Active Human Sensing

Bio: Cristian Sminchisescu is a Research Scientist leading a team at Google, and a Professor at Lund University. He has obtained a doctorate in computer science and applied mathematics with focus on imaging, vision and robotics at INRIA, under an Eiffel excellence fellowship of the French Ministry of Foreign Affairs, and has done postdoctoral research in the Artificial intelligence Laboratory at the University of Toronto. He has held a Professor equivalent title at the Romanian Academy and a Professor rank, status appointment at Toronto, and has advised research at both institutions. During 2004-07, he was a faculty member at the Toyota Technological Institute at the University of Chicago, and later on the Faculty of the Institute for Numerical Simulation in the Mathematics Department at Bonn University. Cristian Sminchisescu regularly serves as an Area Chair for computer vision and machine learning conferences (CVPR, ECCV, ICCV, AAAI, NeurIPS), as a Program Chair for ECCV 2018, and an Associate Editor of IEEE Transactions for Pattern Analysis and Machine Intelligence (PAMI) and the International Journal of Computer Vision (IJCV). Over time, his work has been funded by the US National Science Foundation, the Romanian Science Foundation, the German Science Foundation, the Swedish Science Foundation, the European Commission under a Marie Curie Excellence Grant, and the European Research Council under an ERC Consolidator Grant. Cristian Sminchisescu's research interests are in the area of computer vision (3d human sensing, reconstruction and recognition) and machine learning (optimization and sampling algorithms, kernel methods and deep learning). The visual recognition methodology developed in his group was a winner of the PASCAL VOC object segmentation and labeling challenge during 2009-12, as well as the Reconstruction Meets Recognition Challenge (RMRC) 2013-14. His work on deep learning of graph matching has received the best paper award honorable mention at CVPR 2018.

Talk: Compelling physical interactions in Mixed Reality (Watch)

Abstract: The problem of simulating virtual entities within a dynamically changing real world remains a significant open challenge. We are still far from being able to emulate the rich, physical experience of interacting with real objects, environments and people. Mixed Reality is therefore still a developing field, with most research focussed on developing computer vision and AI technologies for capturing and analysing video of the real world and augmenting it with virtual content. There is now a strong impetus and opportunity to abstract beyond the technological considerations and to consider the convergence of human perception of causality, computer vision, motion capture, computer animation, artificial intelligence and other related fields to deliver compelling physical interactions in Mixed Reality. In this talk I will present some ideas and recent research towards this goal.

Bio: Carol O'Sullivan is the Professor of Visual Computing in Trinity College Dublin. From 2013-2016 she was a Senior Research Scientist at Disney Research in Los Angeles, and spent a sabbatical year as Visiting Professor in Seoul National University from 2012-2013. Her research interests include graphics, AR/VR, perception, Computer Animation, Crowd and Human simulation. She has been a member of many editorial boards and international program committees (including ACM SIGGRAPH and Eurographics), and has served as Editor in Chief for the ACM Transactions on Applied Perception (TAP) from 2006-2012. She has been program or general chair for several conferences, including the annual Eurographics conference, the ACM Symposium on Computer Animation, and the Courses Chair for ACM SIGGRAPH Asia 2018. Prior to her PhD studies, she spent several years in industry working in Software Development. She was elected a fellow of Trinity College in 2003 and of Eurographics in 2007.

Talk: How to Build a Holodeck (Watch)

Abstract: In the age of COVID-19 it is more clear than ever that there is a compelling need for better remote collaboration. Fortunately a number of technologies are starting to converge which will allow us to take such collaborations to a whole new level. Imagine that when you join an on-line meeting you are present with your entire body, and that you can see and hear other people as though you are all in the same room.

There are many challenges to realizing this vision properly. The NYU Future Reality Lab and its collaborators are working on many of them. This talk will give an overview of many of the key areas of research, including how to guarantee universal accessibility, user privacy and rights management, low latency networking, design and construction of shared virtual worlds, correct rendering of spatial audio, biometric sensing, and a radical rethinking of user interface design.

Bio: Ken Perlin, a professor in the Department of Computer Science at New York University, directs the Future Reality Lab, and is a participating faculty member at NYU MAGNET. His research interests include future reality, computer graphics and animation, user interfaces and education. He is chief scientist at Parallux, Tactonic Technologies and Autotoon. He is an advisor for High Fidelity and a Fellow of the National Academy of Inventors. He received an Academy Award for Technical Achievement from the Academy of Motion Picture Arts and Sciences for his noise and turbulence procedural texturing techniques, which are widely used in feature films and television, as well as membership in the ACM/SIGGRAPH Academy, the 2020 New York Visual Effects Society Empire Award the 2008 ACM/SIGGRAPH Computer Graphics Achievement Award, the TrapCode award for achievement in computer graphics research, the NYC Mayor's award for excellence in Science and Technology and the Sokol award for outstanding Science faculty at NYU, and a Presidential Young Investigator Award from the National Science Foundation. He serves on the Advisory Board for the Centre for Digital Media at GNWC. Previously he served on the program committee of the AAAS, was external examiner for the Interactive Digital Media program at Trinity College, general chair of the UIST2010 conference, directed the NYU Center for Advanced Technology and Games for Learning Institute, and has been a featured artist at the Whitney Museum of American Art. He received his Ph.D. in Computer Science from NYU, and a B.A. in theoretical mathematics from Harvard. Before working at NYU he was Head of Software Development at R/GREENBERG Associates in New York, NY. Prior to that he was the System Architect for computer generated animation at MAGI, where he worked on TRON.

Moderators: Sofien Bouaziz (Google), Matt Uyttendaele (Facebook)

Panelists: Michael Abrash, Cristian Sminchisescu, Carol O’Sullivan, Ken Perlin

(Watch)

Fernando De la Torre (Facebook, CMU)