Reflecting for fresh perspectives on the world
Researchers from MIT and Rice University have developed a computer vision technique that uses reflections to turn glossy objects into “cameras,” enabling users to see the world as if they were looking through the “lenses” of everyday objects like a ceramic coffee mug or a metallic paperweight. The method uses reflections to capture images of the world, which is especially useful in autonomous vehicles. For instance, it could enable a self-driving car to use reflections from objects it passes, like lamp posts or buildings, to see around a parked truck. The technique, known as ORCa (Objects as Radiance-Field Cameras), works by taking pictures of an object from many vantage points, capturing multiple reflections on the glossy object. ORCa uses machine learning to convert the surface of the object into a virtual sensor that captures light and reflections that strike each virtual pixel on the object’s surface. Finally, the system uses virtual pixels on the object’s surface to model the 3D environment from the point of view of the object.
Overcoming Challenges
ORCa overcomes the challenges of using reflections by capturing multiview reflections, which the system uses to estimate depth between the glossy object and other objects in the scene, in addition to estimating the shape of the glossy object. ORCa models the scene as a 5D radiance field, which captures additional information about the intensity and direction of light rays that emanate from and strike each point in the scene. The additional information contained in this 5D radiance field also helps ORCa accurately estimate depth.
The researchers evaluated their technique by comparing it with other methods that model reflections, which is a slightly different task than ORCa performs. Their method performed well at separating out the true color of an object from the reflections, and it outperformed the baselines by extracting more accurate object geometry and textures. They compared the system’s depth estimations with simulated ground truth data on the actual distance between objects in the scene and found ORCa’s predictions to be reliable.
Future Applications
The researchers want to apply this technique to drone imaging. ORCa could use faint reflections from objects a drone flies over to reconstruct a scene from the ground. They also want to enhance ORCa so it can utilise other cues, such as shadows, to reconstruct hidden information, or combine reflections from two objects to image new parts of a scene. The research was supported, in part, by the Intelligence Advanced Research Projects Activity and the National Science Foundation.
AI NewsTime