Felix Heide is working to improve the imaging and computer vision capabilities of sensors for autonomous vehicles. Panels show a street scene in heavy snow using a standard vehicle imaging system (left), and the output of ZeroScatter (right), a technique developed by Heide’s team that improves image quality and reveals objects at long distances, such as the house at the end of the road. Images by Zheng Shi et al.

Since arriving at Princeton last year, Felix Heide has unveiled a flock of new cameras that push the edge of the visual world.

Some allow cars to peer through dense fog and heavy snow, while others glance around corners for hidden objects or pedestrians. Another is the size of a grain of salt.

Heide has accomplished these feats by closely integrating hardware and software, aiming to optimize cameras for specific purposes. His group’s research in computational imaging could advance real-time sensing and object detection capabilities for personal devices, medical imaging, robots, and autonomous vehicles.

“We are interested in thinking about codesign of the optics and the algorithms together,” said Heide, an assistant professor of computer science. “We want to build domain-specific camera systems: the best camera for object detection, or low-light imaging, or scene understanding, or medical imaging.”

portrait photo
Photo of Felix Heide by Sameer A. Khan/FotoBuddy

One of the imaging systems that Heide focuses on is vision technology for self-driving cars. His group has developed an automated system that uses radar to allow cars to peer around corners and spot oncoming traffic and pedestrians. And with a new grant from Princeton’s Metropolis Project, they are working on systems for navigation in adverse weather.

Existing systems for driverless cars rely heavily on lidar, which uses laser beams to detect objects in the environment. Lidar and other current imaging systems perform poorly in snow, fog, and rain, and at distances beyond about 1,000 feet (300 meters), which is essential for the safety of trucking. This is a major hurdle to their wide adoption, especially for transportation into cities from outlying areas, which will have less support infrastructure for driverless vehicles.

Using a test vehicle from Mercedes-Benz, a partner on the project, Heide’s team is collecting data from sensors during varied weather conditions in Princeton and New York City, and creating physical simulations based on this data. Using machine learning and optimization methods, they will develop systems enabling suites of sensors to dynamically adapt their activities based on changing conditions.