Multimodal light-sensing pixel arrays

Image sensors used in consumer cameras only detect intensity of light. Spectral, angular, and phase information of incident light is largely lost. For conventional applications like photography, the intensity information with three color bands (RGB) is sufficient. However, for advanced sensing applications, such as autonomous vehicles, biomedical imaging, and robotics, extracting more information from the incident light could help machines to make better decisions. It is expected that, in the future, more cameras will be built for machines than for people, which will be further accelerated by the rapid progress in machine learning and artificial intelligence. We envision that these applications will substantially benefit from multimodal measurement of light fields by advanced imaging sensors. In this Perspective, we discuss recent progress in image sensors with multimodal capabilities.