Cancer affects one in three people worldwide. Surgery remains the primary curative option for localized cancers, but good prognoses require complete removal of primary tumors and timely recognition of metastases. To expand surgical capabilities and enhance patient outcomes, we developed a six-channel color/near-infrared image sensor inspired by the mantis shrimp visual system that enabled near-infrared fluorescence image guidance during surgery. The mantis shrimp’s unique eye, which maximizes the number of photons contributing to and the amount of information contained in each glimpse of its surroundings, is recapitulated in our single-chip imaging system that integrates arrays of vertically stacked silicon photodetectors and pixelated spectral filters. To provide information about tumor location unavailable from a single instrument, we tuned three color channels to permit an intuitive perspective of the surgical procedure and three near-infrared channels to permit multifunctional imaging of optical probes highlighting cancerous tissue. In nude athymic mice bearing human prostate tumors, our image sensor enabled simultaneous detection of two tumor-targeted fluorophores, distinguishing diseased from healthy tissue in an estimated 92% of cases. It also permitted extraction of near-infrared structured illumination enabling the mapping of the three-dimensional topography of tumors and surgical sites to within 1.2-mm error. In the operating room, during surgical resection in 18 patients with breast cancer, our image sensor further enabled sentinel lymph node mapping using clinically approved near-infrared fluorophores. The flexibility and performance afforded by this simple and compact architecture highlights the benefits of biologically inspired sensors in image-guided surgery.

Recent advancements in nanofabrication technology has led to commercialization of single-chip polarization and color-polarization imaging sensors in the visible spectrum. Novel applications have arisen with the emergence of these sensors leading to questions about noise in the reconstructed polarization images. In this paper, we provide theoretical analysis for the input and output referred noise for the angle and degree of linear polarization information. We validated our theoretical model with experimental data collected from a division of focal plane polarization sensor. Our data indicates that the noise in the angle of polarization images depends on both incident light intensity and degree of linear polarization and is independent of the incident angle of polarization. However, noise in degree of linear polarization images depends on all three parameters: incident light intensity, angle and degree of linear polarization. This theoretical model can help guide the development of imaging setups to record optimal polarization information.

Water is an essential component of the Earth’s climate, but monitoring its properties using autonomous underwater sampling robots remains a significant challenge due to lack of underwater geolocalization capabilities. Current methods for underwater geolocalization rely on tethered systems with limited coverage or daytime imagery data in clear waters, leaving much of the underwater environment unexplored. Geolocalization in turbid waters or at night has been considered unfeasible due to absence of identifiable landmarks. In this paper, we present a novel method for underwater geolocalization using deep neural networks trained on $$\sim$$ ∼ 10 million polarization-sensitive images acquired globally, along with camera position sensor data. Our approach achieves longitudinal accuracy of $$\sim$$ ∼ 55 km ($$\sim$$ ∼ 1000 km) during daytime (nighttime) at depths up to $$\sim$$ ∼ 8 m, regardless of water turbidity. In clear waters, the transfer learning longitudinal accuracy is $$\sim$$ ∼ 255 km at 50 m depth. By leveraging optical data in conjunction with camera position information, our novel method facilitates underwater geolocalization and offers a valuable tool for untethered underwater navigation.

Tumor-targeted fluorescent probes in the near-infrared spectrum can provide invaluable information about the location and extent of primary and metastatic tumors during intraoperative procedures to ensure no residual tumors are...

No abstract