In a harbor environment threats like explosives-packed rubber boats, mine-carrying swimmers and divers must be detected in an early stage. This paper describes the integration and use of a heterogeneous multiple camera system with panoramic observation capabilities for detecting these small vessels in the Den Helder New Harbor in the Netherlands. Results of a series of experiments with different targets are presented. An outlook to a future sensor package containing panoramic vision is discussed. We also investigated several aspects of the use of electro-optical systems. As for classification, this paper concentrates on discriminating classes of small vessels with different electro-optical systems (visual and infrared) as part of the larger process involving an operator. It addresses both selection of features (based on shape and texture) and ways of using these in a system to assess threats. Results are presented on data recorded in coastal and harbor environments for several small targets.