Breast cancer screening provides sensitive tumor identification, but low specificity implies that a vast majority of biopsies are not ultimately diagnosed as cancer. Automated techniques to evaluate biopsies can prevent errors, reduce pathologist workload and provide objective analysis. Fourier transform infrared (FT-IR) spectroscopic imaging provides both molecular signatures and spatial information that may be applicable for pathology. Here, we utilize both the spectral and spatial information to develop a combined classifier that provides rapid tissue assessment. First, we evaluated the potential of IR imaging to provide a diagnosis using spectral data alone. While highly accurate histologic [epithelium, stroma] recognition could be achieved, the same was not possible for disease [cancer, no-cancer] due to the diversity of spectral signals. Hence, we employed spatial data, developing and evaluating increasingly complex models, to detect cancers. Sub-mm tumors could be very confidently predicted as indicated by the quantitative measurement of accuracy via receiver operating characteristic (ROC) curve analyses. The developed protocol was validated with a small set and statistical performance used to develop a model that predicts study design for a large scale, definitive validation. The results of evaluation on different instruments, at higher noise levels, under a coarser spectral resolution and two sampling modes [transmission and transflection], indicate that the protocol is highly accurate under a variety of conditions. The study paves the way to validating IR imaging for rapid breast tumor detection, its statistical validation and potential directions for optimization of the speed and sampling for clinical deployment.