Pathogenic bacteria are the leading causes of food-borne and water-borne infections, and one of the most serious public threats. Traditional bacterial detection techniques, including plate culture, polymerase chain reaction, and enzyme-linked immunosorbent assay are time-consuming, while hindering precise therapy initiation. Thus, rapid detection of bacteria is of vital clinical importance in reducing the misuse of antibiotics. Among the most recently developed methods, the label-free optical approach is one of the most promising methods that is able to address this challenge due to its rapidity, simplicity, and relatively low-cost. This paper reviews optical methods such as surface-enhanced Raman scattering spectroscopy, surface plasmon resonance, and dark-field microscopic imaging techniques for the rapid detection of pathogenic bacteria in a label-free manner. The advantages and disadvantages of these label-free technologies for bacterial detection are summarized in order to promote their application for rapid bacterial detection in source-limited environments and for drug resistance assessments.