Microscope examination of Gram stained clinical specimens is used for aiding the diagnosis of patients with infectious diseases. In high volume pathology laboratories, this manual microscopy examination is considered time consuming and labour intensive. Unfortunately, despite the great benefits offered from the application of Computer Aided Diagnosis (CAD) systems, to our knowledge, the highest automation stage for Gram stained slide analysis is only at the preanalytical process. This paper takes the first steps towards the application of computer vision to direct smear, Gram stained images. To that end, we present a novel Gram stain image dataset. In addition, we also propose a multiple covariance approach for leukocyte and epithelial cell detection in Gram stain images. Each covariance matrix represents a particular image region characterising the cell's deformed structure. As covariance matrices form points on an Symmetric Positive Definite (SPD) manifold, the traditional Euclidean-based analysis cannot be used. As such, we first map the manifold points into the Reproducing Kernel Hilbert Space (RKHS). The analysis is done via a novel kernel similarity function that allows comparison between sets of covariance matrices. The proposed approach is contrasted, in the proposed dataset, with two recent state of the art methods in pedestrian detection: Histogram Of Gradient (HOG) and the traditional single covariance matrix approach. We found that the proposed approach outperformed both of these methods.