Milk quality depends on the physicochemical characteristics, hygienic standards, and nutritional quality; however, animal husbandry practices, unhygienic harvesting and processing, may affect its quality. A cross-sectional study was conducted between August 2010 and July 2011 to assess the hygiene of cow milk production environment, raw cow milk physicochemical characteristics, and microbial quality and estimate the prevalence of antimicrobial residues using standard methods in Pemba Island. A total of 98 raw cow milk samples from selected smallholder dairy farms were analyzed, and the judgement on the quality used the East African Standards. Generally, the milk production chain was done under the unhygienic condition, and dirty plastic containers were used for collection and storage of milk under room temperature. Some milk samples had abnormal colour (2.1%), abnormal smell (7.1%), and pH below normal (35.7%), clotted on alcohol test (9.2%), and had the specific gravity below normal (13.3%). All the milk samples had mineral contents within the recommended range. Milk samples with butterfat below normal were 29.6%, while 14.3% had total solids below recommended values. The mean total viable count (TVC) of milk container surfaces was 9.7 ± 10.5 log CFU/100 cm2, while total coliform count (TCC) was 7.8 ± 8.5 log CFU/100 cm2. Up to 55.1% of milk had TVC beyond the recommended levels. The milk mean TVC was 11.02 ± 11.6 log CFU/ml and TCC was 6.7 ± 7.3 log CFU/ml. Up to 26.5% of milk samples had the TCC beyond levels. Results on physicochemical characteristics and nutritional analysis show that the raw cow milk in Pemba Island is of inferior quality. Microbiological results of this study imply heavy contaminations of milk. Antimicrobial residues were detected in 83% of the samples and most of them were from Wete District. Unhygienic milk production chain accelerates microbial contaminations, and antimicrobial residues in milk are a big problem that needs urgent attention from the responsible authority.