Abstract. The implementation of lesion segmentation for breast ultrasound image relies on several diagnostic rules on intensity, texture, etc. In this paper, we propose a novel algorithm to achieve a comprehensive decision upon these rules by incorporating image over-segmentation and lesion detection in a pairwise CRF model, rather than a term-byterm translation. Multiple detection hypotheses are used to propagate object-level cues to segments and a unified classifier is trained based on the concatenated features. The experimental results show that our algorithm can avoid the drawbacks of separate detection or bottom-up segmentation, and can deal with very complicated cases.