Salient object detection can identify the most distinctive objects in a scene. In this study, a novel graph-based approach is proposed to detect a salient object via reliable boundary seeds and saliency refinement. A natural image is firstly mapped to a graph with superpixels as nodes. Saliency information is then diffused over the graph using seeds. For the reason that the boundary nodes may contain salient nodes, it is not appropriate to use all boundary nodes as the background seeds. Therefore, a boundary saliency measurement is proposed to obtain more accurate background seeds. After that, the information of background seeds is diffused by a two-stage scheme. A background-based map and a foreground-based map are generated based on the two-stage scheme. Furthermore, in order to enhance the detection accuracy, a refinement model is presented to fuse the information of background-based and foreground-based maps. Experiments on seven public datasets show the proposed algorithm out-performs the state-of-the-art salient object detection algorithms.