Depth ordering is instrumental for understanding the 3D geometry of an image. We as humans are surprisingly good at depth ordering even with abstract 2D line drawings. In this paper we propose a learning based framework for discrete depth ordering inference.Boundary and junction characteristics are important clues for this task, and we have developed new features based on these attributes. Although each feature individually can produce reasonable depth ordering results, they still have limitations, and we can achieve better performance by combining them. In practice, local depth ordering inferences can be contradictory. Therefore, we propose a Markov Random Field model with terms that are more global than previous work, and use graph optimization to encourage a globally consistent ordering. In addition, to produce better object segmentation for the task of depth ordering, we propose to explicitly enforce closed loops and long edges for the occlusion boundary detection.We collect a new depth-order dataset for this problem, including more than a thousand human-labeled images with different daily objects in various configurations. The proposed algorithm gives promising performance over conventional methods on both synthetic and real scenes.