Despite extensive debate and publications in the management of blowout fracture (BOF), there are still considerable differences in the surgeons’ management of BOF due to a lack of reliable evidence-based studies. This article aimed to evaluate which BOF patients require surgical treatment due to functional and/or cosmetic deformities; evaluate which computed tomography (CT) scan findings predict these problems; and provide an algorithm in the management of BOF. Seventy-nine patients with BOF were treated conservatively and followed up prospectively regarding functional and cosmetic deformities for at least 1 year. The patients’ CT scans were analyzed and several measurements were performed. Patients’ symptoms and the clinical findings were correlated to the CT scan measurements. We found visible deformity in 37% of the patients, but only 10% chose to proceed to surgery due to cosmetic deformities. In patients with inferior BOF and a herniation < 1.0 mL, a visible deformity was found when the ratio between fracture and the fractured orbital wall areas was ≥42%, or the total area of the fracture was ≥ 2.3 cm2. In patients with inferior BOF and a herniation ≥ 1.0 mL, a visible deformity was found when the distance from the inferior orbital rim to the posterior edge of the fracture was ≥ 3.0 cm. In patients with inferomedial fracture, a visible deformity was found when the herniation was ≥ 0.9 mL. Diplopia improved significantly and remained in only 3% of the patients in nonoperated group. Hypoesthesia of the infraorbital nerve improved significantly, but 23% of the nonoperated and 50% of the operated patients still experienced loss of sensation at final control. In this prospective study, we found that not only herniated orbital volume but also other CT scan findings in BOF were crucial to predict late visible deformities. Based on these findings, we propose an algorithm for the prediction of late visible deformity with 83% accuracy. There are indications that diplopia without ocular motility disorder is due to edema and we recommend observation as long as the diplopia improves gradually.