Infrared (IR) Thermography is currently a supplementary technique for breast cancer diagnosis. There have been studies using IR thermography and numerical modeling in an attempt to detect tumor inside the breast. Most of these studies focused on either the “forward modeling” problem or only used idealized or population-averaged patients’ data, whereas identification of the tumor inside the breast based on the thermal pattern is an “inverse modeling” problem dependent on personalized information of the patient. Inverse modeling is based on the idea that the surface thermal pattern of the breast can be used to determine the tumor features based on physical and physiological principles. The current study aims to develop a well-validated inverse thermal modeling framework that could be used to determine the depth and size of tumor inside a breast based on personalized patients’ breast data, such as thermogram and 3D geometry using efficient design optimization techniques and Finite Element Modeling (FEM) to support the process. The numerical modeling was validated by the experiments, conducted using artificial breasts. Results show that although DIRECT Optimization method can be employed to find the depth and size of the tumor with good accuracy, the technique can be very time consuming. On the other hand, Response Surface Optimization method is also able to find the depth and size of the tumor with less accuracy but faster when compared with DIRECT Optimization. The last method tested, Nelder-Mead method, failed to detect the tumor. The study concludes that Response Surface Optimization method should be used first, and after the range of parameters are found, the DIRECT optimization method can be applied for more accurate results. However the GA method was found to be the only viable and efficient design optimization method for reverse modeling when blood perfusion was adopted in the breast model and many parameters were searched for with patient specific data input for breast tumor diagnosis.