Background: Mammography is the preferred method for the diagnosis of breast cancer. However, this diagnostic technique fails to detect tumors of small sizes, and it does not work well for younger patients with high breast tissue density. Methods: This paper proposes a novel tool for the early detection of breast cancer, which is patient-specific, non-invasive, inexpensive, and has potential in terms of accuracy compared with existing techniques. The main principle of this method is based on the use of temperature contours from breast skin surfaces through thermography, and inverse thermal modeling based on Finite Element Analysis (FEA) and a Genetic Algorithm (GA)-based optimization tool to estimate the depths and sizes of tumors as well as patient/breast-specific tissue properties. Results: The study was conducted by using a 3D geometry of patients’ breasts and their temperature contours, which were clinically collected using a 3D scanner and a thermal imaging infrared (IR) camera. Conclusion: The results showed that the combination of 3D breast geometries, thermal images, and inverse thermal modeling is capable of estimating patient/breast-specific breast tissue and physiological properties such as gland and fat contents, tissue density, thermal conductivity, specific heat, and blood perfusion rate, based on a multilayer model consisting of gland and fat. Moreover, this tool was able to calculate the depth and size of the tumor, which was validated by the doctor’s diagnosis.