Lateral flow assay (LFA) technology has recently received interest in the biochemical field since it is simple, low-cost, and rapid, while conventional laboratory test procedures are complicated, expensive, and time-consuming. In this paper, we propose a robust smartphone-based analyte detection method that estimates the amount of analyte on an LFA strip using a smartphone camera. The proposed method can maintain high estimation accuracy under various illumination conditions without additional devices, unlike conventional methods. The robustness and simplicity of the proposed method are enabled by novel image processing and machine learning techniques. For the performance analysis, we applied the proposed method to LFA strips where the target analyte is albumin protein of human serum. We use two sets of training LFA strips and one set of testing LFA strips. Here, each set consists of five strips having different quantities of albumin—10 femtograms, 100 femtograms, 1 picogram, 10 picograms, and 100 picograms. A linear regression analysis approximates the analyte quantity, and then machine learning classifier, support vector machine (SVM), which is trained by the regression results, classifies the analyte quantity on the LFA strip in an optimal way. Experimental results show that the proposed smartphone application can detect the quantity of albumin protein on a test LFA set with 98% accuracy, on average, in real time.