Colorimetric assays typically offer a rapid and convenient method to assess analytes that span healthcare monitoring to water quality testing. However, such tests can only provide qualitative results when employed in resource-limited settings or require bulky and expensive equipment such as lab spectrophotometers to allow quantitative measurements. In this paper, we report on the use of a handheld colorimeter to quantitatively determine the concentration of analytes in a manner that is independent of ambient lighting or initial sample color. The method combines the response of the sensor with first-principles modeling that better describes the nature of the assay compared to linear-in-parameters regression modeling that is typically performed in other studies. This method was successfully demonstrated using a number of colorimetric assays: (1) determination of solution pH using a universal indicator, (2) quantification of the DNase presence using a DNA-gold nanoparticle assay, and (3) quantification of the concentration of the antibiotic tetracycline using a cell-based assay.