Autofluorescence properties of tissues have been widely used to diagnose various types of malignancies. In this study, we measured the autofluorescence properties of H-ras transfected murine fibroblasts and the counterpart control cells. The pair of cells is genetically identical except for the transfected H-ras gene. We applied Monte Carlo simulations to evaluate the relative contributions of Rayleigh and Mie scattering effects towards fluorescence in an in vitro model system of normal and H-ras transfected fibroblasts. The experimental results showed that fluorescence emission intensity was higher for normal cells than the malignant counterpart cells by about 30%. In normal cells, linearity in emission intensity was observed for cell densities of up to 1.0 x 10(6) cells/ml whereas for transformed cells it was up to 1.4 x 10(6) cells/ml. Nuclear volume changes give good account for the differences in the intrinsic fluorescence between normal and malignant cells. The Monte Carlo (MC) code, newly developed for this study, explains both predominant experimental features: the large fluorescence intensity differences between the transfected and the corresponding control cells as well as the phenomena of the red shift in the excitation spectra as a function of cell density. The contribution of Rayleigh scattering was found to be predominant compared to Mie scattering.