Fluorescence in situ hybridization (FISH) continues to play an important role in clinical investigations. Laboratories may create their own cutoff, a percentage of positive nuclei to determine whether a specimen is positive or negative, to eliminate false positives that are created by signal overlap in most cases. In some cases, it is difficult to determine the cutoff value because of differences in both the area of nuclei and the number of signals. To address these problems, we established two mathematical models using probability theory. To verify these two models, normal disomy cells from healthy individuals were used to simulate cells with different numbers of signals by hybridization with different probes. We used an X/Y probe to obtain the average distance between two signals and the probability of signal overlap in different nuclei area. Frequencies of all signal patterns were scored and compared with theoretical frequencies, and models were assessed using a goodness of fit test. We used five BCR/ABL1-positive samples, 20 BCR/ABL1-negative samples and two samples with ambiguous results to verify the cutoff calibrated by these two models. The models were in agreement with experimental results. The dynamic cutoff can classify cases in routine analysis correctly, and it can also correct for influences from nuclei area and the number of signals in some ambiguous cases. The probability models can be used to assess the effect of signal overlap and calibrate the cutoff. V C 2015 International Society for Advancement of Cytometry