Halide perovskites have shown promise to advance the field of light detection in next-generation photodetectors, offering performance and functionality beyond what is currently possible with traditional inorganic semiconductors. Despite a relatively high density of defects in perovskite thin films, long carrier diffusion lengths and lifetimes suggest that many defects are benign. However, perovskite photodetectors show detection behavior that varies with time, creating inconsistent device performance and difficulties in accurate characterization. Here, we link the changing behavior to mobile defects that migrate through perovskites, leading to detector currents that drift on the time scale of seconds. These effects not only complicate reproducible device performance but also introduce characterization challenges. We demonstrate that such transient phenomena generate measurement artifacts that mean the value of specific detectivity measured can vary by up to 2 orders of magnitude even in the same device. The presence of defects can lead to photoconductive gain in photodetectors, and we show batch-to-batch processing variations in perovskite devices gives varying degrees of charge carrier injection and photocurrent amplification under low light intensities. We utilize the passivating effect of aging to reduce the impact of defects, minimizing current drifts and eliminating the gain. This work highlights the potential issues arising from mobile defects, which lead to inconsistent photodetector operation, and identifies the potential for defects to tune photodetection behavior in perovskite photodetectors.