In this paper, we consider the dynamics of a tagged point particle in a gas of moving hard-spheres that are non-interacting among each other. This model is known as the ideal Rayleigh gas. We add to this model the possibility of annihilation (ideal Rayleigh gas with annihilation), requiring that each obstacle is either annihilating or elastic, which determines whether the tagged particle is elastically reflected or removed from the system. We provide a rigorous derivation of a linear Boltzmann equation with annihilation from this particle model in the Boltzmann-Grad limit. Moreover, we give explicit estimates for the error in the kinetic limit by estimating the contributions of the configurations which prevent the Markovianity. The estimates show that the system can be approximated by the Boltzmann equation on an algebraically long time scale in the scaling parameter.