In this article, we provide a rigorous analysis of the solution to elliptic diffusion problems on random domains. In particular, based on the decay of the Karhunen-Loève expansion of the domain perturbation field, we establish decay rates for the derivatives of the random solution that are independent of the stochastic dimension. For the implementation of a related approximation scheme, like quasi-Monte Carlo quadrature, stochastic collocation, etc., we propose parametric finite elements to compute the solution of the diffusion problem on each individual realization of the domain generated by the perturbation field. This simplifies the implementation and yields a non-intrusive approach. Having this machinery at hand, we can easily transfer it to stochastic interface problems. The theoretical findings are complemented by numerical examples for both, stochastic interface problems and boundary value problems on random domains.