This paper presents a general modelling framework for restricted facility location problems with arbitrarily shaped forbidden regions or barriers, where regions are modelled using phi-objects. Phi-objects are an efficient tool in mathematical modelling of 2D and 3D geometric optimization problems, and are widely used in cutting and packing problems and covering problems. The paper shows that the proposed modelling framework can be applied to both median and centre facility location problems, either with barriers or forbidden regions. The resulting models are either mixed-integer linear or non-linear programming formulations, depending on the shape of the restricted region and the considered distance measure. Using the new framework, all instances from the existing literature for this class of problems are solved to optimality. The paper also introduces and optimally solves a realistic multifacility problem instance derived from an archipelago vulnerable to earthquakes. This problem instance is significantly more complex than any other instance described in the literature.