A strategy for the Hybridizable Discontinous Galerkin (HDG) solution of problems with voids, inclusions or free surfaces is proposed. It is based on an eXtended Finite Element philosophy with a level-set description of interfaces. Thus, the computational mesh is not required to fit the interface (i.e. the boundary), simplifying and reducing the cost of mesh generation and, in particular, avoiding continuous remeshing for evolving interfaces. Differently to previous proposals for HDG solution with non-fitting meshes, here the computational mesh covers the domain, avoiding extrapolations, and ensuring the robustness of the method. The local problem at elements not cut by the interface, and the global problem, are discretized as usual in HDG. A modified local problem is considered at elements cut by the interface. At every cut element, an auxiliary trace variable on the boundary is introduced, which is eliminated afterwards using the boundary conditions on the interface, keeping the original unknowns and the structure of the local problem solver. An efficient and robust methodology for numerical integration in cut elements, in the context of high-order approximations, is also proposed. Numerical experiments demonstrate how X-HDG keeps the optimal convergence, superconvergence, and accuracy of HDG with no need of adapting the computational mesh to the interface boundary.