We consider Dotsenko-Fateev matrix models associated with compactified Calabi-Yau threefolds. They can be constructed with the help of explicit expressions for refined topological vertex, i.e. are directly related to the corresponding topological string amplitudes. We describe a correspondence between these amplitudes, elliptic and affine type Selberg integrals and gauge theories in five and six dimensions with various matter content. We show that the theories of this type are connected by spectral dualities, which can be also seen at the level of elliptic Seiberg-Witten integrable systems. The most interesting are the spectral duality between the XYZ spin chain and the Ruijsenaars system, which is further lifted to self-duality of the double elliptic system.