We explore the effects of long-term secular perturbations on the distribution of dark matter particles around supermassive black hole (BH) binaries. We show that in the hierarchical (in separation) three-body problem, one of the BHs and a dark matter particle form an inner binary. Gravitational perturbations from the BH companion, on a much wider orbit, can cause the dark matter particle to reach extremely high eccentricities and even get accreted onto the BH by what is known as the eccentric Kozai-Lidov (EKL) mechanism. We show that this may produce a torus-like configuration for the dark matter distribution around the less massive member of the BH binary. We first consider an intermediate BH (IMBH) in the vicinity of our galactic center, which may be a relic of a past minor merger. We show that if the IMBH is close enough (i.e., near the stellar disk) the EKL mechanism is very efficient in exciting the eccentricity of dark matter particles in near-polar configurations to extremely high values where they are accreted by the IMBH. We show that this mechanism is even more effective if the central BH grows in mass, where we have assumed adiabatic growth. Because near-polar configurations are disrupted, a torus-like shape is formed. We also show that this behavior is also likely to be relevant for supermassive BH binaries. We suggest that if the BHs are spinning, the accreted dark matter particles may linger in the ergosphere, and thereby generate self-annihilations and produce an indirect signature of potential interest.