Let n ≥ k ≥ 2 be two integers and S a subset of {0, 1, . . . , k − 1}. The graph J S (n, k) has as vertices the k-subsets of the n-set [n] = {1, . . . , n} and two k-subsets A and B are adjacent if |A ∩ B| ∈ S. In this paper, we use Godsil-McKay switching to prove that for m ≥ 0, k ≥ max(m + 2, 3) and S = {0, 1, ..., m}, the graphs J S (3k − 2m − 1, k) are not determined by spectrum and for m ≥ 2, n ≥ 4m + 2 and S = {0, 1, ..., m} the graphs J S (n, 2m+1) are not determined by spectrum. We also report some computational searches for Godsil-McKay switching sets in the union of classes in the Johnson scheme for k ≤ 5.