Recently, Jones introduced a method of constructing knots and links from elements of Thompson's group F by using its unitary representations. He also defined several subgroups of F as the stabilizer subgroups and some researchers studied them algebraically. One of the subgroups is called the 3-colorable subgroup F, and the authors proved that all knots and links obtained from non-trivial elements of F are 3-colorable. In this paper, for any odd integer p greater than two, we define the p-colorable subgroup of F whose non-trivial elements yield p-colorable knots and links and show it is isomorphic to the certain Brown-Thompson group.