We report the experimental realization of the prime number quantum potential VN(x), defined as the potential entering the single-particle Schrödinger Hamiltonian with eigenvalues given by the first N prime numbers. Using computer-generated holography, we create light intensity profiles suitable to optically trap ultracold atoms in these potentials for different N values. As a further application, we also implement a potential whose spectrum is given by the lucky numbers, a sequence of integers generated by a different sieve than the familiar Eratosthenes’s sieve used for the primes. Our results pave the way towards the realization of quantum potentials with arbitrary sequences of integers as energy levels and show, in perspective, the possibility to set up quantum systems for arithmetic manipulations or mathematical tests involving prime numbers.