A binary number-conserving cellular automaton is a discrete dynamical system that models the movement of particles in a d-dimensional grid. Each cell of the grid is either empty or contains a particle. In subsequent time steps the particles move between the cells, but in one cell there can be at most one particle at a time. In this paper, the von Neumann neighborhood is considered, which means that in each time step a particle can move to an adjacent cell only. It is proven that regardless of the dimension d, all of these cellular automata are trivial, as they are intrinsically one-dimensional. Thus, for given d, there are only 4d + 1 binary number-conserving cellular automata with the von Neumann neighborhood: the identity rule and the shift and traffic rules in each of the 2d possible directions.