Symmetries of a partial differential equation (PDE) can be defined as the solutions of the linearization (Frechet derivative) equation holding on the space of solutions to the PDE, and they are well-known to comprise a linear space having the structure of a Lie algebra. Solutions of the adjoint linearization equation holding on the space of solutions to the PDE are called adjoint-symmetries. Their algebraic structure for general PDE systems is studied herein. This is motivated by the correspondence between variational symmetries and conservation laws arising from Noether's theorem, which has a well-known generalization to non-variational PDEs, where symmetries are replaced by adjoint-symmetries, and variational symmetries are replaced by multipliers (adjoint-symmetries satisfying a certain Euler-Lagrange condition). Several main results are obtained. Symmetries are shown to have three different linear actions on the linear space of adjoint-symmetries. These linear actions are used to construct bilinear adjoint-symmetry brackets, one of which is like a pull-back of the symmetry commutator bracket and has the properties of a Lie bracket. In the case of variational PDEs, adjoint-symmetries coincide with symmetries, and the linear actions themselves constitute new bilinear symmetry brackets which differ from the commutator bracket when acting on non-variational symmetries. Several examples of nonlinear PDEs are used to illustrate all of the results.