In the present work, we consider weakly-singular integral equations arising from linear second-order elliptic PDE systems with constant coefficients, including, e.g. linear elasticity. We introduce a general framework for optimal convergence of adaptive Galerkin BEM. We identify certain abstract conditions for the underlying meshes, the corresponding mesh-refinement strategy, and the ansatz spaces that guarantee that the weighted-residual error estimator is reliable and converges at optimal algebraic rate if used within an adaptive algorithm. These conditions are satisfied, e.g. for discontinuous piecewise polynomials on simplicial meshes as well as certain ansatz spaces used for isogeometric analysis. Technical contributions include the localization of (non-local) fractional Sobolev norms and local inverse estimates for the (non-local) boundary integral operators associated to the PDE system.