A finite set can be supplied with a group structure which can then be used to select (classes of) differential calculi on it via the notions of left-, right-and bicovariance. A corresponding framework has been developed by Woronowicz, more generally for Hopf algebras including quantum groups. A differential calculus is regarded as the most basic structure needed for the introduction of further geometric notions like linear connections and, moreover, for the formulation of field theories and dynamics on finite sets. Associated with each bicovariant first order differential calculus on a finite group is a braid operator which plays an important role for the construction of distinguished geometric structures. For a covariant calculus, there are notions of invariance for linear connections and tensors. All these concepts are explored for finite groups and illustrated with examples. Some results are formulated more generally for arbitrary associative (Hopf) algebras. In particular, the problem of extension of a connection on a bimodule (over an associative algebra) to tensor products is investigated, leading to the class of 'extensible connections'. It is shown that invariance properties of an extensible connection on a bimodule over a Hopf algebra are carried over to the extension. Furthermore, an invariance property of a connection is also shared by a 'dual connection' which exists on the dual bimodule (as defined in this work).