Generalizing Atiyah extensions, we introduce and study differential abelian tensor categories over differential rings. By a differential ring, we mean a commutative ring with an action of a Lie ring by derivations. In particular, these derivations act on a differential category. A differential Tannakian theory is developed. The main application is to the Galois theory of linear differential equations with parameters. Namely, we show the existence of a parameterized Picard-Vessiot extension and, therefore, the Galois correspondence for many differential fields with, possibly, non-differentially closed fields of constants, that is, fields of functions of parameters. Other applications include a substantially simplified test for a system of linear differential equations with parameters to be isomonodromic, which will appear in a separate paper. This application is based on differential categories developed in the present paper, and not just differential algebraic groups and their representations.Keywords: Differential algebra, Tannakian category, Parameterized differential Galois theory, Atiyah extension 2010 MSC: primary 12H05, secondary 12H20, 13N10, 20G05, 20H20, 34M15 of constants than being differentially closed, Theorem 2.5. This assumption is satisfied by many differential fields used in practice, Theorem 2.8.The importance of the existence of a PPV extension is that it leads to a Galois correspondence, Section 8.1. The Galois group is a differential algebraic group [8,9,44,62,11,55,56,71] defined over the field of constants, which, after passing to the differential closure, coincides with the parameterized differential Galois group from [10], Corollary 8.10. The Galois correspondence, as usual, can be used to analyze how one may build the extension, step-by-step, by adjoining solutions of differential equations of lower order, corresponding to taking intermediate extensions of the base field. For example, consider the special function known as the incomplete Gamma function γ, which is the solution of a second-order parameterized differential equation [10, Example 7.2] over Q(x, t). Knowing the relevant Galois correspondence, one could show how to build the differential field extension of Q(x, t) containing γ without taking the (unnecessary and unnatural) differential closure of Q(t).The general nature of our approach will allow in the future to adapt it to the Galois theory of linear difference equations, which has numerous applications. Differential algebraic dependencies among solutions of difference equations were studied in [31,32,33,18,20,19,21,17,16,26]. Among many applications of the Galois theory, one has an algebraic proof of the differential algebraic independence of the Gamma function over C(x), [33] (the Gamma function satisfies the difference equation Γ(x + 1) = x · Γ(x)). Moreover, such a method leads to algorithms, given in the above papers, that test differential algebraic dependency with applications to solutions of even higher order difference equations (hypergeometric functions, etc.). Gen...