We use the ideas of goal-oriented error estimation and adaptivity to design and implement an efficient adaptive algorithm for approximating linear quantities of interest derived from solutions to elliptic partial differential equations (PDEs) with parametric or uncertain inputs. In the algorithm, the stochastic Galerkin finite element method (sGFEM) is used to approximate the solutions to primal and dual problems that depend on a countably infinite number of uncertain parameters. Adaptive refinement is guided by an innovative strategy that combines the error reduction indicators computed for spatial and parametric components of the primal and dual solutions. The key theoretical ingredient is a novel two-level a posteriori estimate of the energy error in sGFEM approximations. We prove that this error estimate is reliable and efficient. The effectiveness of the goal-oriented error estimation strategy and the performance of the goal-oriented adaptive algorithm are tested numerically for three representative model problems with parametric coefficients and for three quantities of interest (including the approximation of pointwise values).Adaptive techniques based on rigorous a posteriori error analysis of computed solutions provide an effective mechanism for building approximation spaces and accelerating convergence of computed solutions. These techniques rely heavily on how the approximation error is estimated and controlled. One may choose to estimate the error in the global energy norm and use the associated error indicators to enhance the computed solution and drive the energy error estimate to zero. However, in practical applications, simulations often target a specific (e.g., localized) feature of the solution, called the quantity of interest and represented using a linear functional of the solution. In these cases, the energy norm may give very little useful information about the simulation error.Alternative error estimation techniques, such as goal-oriented error estimations, e.g., by the dual-weighted residual methods, allow to control the errors in the quantity of interest. While for deterministic PDEs, these error estimation techniques and the associated adaptive algorithms are very well studied (see, e.g., [EEHJ95, JS95, BR96, PO99, BR01, GS02, BR03] for the a posteriori error estimation and [MS09, BET11, HP16, FPZ16] for a rigorous convergence analysis of adaptive algorithms), relatively little work has been done for PDEs with parametric or uncertain inputs. For example, in the framework of (intrusive) stochastic Galerkin finite element methods (sGFEMs) (see, e.g., [GS91, LPS14]), a posteriori error estimation of linear functionals of solutions to PDEs with parametric uncertainty is addressed in [MLM07] and, for nonlinear problems, in [BDW11]. In particular, in [MLM07], a rigorous estimator for the error in the quantity of interest is derived and several adaptive refinement strategies are discussed. However, the authors comment that the proposed estimator lacks information about the structure of the estim...