We use the ideas of goal-oriented error estimation and adaptivity to design and implement an efficient adaptive algorithm for approximating linear quantities of interest derived from solutions to elliptic partial differential equations (PDEs) with parametric or uncertain inputs. In the algorithm, the stochastic Galerkin finite element method (sGFEM) is used to approximate the solutions to primal and dual problems that depend on a countably infinite number of uncertain parameters. Adaptive refinement is guided by an innovative strategy that combines the error reduction indicators computed for spatial and parametric components of the primal and dual solutions. The key theoretical ingredient is a novel two-level a posteriori estimate of the energy error in sGFEM approximations. We prove that this error estimate is reliable and efficient. The effectiveness of the goal-oriented error estimation strategy and the performance of the goal-oriented adaptive algorithm are tested numerically for three representative model problems with parametric coefficients and for three quantities of interest (including the approximation of pointwise values).