Let E : y 2 = x 3 +A(T )x+B(T ) be a nontrivial one-parameter family of elliptic curves over Q(T ), with A(T ), B(T ) ∈ Z(T ), and consider theRosen and Silverman proved a conjecture of Nagao relating the first moment A k,E (p) to the rank of the family over Q(T ), and Michel proved that if j(T ) is not constant then the second moment is equal to A k,E (p) = p 2 + O(p 3/2 ). Cohomological arguments show that the lower order terms are of sizes p 3/2 , p, p 1/2 , and 1. In every case we are able to analyze, the largest lower order term in the second moment expansion that does not average to zero is on average negative. We prove this "bias conjecture" for several large classes of families, including families with rank, complex multiplication, and constant j(T )-invariant. We also study the analogous bias conjecture for families of Dirichlet characters, holomorphic forms on GL(2)/Q, and their symmetric powers and Rankin-Selberg convolutions. We identify all lower order terms in large classes of families, shedding light on the arithmetic objects controlling these terms. The negative bias in these lower order terms has implications toward the excess rank conjecture and the behavior of zeros near the central point of elliptic curve L-functions.