Background The basis of cognitive deficits in people with bipolar disorder (BD) has not been elucidated. These deficits may be the result of the illness or its treatment, but could also reflect genetic risk factors that are shared between BD and cognition. We investigated this question using empirical genetic relationships within a sample of patients with BD and their unaffected relatives. Methods Participants with bipolar I, II, or schizoaffective disorder (narrow BD, n=69), related mood disorders (broad BD, n=135), and their clinically unaffected relatives (n=227) completed tests of matrix reasoning, trail making, digit-symbol coding, semantic short-term recall, and affect recognition (DANVA2). General cognitive function (g) was quantified via principal components analysis (PCA). Heritability and genetic correlations were estimated with SOLAR-Eclipse. Results Participants with a narrow or broad BD diagnosis showed deficits in g, though affect recognition was not impaired. Cognitive performance was significantly heritable (h2 = 0.322 for g, p<0.005). Coheritability between BD and cognition was small (coheritability for g & Narrow = 0.0184, for g & Broad = 0.0327) and healthy relatives of those with BD were cognitively unimpaired. Conclusions In this family sample, cognitive deficits were present in participants with BD, but social cognition, measured by affect recognition, was not impaired. Deficits were largely not explained by overlapping genetic determinants of mood and cognition. These findings support the view that cognition comprises separable social and non-social domains and that cognitive deficits in BD are largely the result of the illness or its treatment.