Purpose: Light and moderate alcohol use has been reported to be associated with both impaired and enhanced cognition. The purpose of this study was to explore whether reported alcohol use was associated with reduced visual memory in males and females in a large middle-aged birth cohort population. Methods: The study population was from the Northern Finland Birth Cohort 1966 (NFBC1966). The Paired Associate Learning (PAL) test was conducted at the age of 46 years. Reported alcohol use was measured as gram per day use of total alcohol, beer, wine, and spirits, converted into 100% alcohol, and as frequency and amount of use of beer, wine, and spirits. Data on alcohol use was available at age 31 years and at age 46 years. Visual memory was assessed using two outcome variables of the PAL test: the Total errors adjusted score (TEAS) and First Trial Memory Score (FTMS). Both PAL TEA and PAL FTMS were used as continuous variables. The continuous variables were normalised using z-scores. Linear regression analysis was done to measure the associations. Adjusted models included education, marital status, diet, and physical activity. All analyses were done separately in males and in females. Results: Total alcohol consumption (grams per day) was 17.7 at age 31 years and 13.1 at age 46 years in males. The respective figures for females were 7.2 and 5.1. Grams per day use of total alcohol, beer, wine, and spirits, and frequency and amount of use of beer and wine were not associated with visual memory in males or females in cross-sectional or longitudinal settings. Using six or more servings of spirits was associated with worse visual memory in males in both cross-sectional and longitudinal settings. The frequency of use of wine in males was associated with better visual memory in cross-sectional settings. Conclusion: Most participants reported light drinking. The reported gram per day alcohol use was not associated with reduced visual memory in males or females, with the exception of spirits in males. The study suggests a lack of a linear association between drinking and visual memory in the middle-aged population.