Bile acids play an active role in fat metabolism and, in high-fat diets, elevated concentrations of fecal bile acids may be related to an increased risk of colorectal cancer. This study investigated concentrations of fecal and serum bile acids in 36 vegans and 36 omnivores. The reduced rank regression was used to identify dietary patterns associated with fecal bile acids. Dietary patterns were derived with secondary and conjugated fecal bile acids as response variables and 53 food groups as predictors. Vegans had higher fiber (p < 0.01) and lower fat (p = 0.0024) intake than omnivores. In serum, primary and glycine-conjugated bile acids were higher in vegans than in omnivores (p ≤ 0.01). All fecal bile acids were significantly lower in vegans compared to omnivores (p < 0.01). Processed meat, fried potatoes, fish, margarine, and coffee contributed most positively, whereas muesli most negatively to a dietary pattern that was directly associated with all fecal bile acids. According to the pattern, fat intake was positively and fiber intake was inversely correlated with bile acids. The findings contribute to the evidence that, in particular, animal products and fat may play a part in higher levels of fecal bile acids.