Genes, including those with transgenerational effects, work in concert with behavioral, environmental, and social factors via complex biological networks to determine human health. Understanding complex relationships between causal factors underlying human health is an essential step towards deciphering biological mechanisms. We propose a new analytical framework to investigate the interactions between maternal and offspring genetic variants or their surrogate single nucleotide polymorphisms (SNPs) and environmental factors using family-based hybrid study design. The proposed approach can analyze diverse genetic and environmental factors and accommodate samples from a variety of family units, including case/control-parental triads, and case/control-parental dyads, while minimizing potential bias introduced by population admixture. Comprehensive simulations demonstrated that our innovative approach outperformed the log-linear approach, the best available method for case-control family data. The proposed approach had greater statistical power and was capable to unbiasedly estimate the maternal and child genetic effects and the effects of environmental factors, while controlling the Type I error rate against population stratification. Using our newly developed approach, we analyzed the associations between maternal and fetal SNPs and obstructive and conotruncal heart defects, with adjustment for demographic and lifestyle factors and dietary supplements. Fourteen and 11 fetal SNPs were associated with obstructive and conotruncal heart defects, respectively. Twenty-seven and 17 maternal SNPs were associated with obstructive and conotruncal heart defects, respectively. In addition, maternal body mass index was a significant risk factor for obstructive defects. The proposed approach is a powerful tool for interrogating the etiological mechanism underlying complex traits.