The function of antibodies (Abs) involves specific binding to antigens (Ags) and activation of other components of the immune system to fight pathogens. The six hypervariable loops within the variable domains of Abs, commonly termed complementarity determining regions (CDRs), are widely assumed to be responsible for Ag recognition, while the constant domains are believed to mediate effector activation. Recent studies and analyses of the growing number of available Ab structures, indicate that this clear functional separation between the two regions may be an oversimplification. Some positions within the CDRs have been shown to never participate in Ag binding and some off-CDRs residues often contribute critically to the interaction with the Ag. Moreover, there is now growing evidence for non-local and even allosteric effects in Ab-Ag interaction in which Ag binding affects the constant region and vice versa. This review summarizes and discusses the structural basis of Ag recognition, elaborating on the contribution of different structural determinants of the Ab to Ag binding and recognition. We discuss the CDRs, the different approaches for their identification and their relationship to the Ag interface. We also review what is currently known about the contribution of non-CDRs regions to Ag recognition, namely the framework regions (FRs) and the constant domains. The suggested mechanisms by which these regions contribute to Ag binding are discussed. On the Ag side of the interaction, we discuss attempts to predict B-cell epitopes and the suggested idea to incorporate Ab information into B-cell epitope prediction schemes. Beyond improving the understanding of immunity, characterization of the functional role of different parts of the Ab molecule may help in Ab engineering, design of CDR-derived peptides, and epitope prediction.