The structural requirements for ligand binding to the benzodiazepine receptor (BzR) inverse agonist site were probed through the synthesis and in vitro evaluation of 3-substituted beta-carbolines 6, 7, 11, 12, gamma-carboline 13, and diindoles 18-21, 23-25, 27, 28, and 34. On the basis of the apparent binding affinities of these and other analogues, a hydrogen bond acceptor site (A2) on the receptor is proposed to interact with the N(9) hydrogen atom of the beta-carbolines or the N(7) hydrogen nuclei of the diindoles. Likewise, a proposed hydrogen bond donating site (H1) interacts with the N(2) nitrogen atom of the beta-carbolines or the N(5) nitrogen atom of the diindoles. It appears that interaction with both sites is a prerequisite for high affinity since analogues which have either one or both of these positions blocked exhibit substantial reduction in affinity. Moreover, H1 appears to be capable of engaging in a three-centered hydrogen bond with appropriately functionalized ligands, which explains the increase in potency observed in the following series of 3-substituted beta-carbolines: the n-butyl (12, IC50 = 245 nM), n-propoxy (9, IC50 = 11 nM), and propyl ketone (11, IC50 = 2.8 nM) congeners. In addition to H1 and A2, there appears to be a relatively narrow hydrophobic pocket in the binding cleft that can accommodate substituents at the 3-position of the beta-carbolines which have chain lengths less than or equal to C5. There is a 1 order of magnitude decrease in affinity between n-propoxy analogue 9 (IC50 = 11 nM, chain length = 4) and n-butoxy derivative 7 (IC50 = 98 nM, chain length = 5). Furthermore, alpha- and gamma-branching [e.g. ethoxycarbonyl (2), IC50 = 5 nM and tert-butoxycarbonyl (31) IC50 = 10 nM] but not beta- and delta-branching [e.g. isopropoxy (6), IC50 = 500 nM and (neopentyloxy) carbonyl (48), IC50 = 750 nM] at position 3 are tolerated. Occupation of this hydrophobic pocket is clearly important for high affinity as evidenced by the relatively low affinity of 30, a beta-carboline which possesses a hydrogen atom at the 3-position. This same hydrophobic pocket is partially filled by the D and E rings of the diindoles, which accounts for the high affinity of several members of this series. An excluded volume analysis using selected 3-substituted beta-carbolines and ring-E substituted pyridodiindoles is consistent with the presence of this hydrophobic pocket (see Figure 1).(ABSTRACT TRUNCATED AT 400 WORDS)