Protein design and engineering are evolving at an unprecedented pace leveraging the advances of deep learning. Current models nonetheless cannot natively consider non-protein entities within the design process. Here we introduce a deep learning approach based solely on a geometric transformer of atomic coordinates that predicts protein sequences from backbone scaffolds aware of the restraints imposed by diverse molecular environments. This new concept is anticipated to improve the design versatility for engineering proteins with desired functions.