Recent deep-learning (DL) protein design methods have been successfully applied to a range of protein design problems including the de novo design of novel folds, protein binders, and enzymes. However, DL methods have yet to meet the challenge of de novo membrane protein (MP) and β-barrel design tasks. We performed a comprehensive benchmark of one DL protein sequence design method, ProteinMPNN, on MP and β-barrel design tasks, and compared the performance of ProteinMPNN to the state-of-the-art Franklin2023 Rosetta MP energy function. We characterized the ability of ProteinMPNN to capture global sequence properties of transmembrane β-barrels (TMBs), generate diverse sequences for novel folds, and generate sequences likely to fold in vitro. We also tested the effect of input backbone refinement on ProteinMPNN design success. We found that given refined and well-defined inputs, ProteinMPNN more accurately captures global sequence properties and generates TMB sequences with higher sequence diversity of pore-facing residues than Franklin2023. In addition, ProteinMPNN was able to design TMB sequences likely to fold in vitro, suggesting that it could be used in de novo design tasks of diverse nanopores for single-molecule sensing and sequencing. Lastly, the improvement of ProteinMPNN with input refinement indicates that the difficulty of ProteinMPNN in designing sequences for challenging protein folds, such as TMBs, stems from input definition rather than software limitations.