Gene regulatory network inference is a standard technique for obtaining structured regulatory information from, among other data sources, gene expression measurements. Methods performing this task have been extensively evaluated on synthetic, and to a lesser extent real data sets. They are often applied to gene expression of human cancers. However, in contrast to the evaluations, these data sets often contain fewer samples, more potential regulatory links, and are biased by copy number aberrations as well as cell mixtures and sample impurities. Here, we take networks inferred from TCGA cohorts as an example to show that (1) transcription factor annotations are essential to obtaining reliable networks, and (2) even when taking these into account, we should expect between 20 and 80% of edges to be caused by copy number changes and cell mixtures rather than transcription factor regulation.