COVID-19 is a pandemic that shares certain clinical characteristics with other acute viral infections. Here, we studied the whole-blood transcriptomic host response to SARS-CoV-2 and compared it with other viral infections to understand similarities and differences in host response. Using RNAseq we profiled peripheral blood from 24 healthy controls and 62 prospectively enrolled patients with community-acquired lower respiratory tract infection by SARS-Cov-2 within the first 24 hours of hospital admission. We also compiled and curated 23 independent studies that profiled 1,855 blood samples from patients with one of six viruses (influenza, RSV, HRV, ebola, Dengue, and SARS-CoV-1). We show gene expression changes in peripheral blood in patients with COVID-19 versus healthy controls are highly correlated with changes in response to other viral infections (r=0.74, p<0.001). However, two genes, ACO1 and ATL3, show significantly opposite changes between conditions. Pathway analysis in patients with COVID-19 or other viral infections versus healthy controls identified similar pathways including neutrophil activation, innate immune response, immune response to viral infection, and cytokine production for over-expressed genes. Conversely, for under-expressed genes, pathways indicated repression of lymphocyte differentiation and T cell activation. When comparing transcriptome profiles of patients with COVID-19 directly with those with other viral infections, we found 114 and 302 genes were over- or under-expressed, respectively, during COVID-19. Pathways analysis did not identify any significant pathways in these genes, suggesting novel responses to further study. Statistical deconvolution using immunoStates found that M1 macrophages, plasmacytoid dendritic cells, CD14+ monocytes, CD4+ T cells, and total B cells showed change consistently in the same direction across all viral infections including COVID-19. Those that increased in COVID-19 but decreased in non-COVID-19 viral infections were CD56bright NK cells, M2 macrophages, and total NK cells. The concordant and discordant responses mapped out here provide a window to explore the pathophysiology of COVID-19 versus other viral infections and show clear differences in signaling pathways and cellularity as part of the host response to SARS-CoV-2.