KCTD11 has been reported to be a potential tumour suppressor in several tumour types. However, the expression of KCTD11 and its role has not been reported in human non‐small cell lung cancer (NSCLC). Whether its potential molecular mechanism is related to its BTB domain is also unknown. The expression of KCTD11 in 139 NSCLC tissue samples was detected by immunohistochemistry, and its correlation with clinicopathological factors was analysed. The effect of KCTD11 on the biological behaviour of lung cancer cells was verified in vitro and in vivo. Its effect on the epithelial‐mesenchymal transition(EMT)process and the Wnt/β‐catenin and Hippo/YAP pathways were observed by Western blot, dual‐luciferase assay, RT‐qPCR, immunofluorescence and immunoprecipitation. KCTD11 is under‐expressed in lung cancer tissues and cells and was negatively correlated with the degree of differentiation, tumour‐node‐metastasis (TNM) stage and lymph node metastasis. Low KCTD11 expression was associated with poor prognosis. KCTD11 overexpression inhibited the proliferation and migration of lung cancer cells. Further studies indicated that KCTD11 inhibited the Wnt pathway, activated the Hippo pathway and inhibited EMT processes by inhibiting the nuclear translocation of β‐catenin and YAP. KCTD11 lost its stimulatory effect on the Hippo pathway after knock down of β‐catenin. These findings confirm that KCTD11 inhibits β‐catenin and YAP nuclear translocation as well as the malignant phenotype of lung cancer cells by interacting with β‐catenin. This provides an important experimental basis for the interaction between KCTD11, β‐catenin and YAP, further revealing the link between the Wnt and Hippo pathways.