The widespread adoption of smart home devices (SHD) has increased privacy concerns among users, yet user-friendly controls are lacking. While there is a large body of research focused on understanding privacy concerns and threat models of SHD users, there is limited research so far aimed at informing the development of privacy controls in SHDs. This paper presents the results of 25 interviews focused on characterizing the users' needs for privacy controls. Through qualitative analysis of interview data, we present 7 design factors and 32 sub-factors for the design of privacy controls in SHDs. The interview findings inform the design of a survey that was deployed to 440 adult SHD users to gain quantitative insights on privacy control requirements and to complement the interview findings. Based on the findings, we discuss a privacy control framework that guides designers towards user-centric privacy controls.