Mental health resources available via websites and mobile apps provide support such as advice, journaling, and elements from cognitive behavioral therapy. The proliferation of spoken conversational agents, such as Alexa, Siri, and Google Home, has led to an increasing interest in developing mental health apps for these devices. We present the pilot study outcomes of an Alexa Skill that allows users to conduct depression and anxiety self-tests. Ten participants were given access to the Alexa Skill for two-weeks, followed by an online evaluation of the Skill's usability and trust. Our preliminary evaluation suggests that participants trusted the Skill and scored the usability and user experience as average. Usage of the Skill was low, with most participants using the Skill only once. In view of work-in-progress, we also present a discussion of implementation and study design challenges to guide the current literature on designing spoken conversational agents for mental health applications. CCS CONCEPTS • Human-centered computing → Sound-based input / output.