Third-party companies increasingly track users' web browsing behavior, which raises privacy concerns. A number of browser extensions inform users about this tracking, yet the extensions' impact on user attitudes has not been well studied. We conducted a 24-participant, qualitative lab study evaluating how three popular extensions (Ghostery, DoNotTrackMe, and Disconnect) influence users' privacy awareness and privacy concerns. Before using any tool, many participants assumed tracking occurs, yet were unsure of specifics. The extensions provided limited insight; participants remained uncertain who the tracking companies are, what data they collect, and for what purpose. While using a browser extension, participants reported increased privacy concern due to increased awareness of tracking, yet this concern was mitigated by feeling protected by the extension. However, some participants distrusted the extensions or concluded the extension would tracked them. While all three extensions provided some additional awareness, users remained confused about many aspects of data tracking. Permission to freely reproduce all or part of this paper for noncommercial purposes is granted provided that copies bear this notice and the full citation on the first page. Reproduction for commercial purposes is strictly prohibited without the prior written consent of the Internet Society, the first-named author (for reproduction of an entire paper only), and the author's employer if the paper was prepared within the scope of employment.