Background: Artificial intelligence (AI) chatbots like ChatGPT have made significant progress. These chatbots, particularly popular among healthcare professionals and patients, are transforming patient education and disease experience with personalized information. They are especially beneficial for populations such as men with prostate cancer concerns. Accurate, timely patient education is crucial for informed decision-making, especially regarding Prostate-Specific Antigen screening and treatment options. AI chatbots can address the gap in quality prostate cancer information, reaching wider demographics, including remote communities. However, the accuracy and reliability of AI chatbots' medical information must be rigorously evaluated. Studies testing ChatGPT's knowledge in prostate cancer are emerging, but there's a need for ongoing evaluation to ensure the quality and safety of information provided to patients.Objective: To evaluate the quality, accuracy, and readability of ChatGPT-4's responses to common prostate cancer questions posed by patients.Methods: Eight questions were formulated with an inductive approach. These were based on information topics searched for and desired by prostate cancer patients in peer reviewed literature, and Google Trends data. The eight artificial intelligence (AI) outputs were judged by seven expert urologists, using an assessment framework developed to assess accuracy, safety, appropriateness, actionability and effectiveness. Adapted versions of the Patient Education Materials Assessment Tool (PEMAT-AI), Global Quality Score (GQS), and DISCERN (DISCERN-AI) tools were used by four independent reviewers to assess the quality of the AI responses. Readability of the AI responses was assessed using established algorithms (Flesch Reading Ease score, Gunning Fog Index, Flesch-Kincaid Grade Level, The Coleman-Liau Index and SMOG Index). A brief tool (REF-AI) was developed for analysis of the references provided by AI outputs, assessing for reference hallucination, relevance, and quality of references.