Can Artificial Intelligence Pass Accounting Certification Exams? ChatGPT: CPA, CMA, CIA, and EA?

Eulerich, Marc; Sanatizadeh, Aida; Vakilzadeh, Hamid; Wood, David А.

doi:10.2139/ssrn.4452175

Cited by 13 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several strategies can enhance ChatGPT's performance, such as prompt engineering to encourage more thoughtful responses by spreading reasoning across multiple prompts (‘chain of thought prompting’ [ 30 , 31 ]). Methods such as self-consistency [ 32 ], try-again [ 33 ], and backtracking [ 28 ] enable the model to review its output, and its capabilities can be enhanced by connecting it to tools like online apps or calculators [ 25 , 34 – 36 ].…”

Section: Introductionmentioning

confidence: 99%

Using ChatGPT for human–computer interaction research: a primer

Tabone,

de Winter

2023

R. Soc. open sci.

View full text Add to dashboard Cite

ChatGPT could serve as a tool for text analysis within the field of Human–Computer Interaction, though its validity requires investigation. This study applied ChatGPT to: (1) textbox questionnaire responses on nine augmented-reality interfaces, (2) interview data from participants who experienced these interfaces in a virtual simulator, and (3) transcribed think-aloud data of participants who viewed a real painting and its replica. Using a hierarchical approach, ChatGPT produced scores or summaries of text batches, which were then aggregated. Results showed that (1) ChatGPT generated sentiment scores of the interfaces that correlated extremely strongly ( r > 0.99) with human rating scale outcomes and with a rule-based sentiment analysis method (criterion validity). Additionally, (2) by inputting automatically transcribed interviews to ChatGPT, it provided meaningful meta-summaries of the qualities of the interfaces (face validity). One meta-summary analysed in depth was found to have substantial but imperfect overlap with a content analysis conducted by an independent researcher (criterion validity). Finally, (3) ChatGPT's summary of the think-aloud data highlighted subtle differences between the real painting and the replica (face validity), a distinction corresponding with a keyword analysis (criterion validity). In conclusion, our research indicates that, with appropriate precautions, ChatGPT can be used as a valid tool for analysing text data.

show abstract

Section: Introductionmentioning

confidence: 99%

Using ChatGPT for human–computer interaction research: a primer

Tabone,

de Winter

2023

R. Soc. open sci.

View full text Add to dashboard Cite

show abstract

“…Data collection primarily revolves around the design and administration of exams across the 11 subjects. The methodology applied in the research is consistent with the experiments conducted by Wood et al, (2023), Eulerich et al (2023) and Bommarito et al (2023). This experiment involves a qualitative assessment of ChatGPT 3.5's readiness to tackle exams in the domain of accounting and auditing.…”

Section: Methodsmentioning

confidence: 99%

“… Conversely, ChatGPT encountered challenges when dealing with practice questions and providing short answers, achieving lower accuracy rates of 28.7 percent and 39.1 percent, respectively. Eulerich et al (2023) conducted an investigation into the potential of newly released ChatGPT models and their capabilities to successfully pass major accounting certification exams, including the CPA, CMA, CIA, and EA certification exams. Their findings indicate that the initial release of ChatGPT 3.5 was unable to achieve a passing score on any of these exams, with an average score of 53.1 percent across all assessments.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Their findings indicate that the initial release of ChatGPT 3.5 was unable to achieve a passing score on any of these exams, with an average score of 53.1 percent across all assessments. However, through additional refinements and enhancements, including the transition to the ChatGPT 4 model, 10-shot training, and permitting the model to use reasoning and external resources, ChatGPT demonstrated significant improvement, achieving an average score of 85.1 percent across all exam sections and successfully passing all exams (Eulerich et al, 2023). Bommarito et al (2023) conducted a thorough assessment of professional knowledge worker readiness using the Uniform CPA Examination from AICPA as a benchmark.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Performance of ChatGPT in Accounting and Auditing Exams: An Experimental Study in North Macedonia

Atanasovski,

Tocev,

Dionisijev

et al. 2023

Proceedings of the International Conference "Economic and Business Trends Shaping the Future"

View full text Add to dashboard Cite

In the ever-evolving landscape of education, the integration of artificial intelligence (AI) has emerged as a transformative force, reshaping conventional pedagogical methods and assessment approaches. The domains of accounting and auditing, integral for preparing future professionals, have not remained immune to this technological revolution. As AI, including ChatGPT, gains prominence, it offers the potential to enhance learning and examination experiences within these disciplines. This paper presents a pioneering North Macedonian experimental research evaluating the performance of ChatGPT's effectiveness in exam performance in accounting and auditing. Leveraging 11 subject exams comprising a total of 401 questions, this research assesses ChatGPT's proficiency in addressing intricate, context-specific questions within these domains. ChatGPT 3.5 demonstrated its reliability as a "student" by successfully passing 8 out of the 11 subjects, achieving a pass rate of 73%. The findings reveal that ChatGPT 3.5 excels in responding to qualitative open-ended questions, true/false assessments, and multiple-choice questions with a single correct answer, effectively streamlining routine tasks and offering comprehensive explanations. However, it faces challenges when dealing with quantitative calculations and complex multiple-choice questions with multiple correct answers. This study contributes to the ongoing discourse on AI's role in education, sparking discussions about student access to AI tools, the balance between memorization and referencing, the potential for AI to enhance learning experiences, and the unique contributions of educators that surpass AI capabilities. As AI continues to disrupt traditional practices across professions, including education, its responsible integration promises to enhance the educational process and deliver substantial value. ChatGPT, as a versatile AI language model, stands as a powerful tool in this endeavor, offering educators and students the potential to enrich their learning journeys while recognizing the need for judicious application within its defined constraints.

show abstract

“…As generative AI becomes more prevalent in society, expectations have risen substantially. Several articles highlight that emerging technologies such as generative AI often experience stages of initial enthusiasm, followed by disappointment, and finally, practical application, cautioning against overestimating their immediate impact (Eulerich et al, 2023). For some users, their expectations may not be fulfilled probably due to too much expectation.…”

Section: Practical Implicationsmentioning

confidence: 99%

Performance of Artificial Intelligence: Does Artificial Intelligence Dream of Electric Sheep

Ioku,

Kondo,

Watanabe

2024

Preprint

View full text Add to dashboard Cite

This study investigates the performance of generative artificial intelligence (AI) in evaluating the acceptance of generative AI technologies within higher education guidelines, reflecting on the implications for educational policy and practice. Drawing on a dataset of guidelines from top-ranked universities, we compared generative AI evaluations with human evaluations, focusing on acceptance, performance expectancy, facilitating conditions, and perceived risk. Our study revealed a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI, suggesting that generative AI can accurately reflect human judgment in this context. Further, we found positive associations between ChatGPT-rated acceptance and performance expectancy and facilitating conditions, while a negative correlation with perceived risk. These results validate generative AI evaluation, which also extends the application of the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology framework from individual to institutional perspectives.

show abstract

Can Artificial Intelligence Pass Accounting Certification Exams? ChatGPT: CPA, CMA, CIA, and EA?

Cited by 13 publications

References 0 publications

Using ChatGPT for human–computer interaction research: a primer

Using ChatGPT for human–computer interaction research: a primer

Evaluating the Performance of ChatGPT in Accounting and Auditing Exams: An Experimental Study in North Macedonia

Performance of Artificial Intelligence: Does Artificial Intelligence Dream of Electric Sheep

Contact Info

Product

Resources

About