The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?

David A. Wood; Muskan P. Achhpilia; Mollie T. Adams; Sanaz Aghazadeh; Kazeem Akinyele; Mfon Akpan; Kristian D. Allee; Abigail M. Allen; Elizabeth D. Almer; Daniel Ames; Viktor Arity; Dereck Barr-Pulliam; K. Asli Basoglu; Andrew Belnap; Jeremiah W. Bentley; Terje Berg; Nathan R. Berglund; Erica Berry; Avishek Bhandari; Md Nazmul Hasan Bhuyan; Paul W. Black; Eva Blondeel; David Bond; Annika Bonrath; A. Faye Borthick; Erik S. Boyle; Marianne Bradford; Duane M. Brandon; Joseph F. Brazel; Bryan G. Brockbank; Marcus Burger; Dmitri Byzalov; James N. Cannon; Cecil Caro; Abraham H. Carr; Jack Cathey; Ryan Cating; Kimberly Charron; Stacy Chavez; Jason Chen; Jennifer C. Chen; Jennifer W. Chen; Christine Cheng; Xu Cheng; Brant E. Christensen; Kimberly Swanson Church; N. J. Cicone; Wim Maas; Jesse van der Geest; Victor van Pelt; et al.

doi:10.2308/issues-2023-013

The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?

David A. Wood, Muskan P. Achhpilia, Mollie T. Adams, Sanaz Aghazadeh, Kazeem Akinyele, Mfon Akpan, Kristian D. Allee, Abigail M. Allen, Elizabeth D. Almer, Daniel Ames, Viktor Arity, Dereck Barr-Pulliam, K. Asli Basoglu, Andrew Belnap, Jeremiah W. Bentley, Terje Berg, Nathan R. Berglund, Erica Berry, Avishek Bhandari, Md Nazmul Hasan BhuyanPaul W. Black, Eva Blondeel, David Bond, Annika Bonrath, A. Faye Borthick, Erik S. Boyle, Marianne Bradford, Duane M. Brandon, Joseph F. Brazel, Bryan G. Brockbank, Marcus Burger, Dmitri Byzalov, James N. Cannon, Cecil Caro, Abraham H. Carr, Jack Cathey, Ryan Cating, Kimberly Charron, Stacy Chavez, Jason Chen, Jennifer C. Chen, Jennifer W. Chen, Christine Cheng, Xu Cheng, Brant E. Christensen, Kimberly Swanson Church, N. J. Cicone, Wim Maas, Jesse van der Geest, Victor van Pelt, et al.

Research output: Contribution to journal › Article › Scientific › peer-review

133 Downloads (Pure)

Abstract

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research.

Original language	English
Pages (from-to)	81-108
Journal	Issues in Accounting Education ...
Volume	38
Issue number	4
DOIs	https://doi.org/10.2308/issues-2023-013
Publication status	Published - Nov 2023

Keywords

artificial intelligence
crowdsourcing
chat bots
GPT-3
accounting education
assessment

Access to Document

10.2308/issues-2023-013

The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?Final published version, 1.15 MBLicence: Taverne

Notable Contribution to the Literature Award by the AIS Section of AAA
Wood, D. A. (Recipient), Maas, W. (Recipient) & et al. (Recipient), 2024
Prize

Cite this

Wood, D. A., Achhpilia, M. P., Adams, M. T., Aghazadeh, S., Akinyele, K., Akpan, M., Allee, K. D., Allen, A. M., Almer, E. D., Ames, D., Arity, V., Barr-Pulliam, D., Basoglu, K. A., Belnap, A., Bentley, J. W., Berg, T., Berglund, N. R., Berry, E., Bhandari, A., ... et al. (2023). The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions? Issues in Accounting Education ..., 38(4), 81-108. https://doi.org/10.2308/issues-2023-013

@article{b4a29b3e18a14259a70cde20a7311748,

title = "The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?",

abstract = "ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users{\textquoteright} questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research.",

keywords = "artificial intelligence, crowdsourcing, chat bots, GPT-3, accounting education, assessment",

author = "Wood, {David A.} and Achhpilia, {Muskan P.} and Adams, {Mollie T.} and Sanaz Aghazadeh and Kazeem Akinyele and Mfon Akpan and Allee, {Kristian D.} and Allen, {Abigail M.} and Almer, {Elizabeth D.} and Daniel Ames and Viktor Arity and Dereck Barr-Pulliam and Basoglu, {K. Asli} and Andrew Belnap and Bentley, {Jeremiah W.} and Terje Berg and Berglund, {Nathan R.} and Erica Berry and Avishek Bhandari and Bhuyan, {Md Nazmul Hasan} and Black, {Paul W.} and Eva Blondeel and David Bond and Annika Bonrath and Borthick, {A. Faye} and Boyle, {Erik S.} and Marianne Bradford and Brandon, {Duane M.} and Brazel, {Joseph F.} and Brockbank, {Bryan G.} and Marcus Burger and Dmitri Byzalov and Cannon, {James N.} and Cecil Caro and Carr, {Abraham H.} and Jack Cathey and Ryan Cating and Kimberly Charron and Stacy Chavez and Jason Chen and Chen, {Jennifer C.} and Chen, {Jennifer W.} and Christine Cheng and Xu Cheng and Christensen, {Brant E.} and Church, {Kimberly Swanson} and Cicone, {N. J.} and Wim Maas and Geest, {Jesse van der} and Pelt, {Victor van} and {et al.}",

year = "2023",

month = nov,

doi = "10.2308/issues-2023-013",

language = "English",

volume = "38",

pages = "81--108",

journal = "Issues in Accounting Education ...",

issn = "0739-3172",

publisher = "American Accounting Association",

number = "4",

}

Wood, DA, Achhpilia, MP, Adams, MT, Aghazadeh, S, Akinyele, K, Akpan, M, Allee, KD, Allen, AM, Almer, ED, Ames, D, Arity, V, Barr-Pulliam, D, Basoglu, KA, Belnap, A, Bentley, JW, Berg, T, Berglund, NR, Berry, E, Bhandari, A, Bhuyan, MNH, Black, PW, Blondeel, E, Bond, D, Bonrath, A, Borthick, AF, Boyle, ES, Bradford, M, Brandon, DM, Brazel, JF, Brockbank, BG, Burger, M, Byzalov, D, Cannon, JN, Caro, C, Carr, AH, Cathey, J, Cating, R, Charron, K, Chavez, S, Chen, J, Chen, JC, Chen, JW, Cheng, C, Cheng, X, Christensen, BE, Church, KS, Cicone, NJ, Maas, W , Geest, JVD, Pelt, VV & et al. 2023, 'The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions?', Issues in Accounting Education ..., vol. 38, no. 4, pp. 81-108. https://doi.org/10.2308/issues-2023-013