Digital brain | ChatGPT
Article | Innovation

ChatGPT’s latest test: Accounting

When OpenAI’s ChatGPT was released for public use, it prompted people to do two things:

  1. Test its ability to generate information around a topic of interest or start a conversation to experience its natural language capabilities
  2. See how it stacks up against the knowledge of humans

Over the past few months, we have seen stories of these comparisons, including ChatGPT passing a Wharton Business School MBA exam [1] and the Uniform Bar Examination [2]. The latest news of ChatGPT taking on an exam designed for humans hit the accounting world when a global team led out of Brigham Young University administered a test on various accounting topics. Overall, ChatGPT was outperformed by its collegiate counterparts – 47.4% to 76.7%, respectively [3].

It is worth noting some performance results from the study, which featured over 27,000 questions input to ChatGPT, highlighted the strengths of ChatGPT and where we will continue to see improvement. ChatGPT did well on questions regarding accounting information systems and auditing, as well as questions in true/false and multiple-choice formats. Conversely, ChatGPT struggled with tax and financial topics and was not as successful when responding to short-answer questions.

What does this mean for the accounting profession?

The short answer is, for now, we really don’t know. This experiment was conducted with ChatGPT, OpenAI’s GPT-3.5. The company has since released GPT-4, which has already demonstrated significantly more capabilities than its predecessor. We have not yet seen GPT-4 applied in a similar fashion to demonstrate accounting acumen. However, based on this academic experiment, GPT-4 will need to improve responses for certain topics and in particular formats. Considering the rate of development within the AI space, it likely won’t be long before a similar experiment is conducted in the accounting industry and we see how the latest GPT fares.

It is clear companies within the industry need to continue to monitor the advancements of AI and find ways to integrate AI technologies into their accounting services. Uses for increased efficiency and automation feel inevitable, and Baker Tilly remains committed to being at the forefront of AI in the accounting and professional services space to understand how we can better perform our services and effectively bring value to our clients.

Abstract data

Fact or fiction: The bias in AI

Artificial intelligence has the potential to revolutionize how we live and work, but bias can manifest in AI systems. It’s imperative for organizations to recognize the limitations of ANI and implement effective governance measures to mitigate bias.

Close up of woman with data surrounding her

AI on main street

The power of AI technology has reached the mainstream user base, pressuring organizations to understand how tools like ChatGPT will change their processes and business activities.

Team members work together on technology innovation

Innovation culture

Keeping a pulse on current and emerging trends helps us navigate the potential impacts and opportunities for our clients and our firm. 

Risks and controls RPA solutions
Next up

Identifying risks and controls when implementing RPA solutions in a SOX environment