Surgent Boosted Their CPA Candidates’ Exam Pass Rate by 38% with Adaptive Personalized Learning

Many companies today make claims about how their adaptive algorithms personalize the learning experience. Very often, these claims come from the marketing department, rather than the research team. One company, however, has quietly developed some of the most advanced personalized learning courseware currently available. What’s more, it has the results to back up their claims.

Surgent CPA Review, which was founded in 2014 by Liz Kolar (CPA, CGMA) and Jack Surgent (CPA and founder of Surgent CPE), says it can determine, within four points, what a CPA candidate will score on the Uniform Certified Public Accountant (CPA) Exam, before he or she ever walks into a testing center. “And it’s important to note that those four points are in the student’s favor, on average” Kolar said. “Meaning that, if a student is scoring a 75 in our course, their score on the actual exam tends to be a 79.”

Accurately predicting a candidate’s exam score, of course, is not as important as getting that candidate to pass. On this front, the company has achieved significant results. On a national average, roughly half of the CPA candidates who take the exam fail. Among those who train with Surgent, however, 88% pass.

Inspired by a Shift in Student Learning Styles

Kolar has been preparing CPA candidates for the exam for over 30 years. She’s taught at the undergraduate and graduate level at Pace University, Seton Hall University, and currently doubles as a professor at Delaware Valley University. It was during these many years of direct student interaction that she began to realize that CPA Exam preparation needed to evolve.

“Back around 2007 or 2008, I noticed a change in my students,” Kolar said. “I was teaching four nights a week, four hours a night. I was using primarily a lecture approach to prepare my students, and I saw a shift in their ability to attend class. Their employers were placing greater demands on their work schedules. They wanted the course to work for them on their time. Also, their attention spans had shortened and they were looking for more of a customized approach.

Kolar took her course online and broke most of her lectures into short five-to-ten minute videos.

“It was quite amazing—about 70% of my students left the classroom and chose to take my course online,” she said. “So I knew I had struck gold.”

Noticing the success of Kolar’s online course, Surgent approached her with an interest to partner. They agreed that it would be ideal to incorporate an AI component in their courses. So in 2013, the two pooled their resources and set about designing the courseware.

Designing Adaptive Courseware for an Elusive Exam

Developing the adaptive algorithm was not a simple project, considering the structure of the CPA Exam. It’s broken into four sections, and candidate scores are compiled on a statistical scale ranging from 0 to 99. Candidates must clear a score of 75 to pass. Each of the four sections is comprised of multiple choice questions and simulations, which ask candidates to respond to hypothetical situations and tests them on a range of issues. (The Business Environment and Concepts section also contains a written component.)

The multiple choice section doesn’t present too many issues when it comes to designing adaptive courseware in the sense that each question is objectively correct or incorrect. The simulations, however, are more tricky. They involve a higher degree of complexity, and the American Institute of Certified Public Accountants (AICPA) is tightlipped about their process.

“The AICPA will not tell us exactly how they score the simulations,” Kolar said. “They have given some data to us. Think of it this way: there is a table that has two columns and ten answers, so 20 answers in total. Each cell in the answer has value. A student can receive partial credit. So there’s one row and two answers a student has to give. They could get one wrong and one right, and they’ll receive partial credit.”

The Surgent developers, in other words, were put in a difficult situation. They had to design AI to replicate a process that they themselves did not fully understand. Luckily, they had some solid HI (human intelligence) to lean on.

A Change in the CPA Exam Prep Landscape

The team combined the “nano” content that Kolar had begun testing a few years earlier with a new adaptive algorithm and introduced the resulting product as the first adaptive CPA Exam review course to market. The company has continued to focus on refining its technology since, implementing features like continuous individual assessment and a candidate score predictor, which they call ReadySCORE.

“Surgent’s A.S.A.P. Technology is built on the mantra that no two candidates are the same. To us, achieving maximum efficiency and success requires treating candidates as unique, three-dimensional people, rather than static stencils,” said Phil Salgado, Surgent VP of Product Management and User Experience (UX). “To accomplish this objective, we created a proprietary ReadySCORE, which is a real-time approximation of the candidate’s score on the exam, as if they were testing today.”

“Our software is designed to utilize the ReadySCORE, not just as helpful feedback to the candidate, but also to build personalized study sessions comprised of multiple choice questions, simulations, videos, and readings from our database. All of these target the candidate’s weaknesses and reinforce their strengths.”

By comparison, other CPA Exam preparation courses on the market at the time focused on a “linear” learning model that is very similar to the type of in-person classroom-based courses that had been popular for decades. These types of courses guide students through the course’s lectures, books, and practice questions, chapter-by-chapter, in order, until the student reaches the end of the course. Many of these courses now also market an adaptive learning element.

“Our competitors tout their adaptiveness, but we’ve looked at them closely. Their software is really just designed to guide candidates down fairly defined paths based on user personas, rather than being uniquely tailored to each user,” said Salgado . “Further, because these user personas are created via a backward or historical look at previous user paths, there are limits to their predictive capabilities as the CPA Exam evolves over time (i.e. whenever the exam changes, these courses will be far less reliable until they have sufficient data to adjust the personas). Additionally, once users have been assigned personas, they do not change, so these courses fail to account for candidate growth factors like area of employment, time since graduation, and learning efficiencies.”

Focusing Development on Saving Study Time

The company has continued to invest in the development of their AI, using user experience and performance data pulled from its LMS to guide development decisions. One such recent development has been a difficulty progression feature. It allows candidates to choose various levels of difficulty as they continue through their training, in essence, choosing how aggressively they want to push themselves to achieve exam-readiness.

“Our data shows us that students selecting the most aggressive path are completing the course 50% faster compared to those choosing a less difficult path,” Salgado said. “In fact, they’re completing the entire CPA program, while achieving the same results, in the time it would take to complete a single section in less adaptive courses.”

Regarding their impressive 88% pass rate, Salgado is quick to point out that Surgent doesn’t cook their data. “Surgent surveys all users as soon as they pass their exam date to gather their actual score, not simply whether they pass or not. We compare their actual scores to their ReadySCORE at the time they sat. That’s how we know that we’re accurately predicting their real exam scores,” he said. “Further, all users are included in our pass rate calculation. To date, we have received score reports from approximately 20% of our user population, which is a statistically significant sample size. The pass rates have been remarkably consistent across testing windows over the years, despite our continued growth.”

 

Looking Beyond the CPA Exam

Surgent is not just ahead of the pack when it comes to CPA Exam preparation. The company has already applied their courseware to other accounting license exams, including those required for Certified Management Accountants, Certified Internal Auditors, and Certified Information Systems Auditors, with more on the horizon. Numerous other professional fields that require licensing lack this technology altogether, and Kolar sees a huge potential to scale and grow.

“My feeling is that anywhere you have a body of knowledge that is tested for some kind of certification or licensure, our adaptive technology can work,” she said. “And if there’s one thing in common among professionals trying to pass certification exams in all kinds of areas, it’s that they want to do so with the least disruption possible to their busy lives. That’s why I’m confident that our adaptive technology–which can save literally hundreds of hours of study time per student–will catch on everywhere we introduce it.”