Duolingo English Test Logs 1500% Growth During COVID-19

Before the outbreak of COVID-19, some edtech and online education providers were simply better-positioned to serve a remote student base than incumbents. Duolingo was one of these. The Duolingo English Test (DET), launched in 2014, is delivered remotely. It costs $49 and takes roughly one hour to complete (significantly less than market incumbents in both categories). As of the fall 2020 semester, over 2,500 institutions around the world now accept it. Over two-thirds of these are located in the U.S. So far this year, Duolingo has witnessed more than 1500% growth in overall test-takers.

The Duolingo English Test has also grown in Canada and the U.K. This year, the former has increased DET-accepting institutions from 26 to 187, while the former has expanded from 18 to 144.

Despite this success, the DET also has its critics. Temple University Professor Elvis Wagner published an assessment of the Duolingo English Test in 2015. Wagner updated this review in June. He makes several points, but his main criticism is that the DET does not assess learners on the terms in which they will be asked to use English as university students.

Dozens of English proficiency tests are available around the world. Some can be taken for free. But the market is dominated by two players. The Test of English as a Foreign Language (TOEFL), which is administered by the American non-profit ETS, is accepted at over 11,000 institutions around the world. The International English Language Testing System (IELTS) is administered jointly by the British Council, IDP Education, and Cambridge Assessment English. It is accepted throughout U.K. universities, former British Commonwealth countries, and over 3,400 institutions in the U.S. These tests range in cost between $170 and over $250. Both require roughly three hours to complete.

The Duolingo English Test Was Launched to Make English Proficiency Testing More Accessible

For many learners around the world, the demands of taking an English proficiency exam form a significant barrier to entry.

“I grew up in Guatemala and I always had this idea in my head that if I wanted the best education in the world, I should go study in the United States,” said Duolingo CEO and co-founder Luis von Ahn. “But when it came time to take an English test, there were no seats available at the testing center in Guatemala City. In order to complete my college applications on time, I had to fly to El Salvador to take the TOEFL exam.

“We created the Duolingo English Test to make English testing more convenient and affordable,” von Ahn continued. “Our goal is to use technology to help break down barriers to higher education and make it easier for international students to prove their English proficiency.”

An End-to-End AI System

Duolingo has, in part, recouped expenses on two fronts. First, they do not need to maintain in-person test centers. Second, the test is completely scored—and even generated—by an artificial intelligence (AI) system.

“The Duolingo English Test is the first and only high-stakes test to employ Artificial Intelligence (AI) from end to end — including generating test items, administering them, scoring the test, and securing each test session,” said Duolingo Research Director Burr Settles.

“Regarding security, a combination of rigorous protocols, artificial intelligence, and remote proctoring ensure the integrity of each test,” Settles added. “Each test session is recorded via the computer’s camera, microphone, screen, keyboard, and mouse for review by AI and proctors. The test is adaptive and draws from an extremely large pool of test questions, so every test session is unique. Test lockdown prevents test takers from accessing other applications or plugins during the test session.”

https://twitter.com/linoamizo/status/1313318282313703425

While many have concerns about an AI system monitoring and securing a test, the method has become common practice amid remote proctoring and other tests, especially after the outbreak of COVID-19.

As with other AI systems, however, there is always the concern that some form of bias will encroach into the test and favor certain test-takers over others. Settles readily acknowledges the potential issue and says that Duolingo has been taking preemptive measures to ensure it does not occur. A commissioned investigation published in February found no evidence of statistical biases.

“We’re continuing to research any biases (or “differential item functioning” as some say) in our item types to ensure that our test can fairly meet the needs of test-takers from all walks of life,” Settles said. “Furthermore, we are also spearheading research into how to mitigate bias in machine scoring from the start, for example by augmenting the training data in ways that are fair and balanced among different groups (gender, age, ethnicity, etc.).”

The Question of Validity

Some English language proficiency experts maintain a skepticism about the validity of the Duolingo English Test. Temple University Professor Elvis Wagner summarizes his main critique by writing, “the characteristics of the DET test tasks have little in common with the characteristics of language tasks that university students are commonly expected to perform.”

Duolingo Chief of Assessment Alina von Davier pushes back against this judgment.

“As a digital-first product, the Duolingo English Test is always in development and will continue to evolve and improve over time. We welcome and listen to all feedback from the research community and we address them,” von Davier said. “Our research shows that Duolingo English Test scores are highly correlated with TOEFL and IELTS test scores—at the same level the two tests correlate with each other, and that the reliability of measurement is high both for the total score and for the subscores.”

International Adoption

The Duolingo English Test has gained significant ground during 2020 in terms of the number of institutions that accept it. But it remains to be seen how this will translate into adoption by test-takers. Over 3 million learners took the IELTS in 2017, while roughly 2.3 million take the TOEFL.

The Pearson English Language Test, in comparison, is accepted in thousands of institutions around the world and throughout U.K., Australian, and New Zealand universities. But Pearson does not publish how many learners take their test each year.

Duolingo has exhibited strong growth throughout 2020. But it remains to be seen how the dust will settle in the English proficiency testing market.

Media courtesy of Duolingo.