When will the adaptive version of the English TrueNorth Speaking be available?
The adaptive version of the English TrueNorth Speaking Test will be available to all clients starting October 14th, 2020. The most significant feature update to this version release is that Part 1 will be-fully adaptive.
Is the current English TrueNorth Speaking Test not already adaptive?
The previously released English TrueNorth Speaking Test has some adaptive characteristics. We select Part 2 open response questions based on what is learned about the test-taker’s ability in Part 1. Some people might also consider the element where Part 1 items increase in difficulty to be a kind of adaptivity. There is an important distinction that we want to convey about what we mean when we say that we now offer a fully-adaptive speaking test. This is a significant accomplishment. We will be selling the first-ever, fully-automated and fully-adaptive speaking test.
I want to make sure I understand what it means for a test to be adaptive. Could you explain it to me like I am 5 (okay so more like 15)?
Let’s say that instead of measuring a person’s speaking ability, I wanted to measure their physical strength. For ease of comparison, let’s say I have identical twins that are equally strong—perfect mirrors of one another: Jake and Mark. For Jake, I’m going to use a fixed-form weightlifting test that’s close in characteristic to our current version. For Mark, I’m going to use an adaptive-form test that’s close in characteristic to the new adaptive form.
First with Jake (again, fixed-form), I start him with one rep at 5 lbs. After each rep, successful or not, I will increase the weight by 5 pounds up to 100 pounds. Jake will end up performing 20 reps within this predetermined or fixed protocol. Regardless of what is observed during this test, I will not change course since it is fixed. Now, it is important to understand that the most important tasks—the ones that will be the most informative in terms of assessing Jake’s strength—will be the ones closest in difficulty to his ability. When the weight is far too easy or far too difficult, it really does not tell me much about what he can do.
The early reps are clearly below Jake’s ability. They all seem similarly effortless—each does not seem to add much in terms of new insight until about round eight at 40 lbs. Here, we start getting important information. Jake can still lift the weight, but it is clear that he has to be engaged. 45 lbs, even more engaged. He is successful, but it is requiring new effort. At 50 lbs, it is clear that his ability is really on display. At 55 lbs, the weight is too much. 60 lbs, no movement, but it is a fixed-form test, so we continue with eight more rep increments—sorry, Jake.
We now return to getting much less insight into Jake’s strength. 85, 90, 95, 100 lbs all look the same. What do we know after each? Not much that we didn’t know before—Jake can’t lift them. Now, we take our detailed notes and determine that Jake’s strength is, you guessed it, more than 50 but less than 55. To recap, Jake attempted 20 lifts for us to find he can lift between 50 and 55 lbs.
Now on to Mark with an adaptive-form test. Instead of starting him at 5 lbs, I’m going to start him at 20 lbs. Past instances of the test have told me that most people are at or above this point. 20 lbs is clearly in Mark’s “can-do range”. Mark lifts it with ease. Based on what I have seen over the thousands of times observing people like Mark take the test, I think the next most-useful weight to observe is twice as much at 40 lbs. Mark lifts the weight, but not as quickly. Experience was right. Watching his newly-focused effort clearly gave me more insight into Mark’s strength. I am not going to keep doubling the weight because we are clearly closer.
However, we will again increase it by a 20 lb increment to 60 lbs. Too high—failure. Let’s cut that last increase in half. 50 lbs. Success, but with clear effort. Getting very close—this is clearly near Mark’s level. Increase to 55 lbs—too much. Back down to 52 lbs—successful, but strained. Increase by to 54 lbs—fail. Decrease to 53 lbs—success. We’ve got it!
Even Mark can feel that we got it. It was certainly a little more dynamic than Jake’s test so let’s review the reps. Mark attempted eight lifts for us to find he can lift 53 pounds.
Not only did Mark attempt less than half the number of lifts that Jake attempted. He also received a more accurate number for what he can lift.
Will the adaptive version change the process for administrators assigning the test?
No. After an administrator has opted in to the version update, there should not be any other significant differences to their experience assigning the test or interpreting the test results. We took great care to ensure that results will be appropriately consistent, despite the update in the versioning. There will be no new costs as to reap the benefits of adaptivity. Administrators wiIl receive test results that are more accurate and reliable and will be able to better use the TrueNorth Test to measure changes in ability that follow time and intervention. Additionally, administrators will hear less from test-takers that the test feels too easy or too difficult and will have increased confidence in results as there are new barriers to cheating.
Will the adaptive version change the test-taker's experience with the test?
We hope so. The test for all test-taker’s will be shorter and more efficient. We will also spend more time testing the person at their ability level and less time far above or below their level. Additionally, the test-taker can retest multiple times and each experience will fit well to the current abilities. However, the instructions for the test do not change.
Could I get a bulleted list of the advantages of an adaptive test?
Efficiency - The test can much more rapidly narrow what needs to be evaluated since the test responds to the information gained from each task. The adaptive version will have, at most, 20 items in Part 1 of the test, but most test takers will have 12-15 items.
Security - Adaptive presents many barriers to fraud. Even recently, we have uncovered new fraud behaviors that the adaptive test will effectively prevent. It is very unlikely that one test-taker could relay to another information about what their test will be like because the test form is dynamically created.
Reliability - Despite a reduction in test length, we are increasing the number of items presented close to the individual’s ability level. We can actually have a more precise and reliable understanding of their ability despite the shortened test.
Test/Retest - A fixed-form test like the current version is much less desirable for testing people multiple times. The adaptive version will give a better experience to people who want to retest applicants for a variety of reasons, but particularly if there is some type of learning that has occurred between the two tests.
What does uncalibrated test content mean?
Uncalibrated test content is test content that has not been presented to sufficient test-takers for us to know what we would need to know to work it into our scoring models. This process of calibration of new test content is vital to us maintaining the health of our item bank. Shortly, we will be doing some feature work that will enable us to add several unscored new test items to a test-taker’s assessment. For their purposes, the test-taker will not know that this content is unscored for their test. Once the test content has been presented to 200 to 300 test-takers, we will have the insight we need to move it into our test bank and have it begin to contribute to the scoring of test forms.
When will other tests become adaptive?
We do not have a timeframe right now. More urgent priority will be given towards getting a released version of a fixed-form test for Italian, Russian, Tagalog, Chinese, and Korean. We will be continuing work in building out our item banks in the released languages so that we are ready when the time works for us to move to adaptivity for other languages.
Additional Documents & Resources
Release Announcement Email to Current Clients
Blog post from Dr. Jacob Burdis, Founder of Emmersion