Our Commitment to Quality
How is NEWL Developed?
NEWL measures students’ functional proficiency in one of four target languages. NEWL results can be used for predicting success in language learning beyond high school study. The assessment does not assess specific curricular achievement or the accumulation of grammatical, literary, or historical knowledge. Exam specifications reflect the most recent draft of the World Languages Framework (College Board, 2019).
Quality Assurance
NEWL is administered via American Councils’ Language Assessment Support System (ACLASS), which supports the administration of approximately 1,000 examinees annually. The platform facilitates two-way communication between the American Councils administration and the many field experts who develop NEWL. The platform enables on-line item and exam development, data collection, scoring, and certification production. All exam forms are stored in the on-line system for quick assembly, sharing, analysis and quality control.
ACLASS enables standardized exam conditions for all: Examinees take the same exam, administered under the same conditions and protocols, across various locales, via an internet connection. The ACLASS system also allows for efficient examinee registration and real-time monitoring during exam administration through a chat navigator. Customization of exams is also possible when an examinee has unique circumstances such as a need for make-up testing or special needs accommodations.
In sum, the ACLASS computer-based testing system supports NEWL through:
- Collaborative exam development
- Workflow management system
- Online examinee registration
- Easy exam center set-up and proctor certification
- Real time, on-line exam proctoring and communication
- Online constructed response scoring platform (writing and speaking tasks)
- Efficient score reporting and data organization
Exam Development
NEWL assesses four language skills: reading, listening, writing, and speaking. The reading and listening comprehension sections consist of all multiple-choice items; writing and speaking sections have open-ended response tasks. The content development team consists of five assessment specialists and a broad network of professional linguists, exam item writers, editors, and cultural experts. The full time American Councils assessment team has expertise in second language acquisition, assessment design, foreign language standards, item and exam development, exam administration, psychometrics, and statistics. See the Key Personnel section below for more information about the assessment team at American Councils.
Peer Review in Item Development
American Councils’ Quality Assurance starts with a rigorous peer review of all exam items in their development phase. American Councils employs detailed protocols and assigns all personnel clear roles in the item development process. Trained specialists analyze and edit all passages, audios, and their accompanying translations and/or transcripts, as well as their items, through a process that requires several rounds of exchanges between Target Language Experts, Item Writers, Item Editors, and American Councils’ Item Development Managers. The stages of American Councils’ item development process are outlined below in more detail.
Stage One: Passage and Audio Selection for Reading and Listening Sections
American Councils selects culturally authentic reading materials and listening audios for NEWL that are level-appropriate and meet specific content and quality criteria. “Culturally authentic” means that the passages and audios are actual open-source media, taken from sources in the target culture and meant for native speakers. Target Language Experts align them to the NEWL proficiency scale to ensure adequate coverage at all assessed performance levels. A different group of language experts conduct multiple rounds of language reviews to ensure the appropriate levels of the passages and audios as well as the accuracy of their English translations. In the final stage of passage or audio selection, a program manager confirms the passage level and quality - or brings in an additional expert to review if needed - before forwarding the materials to an exam item writer.
Stage Two: Item Writing
Item writers submit their work on a rolling basis according to targets in an item development work plan. Once items are submitted, item development managers review and ensure the appropriate number of items for each passage or audio based on their content and level. Managers also examine the structure and parts of the submitted items and send any problematic items to editors for further review and revision. Once the items are revised by editors, managers ensure that the item sets need no further editing before they are sent to a different expert for a blind review process. A blind review is when the reviewer sees the passage and items exactly as a student would, with no additional information about the item’s history, edits, or expected answer key.
A different expert who has not participated in the development of that item set provides blind review to ensure that it is level appropriate, no distractors are correct answers, the item cannot be answered correctly through background knowledge, and the item set meets other NEWL quality criteria. Managers then check the blind review comments and send the item set back to the original target language expert or editor for further reviews, if necessary. Once completed, an item development manager reviews and approves the item set in its entirety.
Stage Three: Psychometric Review
An American Councils Psychometrician conducts psychometric analyses on the exam items and NEWL test forms. Quantitative item statistics are calculated from the raw examinee item response data to confirm that (1) items, keys, and distractors are functioning appropriately, (2) the items discriminate between high and low performers, and (3) there is no evidence of differential item functioning (DIF), which may introduce bias between sexes and between heritage and non-heritage students. If psychometric quality criteria are not met, items are flagged for further review and revision.
Personnel Performance Monitoring
American Councils ensures item quality through rigorous performance monitoring of target language experts and item writers. We allocate a larger proportion of item development tasks to high performers and train or releases poor performers. Through this process, American Councils also identifies the specific strengths of each expert, such as a particular talent for item development at a lower or higher proficiency level.
Secure Administration
NEWL is administered by trained proctors in computer labs or school classrooms in the United States and abroad. On exam days, proctors are required to sign-in and sign-out each of their examinees by entering their proctor information at each student’s computer. American Councils provides detailed directions on how to administer NEWL through a Proctor Instruction Manual as well as a detailed script to be read aloud to examinees on the day of administration. American Councils also provides online monitoring support for all exam administrations and an emergency telephone number for students experiencing connectivity or login issues.
Language Proficiency and NEWL Scoring
NEWL is a criterion referenced exam developed fundamentally on the ILR proficiency levels. Empirical data are used in the standard-setting sessions to determine the cut scores for each examined proficiency levels on the reading and listening sections. The writing and speaking sections are rated holistically by trained language specialists based upon the rubrics. An estimate of inter-rater reliability across raters is calculated to ensure acceptable consistency across raters.
American Councils developed NEWL to be comparable to the standards of the College Board AP Exams. Because high school teachers are not familiar with the ILR scale, a NEWL score equivalency table with comparable scores across ILR, CEFR, and ACTFL levels are provided with the NEWL score report. In most universities, scores of 5/4/3/2/1 on NEWL earn the same credit and/or placement as the same score on an AP exam in a World Language or any other subject.
For further reference, see the ACTFL proficiency guidelines here.
