 |
 |
 |
 |
 |
- Technical Manual
- Chapter 1: Introduction
- Chapter 2: Theory and Background
- Chapter 3: Administration and Scoring
- Chapter 4: Scores and Interpretation
- Chapter 5: Case Studies
- Overview
- Case Study 1: Clinical Evaluation of an English Speaker Suspected of Having a Speech-Language Impairment
- Case Study 2: Progress Monitoring of an English Learner with Suspected Deficits in Verbal Ability and Language
- Case Study 3: Psychoeducational Evaluation of an English Learner with Suspected Specific Learning Disability
- Chapter 6: Development
- Chapter 7: Standardization
- Chapter 8: Test Standards: Reliability, Validity, and Fairness
- Task Instructions Scripts
Item Generation and Development |
Potential target words were selected using the Merriam-Webster’s Collegiate Dictionary (Merriam-Webster, 2003) and The Educator’s Word Frequency Guide (Zeno, Ivens, Millard, & Duvvuri, 1995). A large number of online word lists, such as the Dolch Sight Words List (Dolch, 1948), English Prepositions List (Essberger, 2012), Frequency Dictionary of American English (Davies, 2009), and grade-level spelling lists (k12reader.com, 2011), also guided the item generation process and were used to determine a preliminary item order based on word frequency and grade-level expectation.
An item pool of 474 target words was selected for the pre-pilot phase. The following criteria guided the target word selection process:
- diverse word categories (e.g., actions, activities/sports, animals/organisms, art, body parts, buildings/places, clothing, colors, concepts, descriptives, emotions, everyday items/miscellaneous objects, food, health terms, household items, math/science terms, occupations, office supplies, toys/musical instruments, transportation),
- a range of difficulty within each word category (from easy to hard vocabulary words based on word frequency and grade-level curricula),
- different parts of speech (i.e., nouns, verbs, adjectives, adverbs, and prepositions), and
- progression from Basic Interpersonal Communicative Skills (BICS) to Cognitive Academic Language Proficiency (CALP; Cummins, 1984) vocabulary, with special consideration given to English learners as they acquire English words.
As mentioned in chapter 2 (Theory and Background, Vocabulary Acquisition and Language Development) and in chapter 4 (Scores and Interpretation, Step 4: Examine Performance by Vocabulary Type), the BICS and CALP distinction is related to word difficulty based on both grade- and age-level expectations. This distinction is also related to the context in which words are learned. BICS words are frequent and simple words that are likely to be encountered in everyday experiences or conversations. CALP words, on the other hand, are typically learned through explicit classroom instruction, reading, or other educational and vocabulary-rich sources (e.g., conversation with educated parents, home environment where reading is encouraged). While BICS words are often used to assess language functioning in terms of day-to-day interactions, CALP words are critical for understanding language functioning in the context of school performance and incidental learning of vocabulary. In order to capture the trajectory of vocabulary acquisition, resources such as Words for Students of English: A Vocabulary Series for ESL were consulted (Rogerson & Menasche, 1988–2004). Within the broad categories of BICS and CALP, words were then selected to represent finer levels of vocabulary acquisition (i.e., Emergent BICS, Intermediate BICS, Advanced BICS, Emergent CALP, Intermediate CALP, and Advanced CALP).
Following the selection of target words, three non-target words (i.e., distractors; incorrect response options) were selected to accompany each target word, with consideration given to semantic or phonetic similarities to the target word, as well as antonyms or contrasting content. Distractors were carefully designed to complement each target word to ensure that the test is challenging and engaging, but not too challenging such that it would render the test items impossible to complete for the appropriate age group (i.e., distractors should be similar enough to the target word in order to appear cohesive [but not to the point of trickery] and to distinguish examinees who have acquired the target word from those who have not).
Stock images were carefully selected to represent the meaning of the 474 target words and the corresponding 1,422 distractor words. Images were thoughtfully chosen to present four response options that were as visually cohesive as possible with respect to color, style, and thematic content. Images that were emotionally charged, potentially distressing, or offensive in content were avoided. Images with the potential to be offensive or biased toward any given cultural group, especially images with a cultural or religious theme (e.g., clothing, flags, landmarks, symbols, and icons), were avoided as well. Efforts were also made to maintain gender balance and ethnic diversity in the images and to ensure that, overall, the images were equally salient to reduce enticement or selection due to an extraneous factor unrelated to the meaning of the target word.
The target words were ordered based on presumed developmental acquisition (i.e., in accordance with the age-based sequence in which they are typically learned). This sequence is akin to word difficulty (i.e., progressing from easiest to hardest), although it is important to note that words are not intrinsically difficult; rather, it reflects the word’s meaning in relation to age-based acquisition and relative to other words. The image for each target word was placed in a 2×2 grid with three corresponding distractor images. All four word images collectively became a single item. Finally, the position of the target word image in each item was randomized, but also controlled to ensure the target word image did not appear in the same position in more than three consecutive items.
The final step in item generation was to create the audio stimuli. Target words were recorded by a professional voice recording artist. A small-scale beta test (N = 62) was conducted on 60 items to identify potential issues with the audio or visual stimuli, item functioning, and user experience design features. Results of this beta test were used to help refine the administration procedure for future development phases.
| << Conceptualization and Initial Planning | Pre-Pilot Phase >> |