Administration Procedure

This section provides a step-by-step guide to the administration procedure of the Ortiz PVAT. Test examiners and evaluators should closely adhere to these procedures to maximize the likelihood of obtaining the most reliable and valid results possible.

Before the Administration

Downloading the Ortiz PVAT Application

The Ortiz PVAT application is available for download on a Windows® device via the MHS Online Assessment Portal (refer to Hardware and Software Requirements later in this chapter to ensure the Windows device complies with the requirements for the Ortiz PVAT). Examiners with a valid user account on the Portal who have purchased the Ortiz PVAT assessment will see a Download button on the Ortiz PVAT product page on the Portal. Clicking the Download button will navigate to the Ortiz PVAT product installation page within the Microsoft Store®. No further purchase is necessary on the Microsoft Store. Follow the on-screen instructions to install the Ortiz PVAT application onto the desired local device. There is no limit to the number of devices on which the Ortiz PVAT application can be downloaded (i.e., the application can be downloaded onto multiple Windows devices [e.g., desktop, laptop, tablet] if desired). Once installation is complete, the Ortiz PVAT application can be accessed on the Windows device by selecting the Ortiz PVAT from the list of applications or by selecting the Ortiz PVAT desktop shortcut.

Once the Ortiz PVAT application is downloaded, re-installation is not required for subsequent administrations, although the software may be updated with new features and improvements over time. Notifications of such updates will be communicated via the MHS Online Assessment Portal.

Client Setup

Clients must be set up prior to test administration on the MHS Online Assessment Portal. This procedure allows the examiner to select an existing client on the Portal; the client’s basic demographic information is then preloaded onto the Ortiz PVAT application. If the client’s name is not on the Portal’s client list, a new client can be added to the examiner’s client list on the Portal. To set up a client in the Ortiz PVAT application via the MHS Online Assessment Portal, select the client on the Ortiz PVAT Setup Client screen. Confirm the selected client’s demographic information, which will be displayed on the screen, and enter any missing information in the mandatory fields:

After the client is set up on the MHS Online Assessment Portal, the examiner can choose to set up other clients or proceed to dock clients from the MHS Online Assessment Portal to the Ortiz PVAT application in preparation for administration of the assessment (either immediately or at a later date). The examiner must select which clients will take the Ortiz PVAT, and the list of selected clients and their corresponding demographic information (previously entered on the MHS Online Assessment Portal) will be transferred to the Ortiz PVAT application via the server (an internet connection is required for this to occur). If the examinee is present, the examiner can log out of their Portal user account and close the internet browser to begin administration of the Ortiz PVAT.

Preparing for the Administration

The following steps should always be taken prior to the beginning of the assessment to ensure that administration of the Ortiz PVAT will run smoothly.

Prepare the Device:
Prepare the Environment:
Prepare the Examinee:

During the Administration

Step 1: Launch the Ortiz PVAT and Log In

To begin, open the Ortiz PVAT application. The first page requests a username and password; the login information is the same information needed to access the examiner’s user account on the MHS Online Assessment Portal. An internet connection is required for this step in the administration process.

back to top

Step 2: Complete Client Information

The application lists the clients who were docked from the MHS Online Assessment Portal. If there are no docked clients in this list, the button to launch the assessment will be disabled; clients must be docked in advance. First, confirm that the demographic information (some of which is pre-populated with information entered on the MHS Online Assessment Portal) is correct by confirming with the examinee or the examinee’s parent/guardian. There is additional demographic information with regard to the examinee’s language use and English exposure in the home and at school that is required for scoring the Ortiz PVAT (see Table 3.1 for a list of all demographic questions on the Ortiz PVAT). This information should be completed with the help of a parent or guardian for children who cannot reliably provide this information on their own. If the parent or guardian is not present and the information cannot be gathered at the time of testing, enter “Uncertain” for these demographic questions, but be sure to have the information ready when results will be scored using the MHS Online Assessment Portal.

back to top

Step 3: Choose a Form Type

After the client’s demographic information has been entered, select the form (Form A or Form B) to be administered. If the client has previously been administered Form A and is being re-evaluated to measure progress in vocabulary acquisition, Form B can be used for the follow-up administration. The two forms have equivalent psychometric properties (see Creation of Parallel Forms in chapter 6, Development, for more details on the development of the forms, and see chapter 8, Test Standards: Reliability, Validity, and Fairness, for a comparison of the psychometric properties of the two forms).

back to top

Step 4: Select Administration Type

There is only one mode of administration for the youngest examinees (ranging in age from 2 years 6 months to 5 years 11 months); they will always begin the test at the easiest item (i.e., Test Item 1; target word: “baby”).

For examinees aged 6 years and older, there are two modes of administration on the Ortiz PVAT: Standard Administration (recommended for all administrations) and Custom Administration.

Standard Administration. Recommended for all administrations, the Standard Administration uses the examinee’s performance on Screener items (see Creation of Screener Items in chapter 6, Development, for more details) to determine the appropriate starting point on the test. As the name suggests, the Screener items function as a quick analysis of the examinee’s ability to determine the point at which the test should begin, rather than determining the starting point based solely on their chronological age.

Performance on the Screener is used to establish a baseline, akin to a basal set. The use of Screener items ensures that examinees are only administered items that are critical to measure their receptive ability, which saves administration time. The Screener is used to determine the first item that will be shown to the examinee. There are six possible starting points for the test. Note that once the first item has been presented, the examinee will no longer be able to navigate back to the Screener items. The Back button is disabled once the Screener is complete to prevent the examinee from changing their responses on the Screener items and potentially altering their starting point or invalidating results.

Special Topic: How the Screener Works

There are five sets of Screener items, which allows for six possible starting points on the assessment (i.e., five starting points for successful completion of each Screener set, in addition to the easiest item on the test, Test Item 1, “baby,” in the event that the examinee fails the first Screener set). Examinees are presented Screener items up to the Screener set that is appropriate for their age group. That is, an examinee may start the test at a point lower than their chronological age based on their performance on the Screener; however, an examinee will never start at a point higher than what corresponds to their chronological age. Table 3.2 presents the Screener item sets, the corresponding starting points on the test for each Screener set if all the items in that set were answered correctly, and the ages permitted to attempt each set.

Custom Administration. In the rare case where difficulty is observed with the examinee completing the test at the starting point chosen by the program via Standard Administration or should the examiner wish to administer the test from the easiest item without skipping any items based on Screener performance, the examiner can re-administer the test using the alternate form and bypass the Screener items by selecting Custom Administration. This mode of administration will always begin the test at the first item (target word: “baby”).

back to top

Step 5: Determine Unassisted or Assisted Administration

Unassisted Administration. Most examinees who are 5 years or older should be able to indicate their response (i.e., select a picture with a mouse or touchscreen) and navigate through the assessment on their own without assistance. However, this mode of administration does not imply that the examinee may complete the task without supervision. Remain present at all times to observe examinee behavior, maintain a good flow between items, and ensure that the examinee is engaged in the task. Unassisted task instructions should be used for examinees who clearly demonstrate competency in responding and navigating through different screens on their own.

Assisted Administration. Examinees who have limited motor skills to control the mouse or use a touchscreen (e.g., under 5 years old or due to physical or neurocognitive disability) may require some assistance to input their responses or advance to the next item. In such cases, the examinee can point to or say the number of the picture that they wish to select; the examiner can then input the examinee’s selection and select Next to present the next item.

For children under the age of 5 years, begin the assessment with assisted administration task instructions and switch to unassisted administration only if the examinee has clearly demonstrated their capacity in responding to items and navigating on their own. Some examinees may be able to indicate their responses with touchscreen input, but may still require the examiner’s help or a reminder to select Next to advance to the next item. Always encourage the examinee to advance to the next item on their own (for examinees aged 5 years or older) by selecting Next, unless it is clear that the examinee is not physically able to reach the mouse or touchscreen or if persistent reminders are required. It is permissible to switch to a partially assisted administration where the examinee may click the mouse (or touch the screen) to indicate a response, and the examiner may select Next to help the examinee advance to the next item if they are unable to do so themselves or require excessive prompting.

back to top

Step 6: Administer the Practice Items

Prior to starting the Practice Items, the examinee should be seated in the main chair; the examiner should be seated next to the examinee. For younger examinees, it is permissible for the parent/guardian to stay in the room, as their responses may be required for some of the demographic questions that precede the test. In such cases, the parent should be seated at the far end of the room such that the screen is not visible to the parent as soon as the demographic questions are complete. They should also be instructed not to speak to the child or provide any type of non-verbal feedback (e.g., nodding) if the child seeks confirmation for selections during the session.

There are two sets of Practice Items designed to familiarize the examinee with the task. Each set contains four items, indicated on screen with the tab “Practice Round 1” and “Practice Round 2” in the top right corner (see Figure 3.1). The examiner should familiarize themselves with the task instructions prior to administering the Ortiz PVAT (also see the translated task instructions in Arabic, Chinese, Russian, Spanish, and Vietnamese).

Provide verbal instructions while Practice Item 1 (“apple”; see Figure 3.1) is on the screen. Once the first Practice Item appears on the screen, read one of the following three task instruction scripts aloud to the examinee. The instructions explain the task and how to use the Ortiz PVAT interface. The three instruction scripts vary slightly depending on the method of administration and input device:

  1. Unassisted administration using mouse input
  2. Unassisted administration using touchscreen
  3. Assisted administration

The audio track for Practice Item 1 (target word: “apple,” as seen in Figure 3.1) is only played when the Speaker button is selected to allow the examiner to explain the task. After selecting Next, Practice Item 2 (target word: “tree”) is presented. This item (and all Practice Items that follow) will behave like an actual test administration item; that is, the audio track for each word will play automatically without the need to click the Speaker button.

Following the presentation of Practice Item 4 (target word: “hot”), there will be an option to start the administration or to continue with more practice items.

At the end of Practice Round 2, the option to either start the assessment or exit the test will appear on the screen.

back to top

Step 7: Administer the Assessment

If the examinee is under the age of 6 years, or if Custom Administration was selected, the assessment will begin with the first test item (target word: “baby”). If the examinee is aged 6 years or older and Standard Administration was selected, the assessment will automatically begin with the Screener items. To the examinee, there is no indication that the items are part of a Screener, as these items are seamlessly integrated into the assessment experience. Regardless of the type of administration selected, the assessment will proceed automatically from the correct starting point. As described in Step 5, assistance with entering the examinee’s responses may be required, particularly for younger examinees; otherwise, simply observe the examinee’s behaviors during the administration and ensure an optimal testing experience to support the examinee’s best performance.

The assessment will automatically stop when the examinee’s ceiling has been reached (i.e., when the examinee makes five errors within ten consecutive items or completes the last item on the test). At this point, the assessment has determined the upper limit of the examinee’s English vocabulary acquisition and no further test items are needed to calculate the examinee’s scores. The screen will display the message, “The assessment is complete. Please click Exit.” This message signifies that the assessment is valid and the examinee’s ability has been determined by the ceiling (see chapter 6, Development, for more details on the development of the ceiling rule). At this point, select Exit to close the program. Data will be sent to the MHS Online Assessment Portal for online scoring.

Special Topic: What Happens if the Internet Disconnects During the Assessment?

An internet connection is required to log in to the Ortiz PVAT application to begin an administration with docked client information sent from the Portal.

After the Ortiz PVAT application has been launched, the examinee’s demographic information has been confirmed, and the test has begun, the test will continue to operate on the device where the Ortiz PVAT application was installed, even if the internet connection disconnects. Test data will be stored on the local device in the event of disconnection until an internet connection is re-established, at which point the data can be uploaded to the MHS Online Assessment Portal.
Examiner’s Role

It is good practice to note any atypical behaviors exhibited by the examinee that might influence the results of the test. Below are some examples of events that may arise during an assessment that may be worth noting:

  • Atypical Behavior: During the administration, note any behaviors from the examinee that might influence the validity of the test (e.g., playing with navigation buttons, inattention).

  • Audio Issues: Note any words that the examinee might have had trouble hearing due to the sound quality of the audio file or the computer speakers. Sometimes the examinee might ask to hear the word again or may select the Speaker button to repeat the word. Repetition of the audio track could be due to the examinee’s unfamiliarity with the word, but it could also be related to the sound quality of the speakers. If the word seems to be within the examinee’s vocabulary level, and they seem to have trouble discerning certain syllables, make a note of the word; it may be useful information for intervention.

No coaching or feedback is allowed during test administration as these practices will invalidate the results. It is important not to hint at the correct or incorrect answer, either verbally or nonverbally. The examinee should never be told whether an answer is correct or incorrect. If the examinee asks whether a response is correct, say, “Pick the one that you think is best.”

The examiner should not:

  • repeat the target word by saying it out loud (the Speaker button is to be used at all times if the examinee wishes to hear the word again),

  • repeat the target word in the examinee’s native language,

  • use the target word in a sentence,

  • define or spell the target word, or

  • provide a synonym for the target word.

If necessary, the examinee’s attention can be refocused by saying, “Listen carefully.” With younger children who require the parent or guardian to be in the testing room, the parent or guardian must be instructed not to provide verbal or nonverbal hints to any items.

Visual Prompts for Examinee Engagement

To keep the examinee engaged during the task and to ensure that the examinee does not linger on any given item for too long, the Speaker button will blink once in orange if the item has been displayed for 10 seconds without a response from the examinee. This change in the Speaker button’s appearance acts as a visual reminder for the examiner to select the Speaker button to replay the audio track of the target word in order to prompt the examinee for a response. After an item has been displayed for 15 seconds, the Next button will turn orange. If the examinee has selected a word, say, “Click Next when you are ready to move onto the next word.” For the remaining items, consider selecting the Next button for the examinee if they continue to indicate the answer without clicking the Next button independently.

Omitted Items

Even though omitted items are allowed, examinees should be encouraged to attempt all items as there is no penalty for guessing. If an examinee does not respond within a reasonable time (approximately 15 seconds after the item is displayed and the Next button turns orange), encourage the examinee to take their best guess. Note that the time period suggested is only a general guideline, as reaction time is not a factor in scoring the Ortiz PVAT. If necessary, the examinee is permitted to spend more than 15 seconds per item; however, the main purpose is to keep the examinee engaged during the task rather than focusing on a particular word that they do not know.

Given the impact of omitted items on Ortiz PVAT scores (i.e., omitted items are scored as incorrect responses), it is strongly advised that examinees attempt every item presented in order to obtain an accurate estimate of their receptive vocabulary ability. Sound judgment must be used to determine when it is appropriate to advance past items that the examinee is uncertain about answering. The following script can be used to encourage the examinee to guess, rather than omitting an item:

“If you are not sure, just click on the one that you think is best. You can always listen to the word again by pressing the Speaker button or asking me to do so.”

If the examinee still does not provide a response, say:

“It is okay to make a choice even if you are not sure. Do you want to try your best to guess, or move on to the next question?”

If the examinee wishes to omit an item and chooses not to guess, the examinee (or examiner) should select the Next button and confirm in the pop-up window that they wish to leave the current item blank. The omission will be recorded as an incorrect response and the test will continue with the next item.

Examinee Disengagement and Common Questions

There are several behaviors that may indicate that the examinee has become disengaged with the assessment. It is important to note any irregularities during the administration as they may have an impact on the examinee’s focus when a break is taken.

To ensure that the examinee is engaged with the task and is not discouraged, it is permissible to provide intermittent positive reinforcement if necessary (e.g., “Good job, keep going!” regardless of the response they select). Such encouragement may be required when administering the test to younger children or individuals who appear bored. Tone of voice and wording should always be positive, regardless of whether or not the examinee responds correctly.

If the examinee starts asking questions during the administration (e.g., “Am I almost done?”), respond by saying, “I can’t answer your questions during the task. Just try the best you can to keep going.” If the examinee cannot go on (i.e., they have disengaged from the task and cannot be re-engaged), consider terminating the assessment (see Terminating the Assessment later in this section). Results from such administrations cannot be considered valid.

If the examinee asks to go the bathroom, ask if they are able to wait: “As soon as this is over, I will bring you to the bathroom. Do you think you can wait until this is over?” Redirect the examinee’s attention back to the test as quickly as possible. If they insist, allow them to go to the bathroom. The test is untimed and does not have a time limit for completion; however, if the test has been idle for over 90 minutes, the test will terminate and a message will appear on the screen to instruct the examiner to close the Ortiz PVAT application. The assessment should be considered invalid. No test data will be retained in the application and the examiner must re-administer the entire assessment. It is recommended that the alternate Ortiz PVAT form be used for re-administration (e.g., Form B should be used if Form A was the form originally administered and was the form that timed out).

Examinee Exclusively Focuses on the Same Quadrant or Side of the Screen

If the examinee appears to only focus on the same quadrant or one side of the screen (e.g., choosing the image in the same position repeatedly or seemingly ignoring a particular section of the screen), it may be necessary to say, “Make sure you look carefully at all the pictures before choosing your answer.” If this behavior continues to be a problem, point to each of the pictures on the screen and say, “Look at this picture, this one, this one, and this one.” Note that this tendency to focus on a portion of the screen may occur more frequently with very young children or when the examinee is approaching their ceiling.

Examinee Misuses Application Interface

During unassisted administrations, the examiner should discourage excessive use of the Speaker button to replay the audio track for words or the Back button to go back and forth between items.

  • Speaker Button: If an examinee does not seem to be having trouble hearing a particular word and seems to be selecting the Speaker button repeatedly (i.e., more times than what one might expect to be necessary) just to replay the audio track for fun, discourage this behavior and ask the examinee to only use the Speaker button when they have trouble hearing the word.
  • Back Button: Examinees are permitted to go back and change a previous answer. However, the examinee should not be allowed to keep moving back and forth between items just to play with the navigation buttons.

Exercise sound judgment to determine typical or atypical use of the application’s buttons and other features of the interface or hardware (e.g., mouse, keyboard). If the examinee plays with the buttons persistently and excessively, consider terminating the assessment.

Terminating the Assessment

The test may be terminated in case of examinee fatigue or any other circumstances that make them unable to continue (e.g., examinee requests to stop the test or appears distressed at any given point during testing). In such cases, it is possible to end the assessment before the ceiling has been reached (i.e., prior to the presentation of the on-screen message, “The assessment is complete. Please click Exit.”). Close the application using one of the following methods (depending on the device and operating system):

  • select the “X” in the upper right corner of the application window using a mouse or a touchscreen device,

  • swipe down from the top of the screen on a touchscreen device, or

  • press Alt+F4 on a keyboard.

A pop-up window will appear to confirm that the examiner wishes to terminate the assessment. Results from this administration are considered invalid and data from this administration will not be saved. If possible, re-administer the assessment using the alternate form at the next available opportunity.

back to top

After the Administration

At the end of an Ortiz PVAT administration, thank and debrief (see Ethical and Legal Guidelines earlier in this chapter) the examinee. Avoid making any premature interpretations. Proper interpretation requires a thorough analysis of the results and, as such, generally cannot be done immediately after an administration.