Download - Massachusetts Department of Education

MASSACHUSETTS DEPARTMENT OF ELEMENTARY
AND SECONDARY EDUCATION
ADULT AND COMMUNITY LEARNING SERVICES
ASSESSMENT POLICIES
FOR USING
The BEST Plus
January 2015
Jane Schwerdtfeger, ACLS
TABLE OF CONTENTS
General Overview of Learning Gains Assessments........................................................................ 3
Overview of the BEST Plus ............................................................................................................ 4
Before Administering the BEST Plus ............................................................................................. 4
Required Staff Training .............................................................................................................. 4
Initial Test Administrator Certification Process ......................................................................... 4
Annual Recertification ................................................................................................................ 6
Required Test Materials .............................................................................................................. 7
When to use the BEST Plus, when to use the TABE CLAS-E Writing Assessment ................. 8
Student Placement and When to Administer .............................................................................. 8
Student Primary Assessment Area May Change in Fiscal Year ................................................. 8
Programs Enrolling Students after April 1 may Receive Credit for Post Tests and Learning
Gains, under Certain Conditions ................................................................................................. 8
Test Accommodations ................................................................................................................ 9
Pre- and Post-Testing Interval Varies by Intensity of Class Hours ............................................ 9
Co-enrolled Learners .................................................................................................................. 9
Cognos Report of all Co-enrolled Students at a Site ................................................................ 10
To Receive a Cognos Account .................................................................................................. 10
Test Security ............................................................................................................................. 11
During BEST Plus Administration ............................................................................................... 11
Who May Administer ............................................................................................................... 11
Following Test Administration Procedures .............................................................................. 11
Using Assessment Reports for Tracking When to Post-Test .................................................... 11
Test Conditions ......................................................................................................................... 12
Recalibrate Every Time Before You Test ................................................................................. 12
After Administering the BEST Plus Test...................................................................................... 13
BEST Plus Scoring Monitor ..................................................................................................... 13
Score Reporting and Exit Criteria for NRS Advanced ESL Students ...................................... 13
ESOL Speaking and Listening Correlations using the BEST Plus Test ................................... 13
Measuring Learner Gains .......................................................................................................... 13
Copying Over BEST Plus Scores to the Next Fiscal Year ....................................................... 14
Questions....................................................................................................................................... 14
Appendix A: Massachusetts' Procedures for Using the BEST Plus Scoring Refresher Toolkit... 15
1. The Scoring Refresher .......................................................................................................... 15
2. The Scoring Activity for Recertification .............................................................................. 16
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
2
General Overview of Learning Gains Assessments
The Massachusetts Department of Elementary and Secondary Education (ESE) is mandated by
the US Department of Education to use valid and reliable assessments to report students’
completion of educational functioning levels. ACLS requires programs use four ABE
standardized assessments, depending on the classes they offer: the Massachusetts Adult
Proficiency Test (MAPT), the TABE Forms 9/10, Best Plus, and the TABE Clas-E.
Massachusetts’s process for using required assessments for measuring learning gains:
1. Learning gains are based on the first test and the last test given in a fiscal year
2. Class Placement (after intake, using any instrument(s) other than MAPT)
3. Pre-test (within 2-4 weeks of class placement)
4. Formative assessment using teacher-made or other assessments during class to
determine if students mastered what was taught (ongoing)
5. Optional Mid-year Test (after 65 hours of instruction; ABE Programs for Incarcerated
Adults may test students after 40 hours of instruction; test up to three times/year)
6. Post-test (after 65 hours of instruction, and on or before June 30. Programs may test up
to three times/year)
7. All assessments administered must be entered into SMARTT.
Required Assessments
Skills tested
MAPT
ABE Reading and Math
ABE Writing (Language subtest)
ABE Reading and Math
Literacy Level Reading
ESOL Speaking and Listening
ESOL Writing
ESOL Reading
ESOL
TABE 9/10, Levels E, M, D, A
TABE 9/10, Level L
BEST Plus
TABE Clas-E Writing
Assessment only
TABE Clas-E Reading
Assessment only
Which Programs Use
ABE
ABE
ABE Programs in Correctional
Institutions
ABE learners below GLE 2
ESOL
ESOL
For assessments to be accurate, they must be administered and scored according to test
developers’ instructions. If staff make what may seem like small changes to test administration
or scoring (such as giving test directions differently or diverging from a test’s rubric when
scoring), test scores lose their accuracy. These changes affect the instruction learners may need
and slows them down in achieving their goals. It also adversely affects the accuracy of statewide
scores that ACLS uses to base annual state projections of learning gains for US DOE.
Formative assessments are crucial to instruction and learner gains. They include authentic,
teacher-made, task-based activities and products, etc. They test various skills determined by the
teacher’s learning objectives and should include content and skills from the Massachusetts ABE
Curriculum Framework for ESOL and the College and Career Readiness Standards for Adult
Education (CCRSAE). Formative assessments are important because they indicate what students
learned (and did not learn) and they guide what to teach next. They also involve students in the
learning process, which improves motivation, persistence, and retention.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
3
Overview of the BEST Plus
The state’s assessment policy requires programs to use the BEST Plus test (computer-adaptive
version) for the following learners enrolled in ESOL classes:
 ESOL students who enter at SPL 0-6 and exit at SPL 7, and whose goal is to improve
their oral proficiency in English.
The following programs may choose to use either the TABE Clas-E Reading or Writing
Assessment or BEST Plus, depending on their learners’ goals:
 Workplace Education (ESOL) Programs
 Distance Learning (ESOL) Programs
 Students enrolled in Pre-Literacy ESOL classes, Levels 1, 2, and 3
The BEST Plus, developed by the Center for Applied Linguistics (CAL), is an individually
administered, scripted oral interview designed to assess English language learners’ interpersonal
communication using everyday language. BEST Plus is a performance-based test that measures
an ESOL learner’s oral skills in three areas: Listening Comprehension, Language Complexity,
and Communication.
In the computer-adaptive version of the BEST Plus, test items are delivered via a computer with
a BEST Plus CD. The test administrator asks the examinee a question that is presented to the
administrator on the computer screen, listens to the examinee’s response, uses a rubric to
determine the score for the item, and then enters the score into the computer. (The examinee
does not operate the computer). Based on that score, the computer determines the difficulty level
of the next question. As a result, the questions are appropriate for the learner’s ability and this
allows for the fewest number of test questions to accurately determine the student’s overall test
score and proficiency level. Test time averages 10-15 minutes, depending on the proficiency
level of the examinee. For more information, please see the BEST Plus website at
http://www.cal.org/aea/bestplus/ca.html.
Before Administering the BEST Plus
Required Staff Training
A minimum of two staff per program must attend and a six-hour BEST Plus initial training
offered by SABES (System for Adult Basic Education Support) and successfully pass the related
requirements before any testing of students can begin. Programs offering ESOL classes must
maintain at least two trained BEST Plus test administrators at all times.
 Trained staff may not train fellow staff members at their programs.
Initial Test Administrator Certification Process
In order to receive Competency status and be able to administer the BEST Plus, the practitioner
must first attend a six-hour BEST Plus training. Directly following the training, she/he must be
approved to continue the certification process by the BEST Plus initial trainer and Joan Ford,
SABES director of assessment. Participants will be contacted within one week regarding their
status. They must then complete the following steps in order to become fully certified.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
4
1. To ensure alignment with the BEST Plus scoring rubric, the trainee must complete a
Scoring Activity (to be determined each year) using a copy of the score sheet in the
BEST Plus Refresher Workbook, Test Administrator’s Edition, and the red CD entitled
“BEST Plus Scoring Activities” from the Scoring Refresher Toolkit. The practitioner
must do the Scoring Activity with no help or discussion from others.
2. To become comfortable with actual assessment administration, the trainee must perform
10 practice test administrations using the black CD entitled “BEST Plus Administrator
Practice CD” from the Test Administration Guide. At least 5 of the practice
administrations must be done with non-native English speaking students; the remaining
number may be done with colleagues, family, or friends. None of the practice test
administrations may be used for a student’s actual pre- or post-test. Also, the practice
administrations may not be given to students before an actual testing session. Trainees
must fill out the Proficiency Development Form with the names of the 10 people tested,
date it, and have it signed by his/her program director or ESOL assessment coordinator.
The Scoring Activity and the Proficiency Development Form, with program director's signature,
should be sent to Joan Ford by mail, email, or fax within three weeks of receiving initial
approval. Joan Ford may be best reached at her email address: [email protected] Her
address is Bristol Community College, 777 Elsbree Street, Building Q, Fall River, MA 02720.
If a test administrator did not turn in a scoring activity by the deadline, he/she will not meet the
requirements for certification and may not administer the BEST PLUS Assessment.
If the practitioner passes the initial Scoring Activity and performs the 10 practice test
administrations, she/he will receive Competency status and be certified to administer BEST Plus
assessments (no certificate will be given). Practitioners may now purchase BEST Plus test
administrations. If the practitioner does not pass the Scoring Activity, Joan Ford will make
contact regarding remediation within two weeks.
A “passing score” for the Scoring Activity is determined by the accuracy with which the
practitioner uses each of the three sections of the BEST Plus scoring rubric, which assesses for
students’ basic listening comprehension, language complexity, and communication. The
possible combinations of scores for the three sections are:
 R-R-R:* no remediation required
 R-R-A in any combination: no remediation required
 R-A-A in any combination: no remediation required
 A-A-A: remediation required
 U in any area: remediation required
*(R = recertification recommended; A = acceptable but may need remediation;
U = unacceptable)
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
5
Remediation will be provided by Joan Ford. If the practitioner remains unsuccessful, Joan will
inform the practitioner and her/his Program Director that the practitioner must retake the initial
training. The status email notification will indicate the valid through-date for administering
BEST Plus assessments. Please note: no certificates will be given; programs must retain their
email notification.
Annual Recertification
To ensure scoring accuracy, BEST Plus test administrators must recertify annually by taking part
in a Scoring Refresher and successfully completing a 30-minute scoring activity. Each year,
practitioners that are currently certified to administer the BEST Plus assessment will be notified
in late January by ACLS that the recertification process will begin soon, and which Scoring
Activity to complete; program directors must give the information to all relevant staff as soon as
it is received.
New: In FY14, a new policy was instituted in which BEST Plus test administrators who have
scored consistently in the "recommended" area for Listening Comprehension, Language
Complexity and Communication in the annual recertification for the past 3 years may recertify
every 3rd year instead of every year. Practitioners who benefit from this change must also
continue to test at least 20 students each year to maintain their fidelity to the BEST Plus scoring
rubric. Joan Ford will send a list of those staff that do not need to recertify until FY17.
For staff who do need to recertify, test administrators will need:
 The BEST Plus Scoring Refresher Toolkit
 The BEST Plus Scoring Refresher Workbook, Test Administrator's Edition (one for each
Test Administrator)
Every ESE-funded program has been given a set of these materials. If you do not have these
materials at your program, please purchase one from the test publisher at
http://www.cal.org/aea/bestplus/products.html. Test administrators should use the Toolkit to
review and recalibrate their scoring within a structured forum so that they can share their insights
and questions about administering and scoring BEST Plus tests. For example, test administrators
may view the Toolkit DVD/video either as a group with a facilitator (preferred method).
However, if this preferred method is not feasible, practitioners may proceed individually.
BEST Plus test administrators will have one month to score a sample set of assessments and
submit them to their Program Director or ESOL Assessment Coordinator so that she/he may send
them to Joan Ford, SABES director of assessment professional development. The deadline for
getting the completed scoring activities to Joan is Friday, February 27.


There are no exceptions for late submissions. Test Administrators who do not return their
Scoring Activity by this date will not be able to administer the BEST Plus until they retake the initial BEST Plus training, refresher, and scoring activity.
(Practitioners receiving their Initial Certification in FY15 will not have to recertify until
FY16.) Recertification status will be valid for one year.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
6



Ms. Ford will inform program directors/ESOL Assessment Coordinators of their staff’s
recertification status by the end of March, 2015. Recertifications submitted earlier than
Feb. 27 will be processed in the order in which they are received, and thus programs will
be notified sooner than the end of March.
Staff who receive a borderline “Acceptable” or an “Unacceptable” score in any one of the
three rubric areas will be contacted by Ms. Ford to receive remediation in order to
continue to administer the BEST Plus.
Please note: no certificates will be given; please retain your notification email from Ms.
Ford about successfully passing the BEST Plus.
New for FY15: Change in Procedure for Submitting BEST Plus Recertifications
This year, there will be a revised method for submitting BEST Plus Recertifications. We ask that
Program Directors (or ESOL Assessment Coordinators) send all the BEST Plus recertification
scoring activities together with a list of practitioner’s names rather than each practitioner sending
theirs separately. Please be sure the practitioner’s name is on every scoring activity sheet. To aid
directors in this process, a BEST Plus Recertification Form has been developed for your use.
Directors/ESOL Assessment Coordinators, please use this form when submitting BEST Plus
recertifications.
There have been problems in the past with some scoring activities getting lost, and because all
statewide assessment activities are now centralized with one person, we hope to avoid any
problems with recertifications becoming lost or unaccounted for. All information regarding
successful recertification or any need for remediation will also go to the director, who we ask to
then pass on the information to his/her staff. We believe this revised system will result in clear
communication between programs and the professional development assessment office, and we
appreciate your help in this matter.
Please send all BEST Plus recertification scoring activities either via email, mail or fax to:
Joan Ford, SABES director of assessment professional development
Address: Bristol Community College, 777 Elsbree Street, Building Q, Fall River, MA
02720 Phone: (774) 357-2190
Email: [email protected]
Fax: (508) 730-3280
For more information and specific instructions on how to use the BEST Plus Toolkit, please refer
to Appendix A.
Required Test Materials
At the BEST Plus initial training, participants receive the Test Administrator’s Guide (TAG) and
CD-ROM needed to access test administrations. The TAG and accompanying CD are provided
free of charge by the Massachusetts Department of Elementary and Secondary Education. There
is a charge for each BEST Plus test administration.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
7
The cost per administration depends on the total number purchased. For more information, go to
http://www.cal.org/aea/bestplus/ca.html To order online, go to http://calstore.cal.org/, or
download their CAL Store Quick Reference Guide at
http://calstore.cal.org/QuickReferenceGuide.pdf. To download the BEST Plus Order
Form/Request a Quote form, go to: http://calstore.cal.org/CALOrderForm.pdf.
When to use the BEST Plus, when to use the TABE CLAS-E Writing Assessment
Student SPL Level
Assessment to Use
If an ESOL student is placed at SPL 0 or
above:
Use the BEST Plus or the TABE Clas-E
Reading or Writing Assessment
If an ESOL student’s goal is to improve
his/her speaking and listening skills:
Use the BEST Plus
If an ESOL student’s goal is to improve
his/her writing skills:
Use the TABE Clas-E Writing Assessment
If an ESOL student’s goal is to improve
his/her reading skills:
Use the TABE Clas-E Reading Assessment
Student Placement and When to Administer
Programs may use any placement tests they choose. The BEST Plus may be given at intake, and
may be given as both placement and pre-test. It must be administered within the first two to four
weeks of instruction.
Student Primary Assessment Area May Change in Fiscal Year
Either BEST Plus, CLAS-E Reading or CLAS-E Writing must be selected as the primary
assessment to report educational gain for students in ESOL classes. Programs are required to
administer only one of the three, but more than one assessment may be administered, with one
counting as the student’s primary assessment area and entered into SMARTT.
 A student’s Primary Assessment Area may change during the fiscal year and assistance
from SABES SMARTT Tech Support is no longer needed.
 Learners who are co-enrolled may have different primary assessment areas at the
different programs where they are enrolled to maximize their time at each program. (See
the section on co-enrolled learners for more information).
Programs Enrolling Students after April 1 may Receive Credit for Post Tests and Learning
Gains, under Certain Conditions
Students who are enrolled in a program April 1 or after and who have 65 hours of attended
instruction between April 1 and June 30 may take a pre- and post-test. The learner’s post-test
will be added to the program’s percentage of pre-/post-tested learners.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
8
Any type of attended hours qualify, including rate-based class hours, non rate-base class hours,
match hours, and distance learning hours. In addition, if students have attended 65 hours of
instruction between April 1 and June 30 and make learning gains, these gains will be added to
the program’s percentage of learner gains.
Test Accommodations
An adult learner with a disability must provide the ABE/ESOL program with a disability-related
documentation if he/she requires accommodation(s). If a counselor or other program staff person
determines through a screening that there is a strong possibility that the learner has a learning or
other disability, a formal assessment may be undertaken. A formal assessment of a learning
disability must be administered by a licensed professional (e.g., psychologist, school
psychologist, or psychiatrist) and is valid for 5 years from the date of the formal assessment. In
some cases, a “licensed professional” may be a speech, vocational, physical, or occupational
therapist with verification by a licensed medical doctor, psychiatrist or psychologist. An IEP
(Individual Education Plan) is not a documented formal diagnosis of a learning disability.
“Disability-related documentation” includes educational assessments, or an Individual Education
Plan (IEP) developed by the public school system to document a person’s disability for an
accommodation in the ABE program. For more information about modifications and
accommodations relating to ABE instruction, please refer to pages 13-15 in the ACLS Disability
Guidelines, at http://www.doe.mass.edu/acls/disability/default.html.
For any questions, concerns, or if staff have a student who should receive test accommodations
on the BEST Plus, please contact Dana Varzan-Parker, ACLS Assessment Specialist, at [email protected] or April Zenisky-Laguilles at [email protected]
Pre- and Post-Testing Interval Varies by Intensity of Class Hours
Programs need to pre- and post-test enrolled students each fiscal year in the student’s primary
area of assessment to capture student educational gain. Programs should generally post-test after
an interval of 65 hours of instruction. Programs may give the BEST Plus test to their learners up
to three times in a fiscal year. The test administered for the pre-test must remain the same for the
post-test (e.g., one may not pre-test with the BEST Plus and post-test with the TABE Clas-E
Reading or CLAS-E Writing).
Co-enrolled Learners
Assessment information for learners co-enrolled or transferred will appear in SMARTT so that
all programs involved with those students can use it.
A student’s required assessment (e.g., BEST Plus, CLAS-E Reading or Writing) will appear in
SMARTT at all sites where the student is enrolled, regardless of the site where a student took the
test.
 Assessments are color-coded in the SMARTT Student Assessment Screen to indicate
where the student took the test.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
9




The Primary Assessment Area may be set at all sites within the current fiscal year, and
may vary between sites. (For example, a student may have a primary assessment area of
math at one site, and reading at a site at which he/she is co-enrolled).
The copy icon will appear next to all May/June tests so that users may copy the last test
taken from any site to their own site in the new Fiscal Year.
A link labeled “Other Site Assessments” located on the SMARTT “Welcome Screen”
shows the list of co-enrolled students who took tests at other sites. There are two panels
for current and exited students. Programs may view the list and drill down to the
assessment screen to view tests for individual students. Programs may decide if they wish
to set the student’s primary assessment area for this test. A column also shows if the
primary area has been set. This feature will help programs determine whether or not
dually enrolled students need to have another assessment administered immediately.
To access the SMARTT Assessment Reports, click on the "Site" link on the left menu in
SMARTT, and then click on the "Assessment Report Primary Set" option. The report will
list all the tests, dates, forms, and total number of attended hours. When the total number
of attended hours is 65 or more, a program may test the student.
Cognos Report of all Co-enrolled Students at a Site
In addition to the SMARTT Student Assessment Screen, there is also a report in Cognos that
allows a program to view all co-enrolled students in a site, sorted by their assessment. This report
lists the co-enrolled student’s name, the other site(s) in which the student is co-enrolled, the site
at which the test was taken, the date taken, the test name, test form and level, and the test score.
This report can be helpful in that the SMARTT Student Assessment Screen lists students
individually, while this Cognos report lists all co-enrolled students in a program, filtered by their
assessment. To use this report, log on to Cognos http://www.doe.mass.edu/acls/smartt/ using the
program’s or an individual Cognos account. Once on the ACLS Homepage, select the tab at the
top labeled “Desk Review” and the report on co-enrolled students will be among the reports
listed.
To Receive a Cognos Account
Program staff may use their Program’s Cognos account since every site has one. Staff wishing to
have a Cognos account may have their program director request one for him/her, as individual
access is at the discretion of the Program Director. Requests for accounts need to be made two
weeks prior to when the account is needed. Program directors, please send requests to Sriram
Rajan [email protected] with the following information supplied:
 Name of staff who will receive the account
 Email
 Official Site Name (no acronyms or abbreviations)
 Role at Program (specify Teacher, Counselor, Site Coordinator, or other)
 The staff request for what they would like the Cognos User ID to be (e.g., 1st initial and
last name, or some other personally meaningful identifier)
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
10
Users will receive their Cognos account information by email, along with a generic password.
When users first log in, they must create their own password. Users need to write down the user
ID and password and keep them in a handy place. Users and Programs must manage their own
passwords and User IDs; ACLS will not have that information.
Test Security
All BEST Plus CDs and BEST Plus student test scores must be kept in a secure place. Although
students’ test scores are stored on the hard drive of the computer used to administer the test, it is
important to back up the hard drive, and/or keep printed copies of the scores in a secure place.
After the BEST Plus assessment is completed, a score report is generated that can be printed out
or saved to the hard drive or disk. Staff should share, explain, and discuss score reports with
students so students can understand their progress.
Staff and test administrators may not use the BEST Plus test items to prepare learners for their
BEST Plus tests. The Massachusetts Department of Elementary and Secondary Education
reserves the right to immediately terminate the program’s grant if any staff are found to be
violating the BEST Plus assessment policy regarding test security.
During BEST Plus Administration
Who May Administer
Teachers who are certified to administer BEST Plus may not administer the BEST Plus to their
own students. Any other certified test administrator may administer the BEST Plus.
Following Test Administration Procedures
Test administrators must follow the test directions in the BEST Plus Test Administrator’s Guide
(TAG). The test publisher states in the TAG, “Test administrators have ultimate responsibility
for the test’s reliability. Test administrators must agree to:





Carefully read all materials presented in the Test Administrator’s Guide
Use the test only for appropriate purposes and with examinees for whom it is intended
(e.g., it is not intended for hearing or speech-impaired, or visually impaired learners)
Follow exactly all administration and scoring procedures
Follow sound professional practice in discussing and releasing scores.
Not duplicate or alter in any manner any BEST Plus materials or test items without
receiving prior written permission from the Center for Applied Linguistics.”
Using Assessment Reports for Tracking When to Post-Test
The SMARTT system generates assessment reports so that a program can track when to
administer the next BEST Plus test. The BEST Plus report will list the date administered for the
previous fiscal year, either the copied May/June test or a new pre-test, and optional mid-year test
for the current fiscal year.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
11
The assessment reports may also be used to check which learners have not yet been post-tested
near the end of the fiscal year. Any of the reports may be exported into Excel.
 To access these reports, go into SMARTT, select the “Site” link on the left menu in
SMARTT and then select “Assessment Report Primary Set.”
 To see BEST Plus assessments taken by all learners at the program regardless of their
Primary Assessment Area, instead select “BEST Plus Report” (shown in green) below the
“Assessment Report Primary Set.”
 Any tests completed after June 30th will be counted as pre-tests with a date of July 1 in
the next fiscal year.
Test Conditions
The testing location must be quiet and comfortable so learners will not be distracted by their
surroundings while taking the test. Learners may be tested in a quiet computer lab, empty
classroom, office, or other space. Learners may not be tested in an occupied classroom or space
where other students are in class working or talking.
Recalibrate Every Time Before You Test
To ensure consistent BEST Plus test scores statewide, all test administrators must achieve interrater reliability, so that all practitioners throughout the state are uniform in their scoring. All
scorers need practice and require refresher work before any testing session to maintain their
uniformity of scoring. The goal is for all scorers to calibrate themselves to the rubric and training
materials, not to each other.
BEST Plus test administrators must recalibrate each time they test to ensure consistent scoring
accuracy among BEST Plus test administrators. Test administrators must review the BEST Plus
score rubric and then read and score the Benchmarks and Scoring Practice activities in Section V
of the BEST Plus Test Administrator Guide (TAG). Test administrators should compare their
scores with the official answers in the TAG. If scores are inconsistent with the scoring activity
two or more times, contact Joan Ford, Dana Varzan-Parker, or April Zenisky to discuss (see page
14 for contact information). Test administrators who have been administering the BEST Plus for
one or more years will find the Scoring Refresher Video in the Scoring Refresher Toolkit helpful
in clarifying scoring beyond what is covered in the initial training.
Test administrators must use the scoring rubric each time they test.
It is very important for test administrators to be aware of their limits for scoring before
exhaustion begins to affect the reliability of scores. Generally after administering four tests in a
row, it is time for a short break. Include time for calibration and re-familiarizing yourself with
the BEST Plus rubric.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
12
After Administering the BEST Plus Test
BEST Plus Scoring Monitor
Programs must designate one person as the BEST Plus Scoring Monitor. The monitor compiles
all the scores and maintains them in one place. She/he ensures that rubric and scoring procedures
are followed, and notes if any of the readers’ scores appear to be inconsistent with the rubric or
potentially inflated compared to other data regarding a student’s oral proficiency. If scoring
inconsistencies occur, the BEST Plus Scoring Monitor follows up with Joan Ford to ensure
scorers are scoring according to the BEST Plus rubric and receive technical assistance.
Score Reporting and Exit Criteria for NRS Advanced ESL Students
BEST Plus scale scores must be recorded in SMARTT. Scale scores are the type of score used
for the BEST Plus, and they are used to compute and derive all other scores associated with the
BEST Plus. The SMARTT ABE database will translate the scale scores into the levels stipulated
by the federal National Reporting System (NRS). Programs may generate reports that portray
student educational functioning levels and gains using SMARTT.
The exit criteria for BEST Plus is a scale score of 541 and above, which corresponds to SPL 7.
Students who test at a scale score of 541 and above during the fiscal year may remain in the
program until the end of the fiscal year.
ESOL Speaking and Listening Correlations using the BEST Plus Test
Scale Scores
400 and below
401 – 417
418 – 438
439 – 472
473 – 506
507 - 540
541 and above
Student Performance
Levels (SPLs)
0-1
2
3
4
5
6
7
NRS level
Beginning ESL Literacy
Low Beginning ESL
High Beginning ESL
Low Intermediate ESL
High Intermediate ESL
Advanced ESL
Exit Criteria
Measuring Learner Gains
Learning gains are calculated each fiscal year from pre- and post-testing and based on learners’
first (pre-test) and last test.
Massachusetts measures learners’ educational gain in two different ways. First, Massachusetts
reports the number of ESOL learners completing or advancing one or more Educational
Functioning Levels (EFL) as defined by the US Department of Education’s National Reporting
System (NRS). Massachusetts is required by the US Department of Education to not only report
learning gains based on EFL completion rates, but also to use EFL completion rates as a measure
of program performance.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
13
In addition to measuring learning gains by EFL completion rates, Massachusetts measures
“meaningful educational gain.” Meaningful educational gain is measured solely by the
improvement in test scores between the pre- and post-test and does not take into consideration
Educational Functioning Levels. Meaningful educational gain is measured by an increase of 33
or more scale score points on the BEST Plus.
Measuring completion of Educational Functioning Levels for the federal US Department of
Education (National Reporting System, Federal Report Table 4)
 The National Reporting System (NRS) requires that all students who have 12 hours or more of
attendance be included in all Federal Report tables, including the tables that report pre and posttested learners and make gains by completing an Educational Functioning Level.
Copying Over BEST Plus Scores to the Next Fiscal Year
Any BEST Plus tests given to students in May or June may be copied to count as the first (pre-)
test in the new fiscal year. Program staff also have the option give a new pre-test. The copy icon
will appear next to all May/June tests so that users may copy the scores from any site to their
own site in the new Fiscal Year if they wish.
 The May/June test will then be dated July 1 of the new fiscal year. The July 1 date is
color-coded to let program staff know it was copied.
 Transitions programs, see specific policies at
http://www.doe.mass.edu/acls/assessment/TCCPpolicy.html.
Questions
Program staff must read the Assessment Updates in the ACLS Monthly Mailings for important
new information: http://www.doe.mass.edu/acls/mailings.
Please contact April Zenisky-Laguilles at [email protected], or Joan Ford, SABES
director of assessment professional development, at Bristol Community College, 777 Elsbree
Street, Building Q, Fall River, MA 02720; Phone: (774) 357-2190; Email:
[email protected]; Fax: (508) 730-3280.
For policy-related questions, please contact Dana Varzan-Parker, Program and Assessment
Specialist, at Adult and Community Learning Services (ACLS), 75 Pleasant Street, Malden, MA
02148; Phone: 781-338-3811; Email: [email protected]; Fax: (781) 338-3394.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
14
Appendix A: Massachusetts' Procedures for Using the BEST Plus Scoring
Refresher Toolkit
Overview
To ensure scoring accuracy statewide, all BEST Plus test administrators need to be recertified by
1) taking part in a Scoring Refresher and 2) successfully completing a 30-minute Scoring
Activity. To recertify, you will need:
• A BEST Plus Scoring Refresher Toolkit
• A BEST Plus Scoring Refresher Workbook, Test Administrator's Edition: one Workbook
for each BEST Plus test administrator
Test Administrators must use the Toolkit to review and recalibrate their scoring within a
structured forum to share their insights and questions about administering and scoring the BEST
Plus. It is also an important way to review and recalibrate their scoring in order to ensure reliable
assessment data statewide.
1. The Scoring Refresher
The Toolkit and Workbook may be used by an individual test administrator or by a large or small
group led by a facilitator. Participating in a group is preferred, since test administrators benefit
from each other's questions and experience. If a group Scoring Refresher is done, a facilitator is
required. If the Refresher is done individually, he/she needs to read the Toolkit and
accompanying facilitator's edition of the Workbook prior to the Refresher. The Toolkit contains
helpful information and a Workbook with extra information for the facilitator.
The Refresher Facilitator's Role
Facilitators need to be experienced with using BEST Plus and in working with the adult ESL
learners, and be:
 Comfortable and confident with the scoring rubric and all three language components tested
(Listening Comprehension, Language Complexity, and Communication)
 Able to use the scoring rubric to justify scores used as benchmarks on the video
 Experienced with facilitating discussions and be able to summarize information presented in
the Scoring Refresher Video
 Empathetic and able to judge when participants need a break or a question answered
 Available to review all Scoring Refresher materials carefully before facilitating a Refresher
session
Facilitators may be a staff person who is in charge of assessment at your site; an on-staff
certified BEST Plus trainer or a certified BEST Plus trainer hired by the program to facilitate.
 Facilitators must read the Toolkit and facilitator's edition of the Workbook prior to leading
the Scoring Refresher.
 Facilitators must make clear to test administrators that although their personal ideas and
concepts of language proficiency and assessment can differ from those represented in BEST
Plus, these differences must be set aside for the purposes of BEST Plus scoring accuracy.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
15
 The Toolkit or DVD/Videos should not be taken home by staff and is the property of the
program. The Toolkit materials must be kept in a secure place when not in use. Programs that
lose their Refresher Toolkit must purchase another for $160 at
http://www.cal.org/aea/bestplus/toolkit.html.
How and When to use the Toolkit:
 Facilitators must follow the procedures outlined in the Toolkit for conducting the Scoring
Refresher. Individuals using the Refresher by themselves must follow the Agenda for selfstudy in Section V-9 in the Toolkit.
 The Refresher will take 2.5 to 3 hours to complete. The Refresher can be separated into two
sessions, but not broken into more sessions than that.
 Facilitators must use the Needs Assessment, Self Assessment, Refresher Planning Worksheet
and Checklist materials in Section V, pages 3-8. All test administrators must complete the
Self Assessment in Section V, page 3-4 in the Toolkit prior to the refresher. (The facilitator
needs to copy and give it to all test administrators.) A detailed agenda for a facilitator-led
Scoring Refresher session is in Section III, page 3.
 Additional helpful materials for facilitating the Scoring Refresher are in Sections III and V.
 Additional information and considerations for when to score are in Section III, page 2-3. A
helpful flow chart is on page 3.
 Be prepared for a number of times where facilitators will need to pause, back up, or fast
forward through the video or DVD as it is used in the Workbook.
 The Toolkit instructions state that you may read the sections of the Workbook either now or
later while watching the video. There is important information in the Workbook that is not
gone into as much detail in the video. Test administrators must read the entire Workbook at
the same time as you are going through the video. Please pay particular attention to the
"Summary" sections in the Workbook. Stop the video and read the corresponding section in
the Workbook before moving on.
2. The Scoring Activity for Recertification
 Up to a week after the Scoring Refresher, BEST Plus test administrators and the facilitator
are required to watch and score responses to a 30-minute Scoring Activity (which Activity
will be announced each year in January). For more information, see Section IV of the
Toolkit. The Refresher must be completed one week or less before doing the Scoring
Activity.
 Use the red scoring video (located in the front of the Toolkit) and note your answers in the
two perforated sheets in Appendix D of the Workbook.
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
16
 Prior to completing the Scoring Activity assigned, please read and the Scoring Activity
Instructions found in Section IV, pages 2-3, and then follow the instructions.
 Test administrators must watch the Scoring Activity on site at the program and must strictly
adhere to proper test procedure protocols (e.g., do not confer with each other during the test)
Do not stop, pause or rewind the video at any time or allow for discussion of any items.
 All Scoring Activities must be received by Joan Ford by the deadline identified in ACLS
correspondence (usually the end of February). Submit the scores by mail or fax to Joan Ford
(see contact information at the bottom of this page).
 Those who do not return the Scoring Activity by this date will not be able to administer the
BEST Plus until re-taking the initial BEST Plus training, Refresher and Scoring Activity.
 The scoring activities will be scored and test administrators will know via email of their
competency status by one month after their deadline for submission (usually the end of
March). Scores will fall into three categories, based on the scoring system developed by
BEST Plus test developers: “Recommended,” “Acceptable,” or “Unacceptable.” An
explanation of scoring is found in Section IV, pages 4-5, 19 and 25. (Please note that pages
6-18 have been removed from the Toolkits.)
 Test administrators receiving a new certificate will be valid to test for at least one year.
Please note: no certificates will be given; program staff must retain their email
notification.
 Staff who receive a borderline “Acceptable” or “Unacceptable” score in any one of the three
rubric areas will have an opportunity to receive remediation and may not administer the test
until successfully completing remediation.
Send all scoring activities to:
Joan Ford, SABES Director of Assessment
Address: Bristol Community College, 777 Elsbree Street, Building Q, Fall River, MA 02720
Phone: (774) 357-2190
Email: [email protected]
FAX: (508)730-3280
Assessment Policies for Using the BEST Plus Test, 2014
Massachusetts Department of Elementary and Secondary Education, Adult and Community Learning Services
17