But will it fit through the straw this time?

The state that loves change for its own sake and the profession that loves acronyms has a new treat for you. Because our SDE has decided to vacate the time, effort, and money sunk into the development of the PARCC exam, we had to come up with our own new thing.

First, let’s flash back about 10 weeks. This was Superintendent Barresi’s message about why we were pulling out of PARCC:

If we move ahead with this, we are going to be asking the state to drink a milkshake using a cocktail straw,” Barresi said. “If you look at what happened with testing this year — kids getting screen frozen, knocked off the test — those were technical issues that were from the districts’ end of things. (The testing vendor) crashed for two days because of server problems, but almost every bit of the rest of it was due to district issues. I’m not pointing fingers, but it is the reality.”

The SDE let the world know that Oklahoma would not administer the PARCC tests, but rather would pay a vendor to develop and administer a test with PARCC-like items. Now, you can go to the state’s Central Purchasing website and view the Request for Proposals (RFP) for Elementary & Secondary School Student Testing.

The first 16+ pages of the RFP are standard legal requirements. The meat of the document begins at the bottom of page 17:

C.1  The purpose of this RFP is to solicit proposals for activities related to the implementation of a statewide summative assessment. The assessment will be aligned to the Grades 3-8 Oklahoma Academic Standards for administrations in the spring of 2015 with additional one year optional renewals for the administrations in the spring of 2016, 2017, and 2018. Henceforth in this document, this test shall be called the Oklahoma College and Career Readiness Assessment (OCCRA). Development and field testing of custom-created items will occur during the 2013-14 school year.

There’s your new acronym, folks. OCCRA. Get ready immediately for field-testing. Keep in mind this is only the proposal to award a testing contract for 3^rd through 8^th grades. The End-of-Instruction (EOI) exams are separate.

Other parts of the proposal got my attention as well:

C. 2  This RFP will specify a complete package of products and services so that the OCCRA will be administered, scored, and reported separately from other state assessments. The OCCRA will be administered via paper and pencil to students in Grade 3-5. The OCCRA will be administered online with paper and pencil options to students in Grades 6-8.

This is the same as what we’re doing now.

C.4.3  Although the state testing dates for spring 2015 have not yet been officially determined, the projected dates are shown in Table 1 below. The final determination of the test administration dates for the OCCRA will be based on statutory requirements for test administration and reporting. Currently, no summative assessment may take place earlier than April 10 of each year. Immediate raw scores are required for all assessments administered online. Complete preliminary reports including the raw score, performance level and scale score must be provided no later than two weeks following the close of the testing window. When standard setting is needed the two week reports do not include performance levels.

Proposed Test Administration Dates
ACTIVITY	2015
Oklahoma Core Curriculum Test Administration (Grades 3-8)	April 13-May 3
OCCRA Administration	April 13-May 10

This section strikes me on a couple of levels. First is the expectation that schools will receive scores no later than two weeks after the end of testing. That’s optimistic, based on what we’ve seen the past couple of years (from two different vendors). As nice as it would be to have raw score reports in a timely manner, without the performance levels, they aren’t that meaningful. Schools like to run remediation programs in the summer and populate those with students who scored poorly the year before. Lacking this information complicates the process of communicating the need to parents. Second is the realization that OCCRA is just a testing program for reading and math. The SDE apparently plans to continue administering the OCCTs for other subjects. The following section of the RFP backs this conclusion:

C.6.1  The supplier(s) will provide the State of Oklahoma with a criterion-referenced achievement test measuring mastery of the Common Core State Standards in English Language Arts and Mathematics proficiency for administration to students in Grades 3–8 in all Oklahoma public schools. The student achievement scores must be interpretable in terms of the criterion and must be valid and reliable for use in student promotion, retention, and graduation decisions. The scores must also be valid and reliable for use in teacher, school, and district accountability measures. This RFP does not require nor expect the Supplier to create an entirely custom test but, instead, assumes the purchase of products and services derived from existing products published by the supplier(s). The proposed OCCRA must meet the design shown in Tables 3 and 4.

If you’ll look closely at the proposal, it doesn’t seem to contain anything labeled Table 3 or Table 4. Never fear – here they are in a separate document. They show that the selected vendor will report student data in terms of the strands of the Common Core. For anyone thinking that the SDE can take cover from the national criticism of the Common Core by hiding it under the newly branded Oklahoma Academic Standards, this is a rather prominent inclusion.

C.8.4  Throughout the development and implementation of the OCCRA, the supplier is required to protect student privacy.

It is important to note that no such requirement exists for the SDE, apparently.

C.8.5  Reading passages should expose students to grade-level text of appropriate complexity. Texts should reflect a balance of 50 percent authentic literature (stories, drama, and poetry) and 50 percent informational text/literary nonfiction which would include passages in grades 6-8 that address the literacy standards requirement for history/social studies, science, and technical subjects that students are expected to read. Selections should include a combination of single texts and paired passages of sufficient complexity for close, analytical reading which would allow students to discern and locate evidence from the text to support assertions in multiple-choice items or writing responses.

This is a bone of contention in the Common Core with many teachers. The 50/50 literature/informational text split is proscribed for all grades. If we were talking about high school students, there might be less frustration over this. With third grade students, who will have to pass this new test to move on to fourth grade, this is a huge transition. While schools have had three years to start the transition to Common Core, they haven’t had this split in mind.

C.8.6  English/language arts items aligning to each passage should include single and/or two part multiple choice items as well as short, constructed responses to text-dependent questions. One extended response item should be drawn from a paired or multiple-text passage set. Extended writing prompts should allow students to produce opinion/ arguments, informative/explanatory responses, and narratives. Grade 3 students should be able to generate a summary as well.

While the writing tests for fifth and eighth grade will now be separate, all grades will have to show proficiency by writing. And who will score all of these student-written responses? We’ll get to that in a bit. I’m going in order.

C.12.3  The proposal must include details as specified below about its computer-based testing delivery system. The proposal must list a minimum of three customers, within the preceding twenty-four months, utilizing the base platform proposed, or list a minimum of three customers utilizing fully customized platforms within the preceding twenty-four months. Their names, titles, email addresses, and phone numbers must be included in the proposal. The supplier must be prepared to demonstrate significant portions of the proposed system during the proposal review meeting.

This is pretty self-explanatory. The proposal continues talking about technology specifications and requirements of the vendor, including:

C.13.1  Support all eligible devices that meet the specifications in the Table 6. The supplier must also commit to continued support for future updates made available on the latest version of Partnership for Assessment of Readiness for College and Careers’ (PARCC) Technology Guidelines that can be found at (http://www.PARCConline.org/technology).Other assistive technologies may be needed for students requiring accommodations.

So we’re still not entirely pulling away from PARCC. We just love them for their technology specs.

C.30.1  The supplier shall describe the online accommodations that are available for administration of the test to students with disabilities and to students with limited English proficiency (LEP). Suppliers should refer to the existing Oklahoma policies in this regard which can be found in Appendix B of this RFP. The proposal shall confirm that the test can be administered according to these policies. If there are variations of accommodations to be proposed by the supplier, this should be made clear in the proposal.

C.30.5  Describe how the needs of students with disabilities and limited English proficiency were included in the early development of the proposed test items and the extent to which such students were included in field tests and other related studies.

Together, these two sections give us our only glimpse into the treatment of special education students by OCCRA. Accommodations will be made. I’m glad they were specific.

C.36.2 The SDE requires that written responses be read and scored by two sources with at least one of those sources being a human reader. This may include a machine score and a human score. If using a machine score the supplier shall address appropriate procedures to ensure validity and reliability.

E.18.1.3  All multiple-choice items will be machine-scored/computer-scored, and short constructed response items can be hand scored by humans or by Artificial Intelligence (AI). Reporting of scores on these items will be based on the procedures specified by the SDE and will include results at the domain level. The supplier will hand score any documents containing student responses upon SDE request to verify student scores.

Now we’re there. We casually enter the world of machine-scored essays and AI-scored constructed response items. It comes not as a thump at your door, but in a sub-clause of a sub-clause of a business document.

E.20.1  The supplier will conduct standard settings for each content and all grade levels following the Spring 2015 test administration. The SDE will approve the method for setting standards. The supplier will provide psychometricians and content area specialists to facilitate the standard setting meetings. The procedures shall include creating  descriptors of what students can do at each of five performance levels. The performance level descriptors are to be unique to each content area. The supplier shall supply evidence of successful standard settings for large-scale assessment programs. The supplier will provide psychometricians and to assist with the information needed for the presentation of cut scores at state board meetings.

We all know how much input educators had this year in standard setting. Their recommendations largely went ignored.

That’s the heft of the RFP. I do want to turn back to page 10 for a moment – back in the legal mumbo jumbo. I did a keyword search for penalty, and I found this:

B.3. Penalty Clause

B.3.1. Once completion dates are mutually agreed upon, and the supplier fails to perform any of the services and/or make deliveries within the time Specified in the contract, or any extension period, the supplier shall in place of actual damages, pay the State of Oklahoma the sum of $15,000 per Calendar day for delayed services/deliveries. The supplier shall NOT be charged when delay in performance and or delivery arises out of causes beyond the control and without fault or negligence of the supplier.

B.3.2. Failure to correct any errors in materials or prevent disruptions that negatively alter the ability to administer the test, to use the score reports, or that denigrates confidence in the testing program will be viewed as a violation of the contract and the will pay liquidated damages to the SDE in the amount of 3% of the total annual contract amount for each day during which the online, electronic, paper, print or other document is incorrect until a corrected online, electronic, paper, print or other document approved by the SDE is distributed by the supplier. SDE agrees that when liquidated damages become a possibility, it will expedite its responses and requirements in this regard in an effort to limit the amount of liquidated damages.

I guess we’ve learned at least one thing.

***

By the way, if you haven’t read Rob Miller’s analysis of how OCCRA will impact his middle school, I encourage you to give it a look.