Independent Study of Effects of Testing Disruption

Today, the SDE issued this press release regarding the disruption to student test scores:

Student Test Score Study Presented to State Board of Education OKLAHOMA CITY (Aug. 20, 2013) – The Oklahoma State Department of Education announced today that the independent study to determine if student tests scores had been compromised by the two days of disruption this past spring is complete. The HumRRO Statistical Investigation of Oklahoma Disruptions indicated that students appeared to do as well on the test as students not involved in the disruption. Based on the result of this study, State Superintendent Janet Barresi determined the state will retain all scores of impacted students who scored proficient or advanced. While students as a whole did not experience depressions in scores, it is possible that some individual students did not perform to their highest potential during the period of disruption. Therefore, Superintendent Barresi will not report the scores of impacted students who scored limited knowledge or unsatisfactory. The full report was provided by Dr. Arthur Thacker, Program Manager for Human Resources Research Organization (HumRRO), Inc. to the department and delivered during Tuesday’s State Board of Education’s meeting by Dr. Maridyth McBee, Assistant State Superintendent of Accountability and Assessment. “I’d like to thank Dr. Thacker and HumRRO for conducting this study,” said State Superintendent Janet Barresi. “Last year’s testing period was very difficult for our students, teachers and testing coordinators. The difficulties they experienced were unacceptable. It was a high priority for everyone to commission an independent study to determine what effect the disruptions had on the student test scores. Now that the study has concluded, we will work with the districts to take action on behalf of students and schools where the disruptions merit such action. “Even though this study suggests no systematic impact on test scores, not reporting the scores of students who scored limited knowledge or unsatisfactory will ensure there is no lasting impact on student performance. This is the right thing to do for students and for schools.” Server capacity problems by testing vendor CTB/McGraw-Hill on April 29 and 30 caused a number of students to be interrupted during their testing experience, making it necessary to determine the extent to which the disruptions impacted students’ test scores. The State Department of Education asked HumRRO to conduct a third-party independent study to investigate the impact. HumRRO was selected by the SDE because the organization is recognized for their objectivity and independent approach to evaluating the performance of testing vendors. They have also conducted similar work for the State Department of Education during a previous challenge with a different testing coordinator. A state Technical Advisory Committee of researchers and assessment experts who meet twice per year to advise the State Department of Education on accountability and assessment issues recommended the design for HumRRO’s disruption study. HumRRO conducted the study independently of the SDE and CTB/McGraw-Hill. Studies conducted for Indiana by another organization found similar results, as did a study conducted for Minnesota by HumRRO. “The findings of the report in relation to the discrepancies did not merit action in HumRRO’s opinion,” said Superintendent Barresi. “Because the two-day incident resulted in so much stress and chaos of the school day I wanted to assure students, teachers and schools were not held accountable for circumstances beyond their control.” Additionally the following actions were taken to address discrepant scores: About 1,400 students received an immediate raw score that differed from the score they received on the two-week preliminary test score report. The discrepancy occurred for two reasons: o There appeared to be a short time period on the second day of the interruptions when some student responses to test questions were not saved and therefore not included on the two-week report. o CTB inadvertently left the “winter test system” as an option for download which caused incorrect data to be saved. Corrective action will include: Students will receive the higher score between the raw score and the two-week report score. Students with higher raw scores will receive a letter from CTB indicating their proficiency level. They will be included in the report card, but not in the school or district summary report. Students with a higher two-week score will receive a traditional score report that includes performance level and performance by content standard.

Already, my inbox is flooding with comments. Here is one example:

Interesting that the SDE presented the “independent” report. (I believe HumRRO is indeed independent, by the way.) …… The HumRRO study compared “disrupted” students to “nondisrupted” students. Given the fact that the SDE has insisted that only a very small number of students were officially disrupted, how do we know that the “nondisrupted” group was really that much different, in terms of exasperation, from the officially disrupted group? I hope this will be addressed in the HumRRO report, but the report wasn’t published. I think I heard Maridyth say that it was not yet ready. (I note that the Board acted before it was, merely implementing the SDE recs….)

I’m with the commenter. I look forward to reading the report in its entirety. I tend to like other people digesting long research reports for me and spitting out bullet points. However, in this case, I look forward to doing my own synthesis.

I am struck by three key points at this time, however:

1. It appears that students whose tests were disrupted won’t be penalized if they didn’t pass. Allow me to sum this up in a little matrix:

	Tests Disrupted	Tests Not Disrupted
Didn’t Pass	Scores Don’t Count	Scores Count
Passed	Scores Count	Scores Count

2. The press release carefully compares this report and subsequent corrective actions to those in Indiana. This is a politically astute move since people like me would have done so anyway.

3. We are just now learning (at least publicly) of another mistake made by CTB. They failed to inadequately prepare the system for testing, leaving the winter option on the menu for testing coordinators. I do not know the extent to which this caused data problems from that statement alone.

As always, I’d love to hear your thoughts.