Home > Uncategorized > He Who Casts the First Aspersion

He Who Casts the First Aspersion

December 2, 2013

The Oklahoma State Department of Education wants you to know that they too have access to “researchers.” Of course theirs is only one person and comes without the air quotes commonly found during campaign shindigs. Oh, and she’s not from around here.

From the Tulsa World:

A controversial study by researchers at two Oklahoma universities that deems the state’s A-F school grading system as flawed is “misleading,” according to an in-house analysis by staffers at the Oklahoma State Department of Education.

“It’s not to cast aspersions on the researchers at all,” said department spokesman Phil Bacharach. “But we think it’s important to put that research in context, especially if it’s going to be used as a sort of blanket criticism of the grades.”

Nothing against these fine researchers, but their work is misleading. Gee, how could that be seen as criticism?

The SDE has borrowed a “Harvard University strategic data fellow” for two years (at a cost to the state of $85,000 per year) to conduct analysis of … well, of hopefully more than this. While the OU/OSU researchers found flaws with the A-F Report Cards, Clifford found flaws with the design of the study. These include:

  • Using only urban schools representing about three percent of the total enrolled population in Oklahoma
  • Using a population that is not representative of the state population as a whole
  • Inserting controls for poverty and race into the calculations

So many things puzzle me about this article, but Clifford dismissing the sample size is just intellectually dishonest. The OU/OSU study included more than 15,000 data points. From a research perspective, that’s huge. Whatever your findings are at that point, they are bound to be statistically significant. Plus, just last month, Superintendent Barresi said accountability would be calculated for schools on subgroups as low as 10 students, because that is a significant number (as opposed to previous reporting requirements of 30 and 52).

Think about that student count again. The researchers had student-level data for 15,000 students – three percent of the students in Oklahoma. The federal government doesn’t come anywhere close to testing three percent of students for NAEP, yet we have to listen to scores of politicians bloviate over their results. Tomorrow, when international PISA results come out, the sample size will be even less representative. Yet these results will get the words flowing from think tank after think tank.

Clifford conducted her own study, and for some reason, did not control for poverty and race. She also translated results into months and years of learning. The problem is that nothing in the design of Oklahoma’s assessments lends itself to that kind of output. The technical manuals for the tests don’t equate differences between scaled scores to months and years of learning. Honestly, we can all conduct our own studies with our different methodologies. Each would have their merits and limitations. For example, a few weeks ago, I ran regression tests using site-level data and then district-level data. The greatest limitation of my findings was that I was comparing units of very different sizes. Nonetheless, the conclusion that poverty matters more than any other variable was undeniable. As such, the decision to exclude poverty from any model studying student achievement or school accountability probably should come with a thorough and compelling theoretical framework.

As for the finding in the OU/OSU report that a single letter grade is not a clear or reliable measure of school performance, Assistant State Superintendent Maridyth McBee had this to say:

It definitely is accurate in telling everyone what percent of students at that school are proficient in reading, math, science and writing, and what percentage of students are growing from a lower achievement level to a higher achievement level.

If that is the goal, all the SDE has to do is publish test data for schools. What percent of students passed the Algebra I EOI? The A-F Report Card actually doesn’t tell us that. What percent of students passed the third grade reading test? That either.

Not to cast aspersions, but useful data like that wouldn’t give education reformers their talking points.

  1. Jennifer B.
    December 2, 2013 at 10:59 pm

    Her background is in higher education in developing countries. Hmmm.


  2. Jackson P.
    December 3, 2013 at 9:19 pm

    I would like to read Clifford’s report as at one point the article says she did not include poverty and race, then later it says she did. But the process of trying to get a complex idea into a general circulation publication can sometimes leave out important pieces.

    The sample size is not the issue – it is the sample that is the issue. We all know that a sample size even smaller than 3% can still result in statistically valid data. (Well, based on the math scores for the state, only a few of us know that a sample size even smaller than 3% can result in statistically valid data.) However, the report’s authors only used urban schools. They did so purposefully and wrote their justification. Even they did not claim that their 3% represented the whole state – they were looking for other effects. Calling Clifford intellectually dishonest based on how a newspaper reported the way she questioned the choice and presented her conclusions seems a bit of a stretch.

    While I am not a fan of A-F as there are more meaningful ways to assess schools, I have been struck by the contradictions of superintendents opposing it:

    –Superintendents who are quoted as against A-F almost always say, at some point, something like, “Of course we believe in school accountability but…” If they believed in school accountability, they would have created their own easy-to-comprehend systems that they published for their constituents long before A-F began. The A-F report mentions several such systems – why didn’t any Oklahoma school adopt them on their own before A-F?

    –Superintendents complain about getting a “C”. Yet how many of those, when in the classroom, told their own students, “A ‘C’ is acceptable because it’s average?” The A-F scale used on students is far more arbitrary than that used to grade schools.

    –Superintendents complain that a single grade does not reveal the complexity of the good in their schools, but how many of them give their students letter grades that then become the GPA, a single grade purporting to tell about four years of a student’s life?

    If they were serious about their complaints about A-F, they would abolish A-F as a way to rank their own students.


  1. June 5, 2014 at 7:33 am
  2. June 11, 2014 at 9:33 pm
Comments are closed.
%d bloggers like this: