Reason #9 to Pick a New State Superintendent: The 2013 A-F SNAFU

Since you’re reading a series about selecting a new state superintendent (and reasons not to keep the current one), it’s worth noting that six of the seven candidates showed up at a debate last night at Rogers State College in Claremore (Joy Hofmeister was not there). It was the first time I had seen any footage of Brian Kelly interacting with the other candidates.

There were several notable statements by the candidates – both good and bad. One confused the RSA with the Common Core. One continues to confuse A-F Report Cards with accountability. Several favor the ACT as the state assessment we will use for all things. Since I’m not going to go back through the entire hour and pick each candidate apart, however, I won’t single anyone out over this one event. If you have an hour, you should watch for yourself (or turn it on as background noise while you do other things.

As for Janet Barresi herself, I wish I had arranged my top 20 (and growing honorable mention) list into a bingo card. Between her responses and those of her opponents, we could have called several numbers during the debate, such as:

#20 – Oklahoma’s ESEA Waiver

#19 – Hiring CTB/McGraw-Hill in the First Place

#15 – In and Out of PARCC

#14 – Value-added Measurements

I would also include today’s post, as well as six of the eight remaining from the countdown (and three off the honorable mention list).

#9 – The 2013 A-F SNAFU

In October 2013, we learned that the state superintendent probably didn’t know what the acronym snafu means. Don’t get me wrong – it was the perfect word to describe the series of mistakes made by the SDE in rolling out last year’s A-F Report Cards. I just don’t think the tone of that word fits with how she usually speaks publicly.

To refresh your memory, let’s return to October 16, 2013 and go from there. Early that afternoon, I had this to write:

We already didn’t take A-F Report Cards seriously. They are statistically unreliable. They include calculations with arbitrary weights. And they are more subject to political narratives than they are to reality.

Today, after much fanfare, schools finally received their preliminary letter grades. There were some surprises, but nothing far off the expectations that had been communicated.

Thirty minutes later, that all changed. The SDE had applied their own formula incorrectly. Apparently, someone mistook the top quartile of students in math for the bottom quartile.

They recalculated grades, and schools saw drastic changes. Many dropped by more than a letter grade.

How can anyone continue expecting people to take this seriously? In thirty minutes, we saw the biggest problem with A-F Report Cards. They depend more on the formula than they do on the students. They are useless in highlighting school performance.

For many schools, grades changed again that evening. The next day explanations were in abundance.

Yesterday, when the grades were finally released, they looked a little high. Thirty minutes later, the SDE adjusted them, and they looked really low. Several schools received the explanation that the people plugging numbers into the formula had inadvertently mistaken the top quartile of last year’s test-takers for the bottom quartile. (Ironically, it was the math scores they had miscalculated.) And they had fixed the scores. And the grades were final.

The emails actually told administrators that the grades were final.

Then last night, I started receiving messages from people telling me that the grades had changed again. Now, they were somewhere in the middle of the first two iterations.

It takes months to calculate the grades, but only minutes to re-calculate them? And then a few hours to do it again – this time with no new explanation?

If anyone has ever doubted the idea that the biggest problem with A-F Report Cards is how easy they are to manipulate, this should be the day they stop. As Rob Miller says, the report cards are DOA.

That afternoon, the SDE released a memo explaining what had happened.

***SDE*** Report Cards Update OK State Dept of Ed sent this bulletin at 10/17/2013 12:42 PM CDT Dear Superintendents, Principals and DTC’s, It came to our attention yesterday that the bottom 25 percent growth on the A to F report card was calculated incorrectly.  A last-minute correction was made immediately before posting that inadvertently caused the errors. We are working to remedy this problem as swiftly as possible, and we will notify all districts once this has been corrected. The date for submitting Data Verification Forms for calculation errors is extended until 10:00 am, Oct 28th. I deeply regret the challenges you experienced yesterday afternoon.  If additional calculation changes are needed, please submit the Data Verification Form and we will be happy to process it. Maridyth McBee, PhD Assistant State Superintendent Accountability and Assessment

As I mentioned last night, the legislature simplified the way the SDE was to calculate A-F Report Cards earlier in 2013. Still, the SDE couldn’t make it work. By the end of the second day, many districts reported being on the fifth draft of their report cards.

All of this happened over the week that most districts took Fall Break, which gave many of us ample time to analyze every version that would come out. Eventually, some schools would report seeing their scores change more than 10 times.

By day three, though, Superintendent Barresi had to assert that someone was in charge, so she released a statement of her own.

**SDE*** A-F Report Card Update OK State Dept of Ed sent this bulletin at 10/18/2013 09:42 AM CDT School districts and schools now have access to review their A-F report card and double check their data and calculations. I understand the experience of the past few days has been frustrating for school and district administrators. I am deeply sorry for the resulting delay and confusion, as are all of us at the Oklahoma State Department of Education. A commitment to transparency can have an embarrassing downside. That was certainly the case when the grades were posted Oct. 16 even as they continued to undergo several versions. A last-minute correction in the calculation resulted in errors that subsequently had to be fixed. To ensure transparency in this process, the decision was made to leave the grades up as they were modified. Fortunately, the problems were addressed and corrected during the designated 10-day window for schools, districts and the OSDE to check for errors. That doesn’t excuse the snafu, but only explains it — and we thank you for your patience. Because of the delay, districts and schools have an extended period, until 10 a.m. Oct 28, to review and seek corrections on their prescribed grades. After that point, the proposed grades will be brought before the State Board of Education at the board’s Oct. 29 meeting for final approval. Some opponents of school accountability will no doubt seize on the recent delay as yet another reason to postpone, reconfigure or simply trash the A-F report cards. Oklahoma parents, students and all interested parties can rest assured that will not happen. The annual grades are critical to heightening accountability, arming parents with important information and furthering the simple proposition that all children can learn. Janet Barresi State Superintendent of Public Instruction

Predictably, she blamed her opponents. She claimed that none of this would have happened if she – and her people – weren’t just so doggone transparent. She even faulted schools for missing deadlines.

And she called it a snafu. Rob could hardly contain his excitement. This word has been a source of hashtag fun for bloggers and social media magnates such as we are ever since.

The fun lasted until November. First, the SDE had to delay the release of the A-F Report Cards for the second straight year. Barresi complained that complaining district superintendents were engaging in “cheap political theater.” She may not know much about public education, but on that topic, she is a subject matter expert.

I’m intentionally separating last year’s debacle from the overall conversation about the utility of simplifying school performance into A-F grades. This span of time showed clearly that the SDE can’t even manage to compute its own formula correctly. For all the things we can rightfully blame on CTB (such as the #8 post in the countdown), we still have the right to expect competence out of the people running the show. Whether it’s the SDE’s failure to listen to stakeholders, poor hiring decisions, creating a working environment that talented people would rather leave, or simply choosing the wrong vendor, I struggle to find evidence of what these people can do right.

The snafus end June 24 – not a moment too soon.