Rules Not Withstanding, A-F Report Cards are Lousy

I was amused today at the editorial in the Oklahoman (that sounds really familiar). Using comments by a researcher with ties to Jeb Bush and the Foundation for Educational Excellence, the paper has decided that the work of researchers with a background in education and statistics is still bunk and that Oklahoma’s A-F Report Cards are still great.

There have always been two separate lines of criticism with the A-F Report Cards. First is the idea that you can distill everything a school does down into a letter grade. You can’t. Schools are too complex for that. Even if you accept this idea, though, there are fundamental flaws with the A-F Report Card methodology. That is where I usually focus my attention.

Before I go back to that, I want to spend a few words focusing on the first part again. We all know that implementation of A-F Report Cards is a key piece of the Jeb Bush reform movement playbook. The reason for this probably bears repeating. Parents understand that schools are complex. Every campus has something to distinguish itself from every other campus. Letter grades group all the schools together and set them up as either winners or losers. This is a necessary part of developing the narrative that public education is failing. Communities that might otherwise be content with their schools begin to doubt what they have always known.

Ancillary to this is the fact that you can do the exact same thing in two schools and get entirely different results. The kids come from different homes. The experience levels of the teachers vary. The school is a different size. There are many reasons why the same inputs do not produce the same outputs. And no matter what reform darling Steve Perry said in Tulsa Friday, poverty does matter. A great article by Carrie Coppernoll today shows what I’ve been saying about this since April.

As an example, I give you the table below, for which I have cherry-picked ten elementary schools with varying characteristics and results. (I plan to do a multi-part review of the proposed A-F rule changes this week, and I will refer to this list again for that.)

School	Free/Reduced Lunch %	Student Achievement	Student Growth	Bottom Quartile	Whole School	Letter Grade
A	3.5%	101	98	***	96	A
B	13.4%	97	91	80	96	A
C	24.8%	104	93	***	96	A
D	25.9%	100	92	97	96	A
E	30.1%	97	98	70	96	B
F	37.2%	90	96	51	96	B
G	59.8%	83	90	63	95	B
H	62.6%	101	80	78	96	B
I	68.4%	88	82	70	96	B

Looking at these ten schools – not knowing where they are, how big they are, whether they are part of a large or small school system – ask yourself two key questions:

1. Which school is doing the best job?
2. Which school would you want your child to attend?

I only picked A and B schools for a reason. If I added in C schools, you wouldn’t consider them as an answer to either question. Even if I made a strong case for why a C school somewhere is beating the odds just to get that high, and you were partially convinced that I was right, you would look at that letter grade and overlay the concerns that every parent has and look for some place different. That’s part of the plan.

Among the four A schools, there is a pretty good range of free/reduced lunch participation. Looking at the whole set of data, It’s pretty common for elementary schools with less than a quarter of its students qualifying for lunch assistance to get an A. It’s also notable that two of these schools didn’t have to count students in their bottom quartile because there were a minimal amount of students below proficient in the first place. That’s the measure that caused schools the most problems.

Among the B schools, there’s an even wider range in poverty. Is School I doing a better job than School F, considering their poverty levels? You could make that argument, but I would say there is still not enough evidence to say for certain. Just as a letter grade is weak as a singular representation of a school’s outputs, poverty is weak as a singular representation of a school’s inputs. There are always more pieces to the puzzle than data can capture. That’s the problem with accountability measures, VAM, the qualitative portion of TLE, and the third-grade retention law.

The point is that no single number or letter can tell us how effective a school is. We’re obsessed with trying to measure everything and then assign meaning to it. Ask yourself a third question: No matter what formula you use, is this good for kids?