Archive

Archive for February, 2013

Questions I Hope the State Board of Education Will Ask

February 28, 2013 Comments off

Today’s State Board of Education meeting should be interesting. Right off the bat, during the superintendent’s report, the agenda calls for “Discussion [of] the A-F Report Card examination study conducted by the Oklahoma Center for Education Policy and The Center for Educational Research and Evaluation.” That’s a nearly-grammatical mouthful, which is why I usually just refer to it as the OU/OSU Report.

Expect these lines of defense to be pushed forward, maybe by board members rather than Superintendent Barresi herself:

  1. The study was commissioned by CCOSA and OSSBA, which represent administrators and school board members; of course it’s negative.
  2. Other researchers have found our accountability process to be a model for other states.
  3. We’ve spent a year listening to the input of critics and adjusting the rules accordingly; they’re just never going to be satisfied.
  4. This really comes down to districts not wanting to do their jobs or using poverty as an excuse for poor performance.
  5. The conversations that have begun in communities because of these report cards would not have happened otherwise.

Starting from the beginning, allow me to refute these:

  1. Whoever commissioned the study, it was performed by scholars, who then went to external reviewers out-of-state to validate their findings. What they found was in fact predictable – not because it was an inside job, but because we’ve all been saying the same things for months, just not in such strong, academic language.
  2. The one researcher the SDE and its defenders clings to right now, David N. Figlio of Northwestern (Chicago, not Alva) has praised Oklahoma’s system, even though he previously called Florida’s “unrelated to the school’s contribution to the school’s contribution to student performance” and said that the results contained “inherent randomness.” Oklahoma’s accountability system, in case you haven’t heard, is modeled after Florida’s.
  3. Yes, school people complain a lot. There also is a group of legislators in Barresi’s party who have listened to their constituents and want the A-F Rules thrown out. The revised rules proposed last week are a slap in the face to constructive dialogue. In truth, the people at the SDE writing these rules and running this program are listening to their supporters out-of-state rather than the people they purport to serve.
  4. Even the Oklahoman over the weekend ran a story showing the linkage between poverty and school performance. They also discussed one school that in statistical terms would be considered an outlier. The exception neither proves nor disproves the validity of the accountability process. It should be noted, by the way, that many superintendents of districts with great results also disliked the A-F process.
  5. If anything, the report cards have stopped all conversation. Given an easy-to-understand answer, people tend to ask few questions. There have been plenty of grass roots conversations about the flawed process and product, however.

All that said, I hope today’s conversation is fruitful. If I were an inquisitive board member (and there is at least one), I would ask the following questions:

  • What influence did groups like the Foundation for Educational Excellence have in creating this process?
  • How does the SDE respond to the statistical concerns – no recognizable metric, introduction of grouping error, the threats to validity – discussed in the report?
  • Why did Superintendent Barresi tell a group of parents that the researchers had changed their mind when nothing can be further from the truth?
  • Why do the proposed rule revisions address none of the research concerns?
  • Has the SDE noticed that one of the rule revisions would appear highly-punitive to high-performing schools?
  • What will the SDE do if the legislature in fact follows through with last week’s committee action and throws out the existing rules?
  • If some of the more problematic parts of the A-F rules are tied directly to pieces of the state’s NCLB waiver, will the state work to extricate itself from those regulations?

If today is contentious, it will be because some of these questions are being asked. If it is merely an assault on the SDE’s critics (as I think it might be), that will be interesting in its own right. In any case, check back this evening. I’ll be eager to give you my thoughts on how it goes.

A-F Proposed Revisions Part Four: Whole School Improvement

February 27, 2013 1 comment

A quick review of where we are right now…

A couple of weeks ago, Superintendent Barresi misled parents to believe that the OU/OSU researchers had changed their minds and fallen in love with the A-F Report Cards.

Last Wednesday, a House committee voted 10-1 to abandon the adopted rules for A-F Report Cards.

Friday, the SDE issued new rules, along with a statement that they have heard all of our concerns.

Later Friday, I read the proposed rule changes and concluded that in some sections, nothing is changing. In others, things are getting worse.

Sunday, the Oklahoman ran an editorial saying that not everybody has a problem with the report cards. As evidence, they cited Northwestern economics professor David N. Figlio, who called our system “an exemplar of systems of its type and a model for other states and jurisdictions to follow.” Figlio, is a favorite of Jeb Bush and the Foundation for Education Excellence. He is also a researcher who has previously written critically of accountability systems:

There is good reason to believe that school accountability systems, both in Florida and nationally, evaluate schools very noisily….due to measurement problems, schools that are rewarded in systems such as the new federal program will tend to be punished in the next, suggesting that school rankings under these systems tend to be quite unstable….school grades under systems such as that introduced by the federal accountability plan are highly uncorrelated with any conception of school ‘value added’ that economists typically consider. Taken together, these papers indicate that school accountability measures such as the Florida system and the new federal system are largely unrelated to the school’s contribution to student performance and are likely measured with considerable noise. This inherent randomness, as well as the disconnection between school grades and what these grades seek to measure, provide the impetus for the current study.

He goes on to find that perceptions of school grades can impact housing prices and where people choose to relocate. So for all of us who have ever moved and looked for the “best schools” for our kids, we’ve probably been fooling ourselves with noise and inherent randomness. In truth, I can’t reconcile his quote from the Oklahoman with his statements above.

What will be interesting is to see how Superintendent Barresi responds to all of this criticism Thursday at the State Board of Education meeting. Discussion of the OU/OSU report is on the agenda pretty early. If I were a board member, I’d want an honest appraisal of the report and an explanation of Barresi’s comments to parents – something better than calling it a misunderstanding.

Now, back to the review of the proposed rule changes.

If the Bottom Quartile Growth section is frustrating to schools, the Whole School Improvement section is downright confusing. There are grade span differences. Some criteria are weighted more heavily than others. And the use of surveys for bonus points (now eliminated) made schools wonder if their time was just being wasted (it was).

Let’s start with the name of this section. Why is it Whole School Improvement? If you’ve seen the table of ten elementary schools I’ve used several times this week, you may have noticed that all had either a 95 or 96, which is simply a measure of attendance. That’s right, elementary schools got 33 percent of their A-F Report Card grade on one statistic that is largely attributable to factors outside of their control. This hardly captures the whole school climate and has nothing to do with improvement.

There are no rule changes listed for elementary schools, but the guidelines issued by the SDE in the fall indicate that for the 2012-13 school year, elementary schools will also have to include dropout rate as an indicator of Whole School Improvement. The dropout rate, however, will only count for four percent of this section. This means that a school with an A for attendance will likely have an A for whole school improvement, no matter what the dropout rate is. In other words, a new reporting measure has been added, but it will not change a school’s section grade or overall letter grade.

The revisions for middle schools leave intact the three criteria that were in place last year for Whole School Improvement: attendance, higher level coursework, and dropout rate. The proposed revisions clean up the coursework section so that schools receive more credit for students taking multiple advanced or honors courses.

The high school indicators for Whole School Improvement have undergone the greatest revisions, and they are not all bad. The two most notable changes are allowing schools to count more than one advanced course for students (as with middle school) and merging the AP performance criteria with the advanced coursework performance criteria. This includes the industry standard (Career Tech) component, which is a pretty good change. For some unknown reason, the ACT/SAT performance indicator will count the most recent test taken, instead of the highest one.

One fundamental problem with the rules – both as written before and as now revised – is that they do not assign weights to the different indicators in each section. While we knew last spring that high schools would have eight indicators, we were not aware, however, that one of them would be worth 79 percent of Whole School Improvement and that the other seven would not collectively have the statistical power to override that impact. Apparently, even after state law passes and administrative rules are adopted, the SDE has the authority to do whatever it pleases. How’s that for accountability?

Meanwhile, we have until March 25th to comment on the proposed rules.

A-F Proposed Revisions Part Three: Bottom Quartile Student Growth

February 27, 2013 6 comments

I’m taking a few posts this week to highlight the new proposed rules for A-F Report Cards issued last week by the State Department of Education. Part One looked at the Student Achievement section of the report cards (first 33 percent), containing no changes whatsoever. Part Two examined the rule change to the Student Growth Index, which merely solidifies the controversial practice of calculating average growth without a real average. This post will discuss changes to the calculation of the Bottom Quartile Growth.

These are a big deal, and not to spoil the ending for you, not very well thought out.

Last year, this 17 percent of the Report Card included the matched scores (from one year to the next) of the lowest 25 percent of students in the school. The catch was that if a school had less than 25 percent of its students failing the tests, this section would only count the students below proficient. For many schools receiving an A, this led to a negligible number of students needing to show improvement. In fact, using the sample of ten elementary schools I put together last weekend, we can see that two of them did not have enough students to count for this section.

School

Free/Reduced Lunch %

Student Achievement

Student Growth

Bottom Quartile

Whole School

Letter Grade

A

3.5%

101

98

***

96

A

B

13.4%

97

91

80

96

A

C

24.8%

104

93

***

96

A

D

25.9%

100

92

97

96

A

E

30.1%

97

98

70

96

B

F

37.2%

90

96

51

96

B

G

59.8%

83

90

63

95

B

H

62.6%

101

80

78

96

B

I

68.4%

88

82

70

96

B

That is all changing. The proposed rules read this way:

Improvement of the lowest twenty-five percent (25%) of students in reading and math shall be aggregated, unless the students in this category are exhibiting satisfactory performance, as defined by scoring Satisfactory, Proficient or Advanced as required by 70 O.S. § 1210.545. The score shall be calculated in whole and by subject-matter by assigning points for a positive change in proficiency score for eligible students from the previous school year to the current school year or by a positive change in Oklahoma Performance Index (OPI) score that meets or exceeds the State’s positive average growth change.

The underscores are additions. The strikethroughs are deletions. This will mean that schools at the top – the ones with few students struggling on the state tests – they will have to show growth as well. This is the problem that high-performing schools have feared all along, and rightly so.

I’ve written (endlessly, some would argue) that poverty impacts performance. Never have I suggested that we make things harder for the schools that have less poverty. This rule change would do exactly that, and without good reason. As written, students aggregated for this calculation would be counted according to the following scale:

  • Unsatisfactory to Limited Knowledge: 1 point
  • Unsatisfactory to Proficient: 2 points
  • Unsatisfactory to Advanced: 3 points
  • Limited Knowledge to Proficient: 1 point
  • Limited Knowledge to Advanced: 2 points
  • Meets or Exceeds State Average Positive Change: 1 point

Do you see what’s missing? What happens to the students who were proficient or advanced last year, but were in the bottom 25 percent of all schools for a particular school? Don’t shake your head; it happens. Any student in the aggregate calculation not accounted for in the scale above will count for zero points and lower the school’s average. A student who was proficient last year and remains proficient this year will actually lose points in this section.

One problem I highlighted with Bottom Quartile Growth in the fall was that there is already a strong difference in how the calculation affects schools by grade span. In simpler terms, elementary schools are hit harder. This is mostly due to the fact that the cut scores for the reading and math tests are set higher in third, fourth, and fifth grade than they are in sixth, seventh, and eighth. The proposed change to this section will only amplify this effect.

Again I go back to the main takeaway from the OU/OSU report. These scores are already “meaningless for school improvement purposes.” Proposed revisions to sections one and two do absolutely nothing to fix this. What they’re doing to part three makes things worse.

*****

Note – I will try to have Part Four up this evening and a quick preview of tomorrow’s State Board of Education meeting up by morning. Then I’ll likely have a recap of the meeting Friday. If I do a February recap/March preview post, it may have to wait until Saturday.

A-F Proposed Revisions Part Two: Student Growth

February 26, 2013 4 comments

In Part One, I discussed the fact that this round of proposed rules contains no changes to the Student Achievement part of the A-F Report Card, in spite of the evidence that this component is based on a scale that is completely arbitrary. This section will look at changes to the student growth component, which counts for the next 17 percent of the report card.

The Student Growth Index tracks reading and math scores from one year to the next. No other subjects are counted. One unfortunate by-product of this is a reduction of emphasis on all other academic content areas. Science, social studies, music, art, physical education, computer instruction, world languages – they all are afterthoughts, especially at the elementary level.

As written last year, the index was calculated for all students for whom the state data system could create a matched pair. If Johnny tested in 3rd grade in Choctaw, then tested in 4th grade in Choteau, the system would find him and match the pair.

One concern was that if students already scored highly, schools would struggle to have a high index. This proved not to be true, as each matched pair received points according to the following scale:

  • Unsatisfactory to Limited Knowledge: 1 point
  • Unsatisfactory to Proficient: 2 points
  • Unsatisfactory to Advanced: 3 points
  • Limited Knowledge to Proficient: 1 point
  • Limited Knowledge to Advanced: 2 points
  • Proficient to Proficient or Advanced: 1 point
  • Advanced to Advanced: 1 point

None of this was terribly controversial. It was the next part that caused most of the debate over methodology:

  • Meets or Exceeds State Average Growth: 1 point

Newspapers, education groups, and even the governor weighed in with opinions of what that meant. Is it really growth if you’re measuring all students (in which case the growth was negative)? Or is it really an average if you leave out students who stayed the same or declined? Either way, you’re excluding a part of the story.

(Though I am typically wary to use sports analogies, I think it is worth noting that as of tonight, the Oklahoma City Thunder have a 9.1 average margin of victory, and that the NBA uses all games – not just games won – in calculating that statistic.)

The rule change here is subtle. They change this section to read “Meets or Exceeds State Average Positive Change.” It is supported above by two paragraph additions:

The score shall be calculated in whole and by subject-matter by assigning points for a positive change in proficiency level for eligible students from the previous school year to the current school year or by a positive change in Oklahoma Performance Index (OPI) score that meets or exceeds the State average of students with a positive OPI change.

And under the lead in “eligible students who have” the section also adds:

Improved their state standardized assessment achievement level or state standardized alternative assessment achievement level and such change in OPI from the previous school year to the current school year met or exceeded the State average of students with a positive OPI change.

The only thing notable about these changes is that the SDE is now codifying the method they used last year, when they did whatever they wanted without any real input. Taking what they did previously and making the rule read that explicitly, Barresi is hardly living up to her comments from Friday’s press release (the one with the wrong date). The part where she attributes these changes to “concerns previously expressed by education stakeholders across the state” rings completely hollow.

This rule change in fact is no change at all. It is semantic at best, like something Shakespeare would have enjoyed.

Abraham: Do you bite your thumb at us, sir?
Sampson: I do bite my thumb, sir.
Abraham: Do you bite your thumb at us, sir?
Sampson (to Gregory): Is the law of our side if I say ay?
Gregory: No.
Sampson: No, sir, I do not bite my thumb at you sir; but I bite my thumb, sir.
Gregory: Do you quarrel, sir?
Abraham: Quarrel, sir? No, sir.

Part Three will discuss the third section of the Report Cards – Bottom Quartile Student Growth. This will be the greatest concern yet.

A-F Proposed Revisions Part One: Student Achievement

February 25, 2013 3 comments

The proposed rule changes are alarmingly scant in reflection of the concerns raised by more than 300 school superintendents last fall. And these changes are dramatically short of the complete overhaul suggested by the OU-OSU report last month.

Sapulpa Superintendent Kevin Burr

Rather than just trying to patch and tape and fix, I thought it would be better to just wipe the slate clean and start over with a new set of rules, particularly after the research documented fundamental flaws with the system.

 Rep. Gus Blackwell, R-Laverne

Those opposed to A-F can twist it around. They throw up their hands and go back to their offices and close the door. That’s not acceptable. Children in poverty have special challenges, so how are we going to get them to proficiency?

State Superintendent of Public Instruction Janet Barresi

As I mentioned Friday, the SDE has issued new proposed rules for the A-F Report Cards. The public comment period is open until March 25. Hopefully this time, they will listen to critical input.

The rollout of school grades last October was a disaster, and frankly, a waste of time. By the time schools received their reports, their improvement efforts were well under way. Schools typically take the earliest achievement data they get and use this information to start planning for remediation and intervention. Nothing in the A-F Report Cards was what you would consider new information. Accordingly, they changed very little. This does not mean, however, that teachers and principals are throwing their hands up, closing the door, and giving up. Anyone who spends meaningful time in schools knows this.

The proposed rules offer little in the way of substantive change. I’m going to take multiple posts this week to go through the four sections of the report card. That way I can be thorough and not just sit at the computer and punch out a 5000 word blog post. My critique will include original concerns expressed last spring with the first draft of the A-F Rules, ongoing concerns from their release last fall, and concerns of researchers from both the OCU report and the OU/OSU report.

As you’ll recall, the A-F Report Cards are divided into four sections:

  • Student Achievement (33%)
  • Student Growth (17%)
  • Bottom Quartile Student Growth (17%)
  • Whole School Performance (17%)

I made a quick table of ten elementary schools yesterday to illustrate some trends in these calculations. I’ll use that throughout the week as I go through the different sections. I may add a different table for secondary schools when I get to the Whole School Performance discussion.

School

Free/Reduced Lunch %

Student Achievement

Student Growth

Bottom Quartile

Whole School

Letter Grade

A

3.5%

101

98

***

96

A

B

13.4%

97

91

80

96

A

C

24.8%

104

93

***

96

A

D

25.9%

100

92

97

96

A

E

30.1%

97

98

70

96

B

F

37.2%

90

96

51

96

B

G

59.8%

83

90

63

95

B

H

62.6%

101

80

78

96

B

I

68.4%

88

82

70

96

B

It doesn’t really matter which schools these are, where they are, or how big or small they are. For the sake of this discussion, the poverty column is also irrelevant. The Student Achievement factor consists of all test scores for a school – not just reading and math, and not just regular education. Modified tests count too. Scores are calculated following this math:

  • Advanced 1.2 points
  • Proficient 1.0 points
  • Limited Knowledge 0.2 points
  • Unsatisfactory 0 points

The OU/OSU report complained that these scores do not seem to follow any recognizable metric. In theory, if every tested student scored exactly Proficient, the Student Achievement factor would be 100. Falling to Limited Knowledge causes a significant drop. Increasing to Advanced causes a minor gain. This scale is arbitrary. Looking at the numbers above, can you tell me anything about the percentage of students passing the tests?

This is actually an improvement over the first draft of rules that the SDE issued. At that time, Limited Knowledge was worth nothing. In any case, these scores fail to recognize that student achievement exists along a spectrum. The cut points for each test are somewhat arbitrary themselves. That is what makes assigning scores this way so meaningless.

The OU/OSU report also expressed concern about the fact that this treatment of the scores introduces grouping error into the results. Nothing in the revision issued last week corrects for this. In fact, the first section of the report cards is completely unchanged. This tells me that the SDE feels like they got this right the first time.

I should mention that one event this year will likely cause scores to drop. The social studies tests are being revised and given to students as a field test only. Last year, most schools got a boost from scores on those tests, which typically have very high scores.

In Part Two, I will look at proposed changes to the Student Growth section, which was the focus of a last-minute compromise that the governor’s office almost ironed out between school leaders and the SDE. I’ll even get into a discussion of why the proposed solution really wasn’t that beneficial.

Rules Not Withstanding, A-F Report Cards are Lousy

February 24, 2013 Comments off

I was amused today at the editorial in the Oklahoman (that sounds really familiar). Using comments by a researcher with ties to Jeb Bush and the Foundation for Educational Excellence, the paper has decided that the work of researchers with a background in education and statistics is still bunk and that Oklahoma’s A-F Report Cards are still great.

There have always been two separate lines of criticism with the A-F Report Cards. First is the idea that you can distill everything a school does down into a letter grade. You can’t. Schools are too complex for that. Even if you accept this idea, though, there are fundamental flaws with the A-F Report Card methodology. That is where I usually focus my attention.

Before I go back to that, I want to spend a few words focusing on the first part again. We all know that implementation of A-F Report Cards is a key piece of the Jeb Bush reform movement playbook. The reason for this probably bears repeating. Parents understand that schools are complex. Every campus has something to distinguish itself from every other campus. Letter grades group all the schools together and set them up as either winners or losers. This is a necessary part of developing the narrative that public education is failing. Communities that might otherwise be content with their schools begin to doubt what they have always known.

Ancillary to this is the fact that you can do the exact same thing in two schools and get entirely different results. The kids come from different homes. The experience levels of the teachers vary. The school is a different size. There are many reasons why the same inputs do not produce the same outputs. And no matter what reform darling Steve Perry said in Tulsa Friday, poverty does matter. A great article by Carrie Coppernoll today shows what I’ve been saying about this since April.

As an example, I give you the table below, for which I have cherry-picked ten elementary schools with varying characteristics and results. (I plan to do a multi-part review of the proposed A-F rule changes this week, and I will refer to this list again for that.)

School

Free/Reduced Lunch %

Student Achievement

Student Growth

Bottom Quartile

Whole School

Letter Grade

A

3.5%

101

98

***

96

A

B

13.4%

97

91

80

96

A

C

24.8%

104

93

***

96

A

D

25.9%

100

92

97

96

A

E

30.1%

97

98

70

96

B

F

37.2%

90

96

51

96

B

G

59.8%

83

90

63

95

B

H

62.6%

101

80

78

96

B

I

68.4%

88

82

70

96

B

Looking at these ten schools – not knowing where they are, how big they are, whether they are part of a large or small school system – ask yourself two key questions:

  1. Which school is doing the best job?
  2. Which school would you want your child to attend?

I only picked A and B schools for a reason. If I added in C schools, you wouldn’t consider them as an answer to either question. Even if I made a strong case for why a C school somewhere is beating the odds just to get that high, and you were partially convinced that I was right, you would look at that letter grade and overlay the concerns that every parent has and look for some place different. That’s part of the plan.

Among the four A schools, there is a pretty good range of free/reduced lunch participation. Looking at the whole set of data, It’s pretty common for elementary schools with less than a quarter of its students qualifying for lunch assistance to get an A. It’s also notable that two of these schools didn’t have to count students in their bottom quartile because there were a minimal amount of students below proficient in the first place. That’s the measure that caused schools the most problems.

Among the B schools, there’s an even wider range in poverty. Is School I doing a better job than School F, considering their poverty levels? You could make that argument, but I would say there is still not enough evidence to say for certain. Just as a letter grade is weak as a singular representation of a school’s outputs, poverty is weak as a singular representation of a school’s inputs. There are always more pieces to the puzzle than data can capture. That’s the problem with accountability measures, VAM, the qualitative portion of TLE, and the third-grade retention law.

The point is that no single number or letter can tell us how effective a school is. We’re obsessed with trying to measure everything and then assign meaning to it. Ask yourself a third question: No matter what formula you use, is this good for kids?

These are not the Rules You’re Looking For

February 22, 2013 4 comments

You probably heard that a House committee voted 10-1 Wednesday to throw out the existing A-F Report Card rules because they are “not consistent with legislative intent.” I’m not so sure how true that is. We’d probably have to ask Jeb Bush, since both the law and the administrative rules were written by groups that he controls. In any case, the committee vote changes nothing. The full legislature would have to vote to throw the rules out, and the governor would have to approve as well.

The Oklahoma State Department of Education responded quickly. In a press release issued today, they announced that a new set of rules were available on their website for public review and comment. Interestingly, the release is dated February 27, 2013, which could be a mistake. It could also be an indication that the SDE was set to release these revised rules next week, but rushed them to completion with the political firestorm they have been enduring. From the press release:

OKLAHOMA CITY (Feb. 27, 2013) – Today the Oklahoma State Department of Education released for public comment several proposed changes to the A-F Report Card rule. These proposed changes are the result of concerns previously expressed by education stakeholders across the state including board members, superintendents, administrators, teachers, parents and state legislators.

Unfortunately, this latest iteration of rules is anything but responsive to previously expressed concerns. In October, I summarized my concerns into five categories:

  • There are vast differences in the scores earned by schools based on poverty levels.
  • The grade span disparity (easier to score well for secondary schools) diminishes the comparability between schools within districts.
  • Certain criteria were too heavily weighted, creating imbalances that adversely impacted scores.
  • Some schools serve special populations, and the formula does not account for these situations.
  • The grades fail to give parents simple information, such as the percentage of students passing tests.

Nothing in the proposed rules addresses these concerns.

Of course I’m not the only critic of the report cards. The OU/OSU research report called the grading system “meaningless for school improvement purposes.” Barresi has since told a group of parents that the researchers have changed their mind, which is completely false. More recently, a group of 25 superintendents asked for a public response to the report from Barresi, which they will never get.

The collective response to these rules changes should be outrage. This new draft solidifies the bad things and clarifies little. The practical concerns issued here and elsewhere – as well as the statistical and methodological concerns raised by the researchers – need to be repeated to the SDE, to your legislators, and to every media outlet in Oklahoma that still practices journalism.

%d bloggers like this: