Roster Verification: The Pilot

In this episode of Roster Verification, Janet and her friends experience some wacky shenanigans and unfortunate misunderstandings. Mr. Roper comes in and makes everybody feel terribly uncomfortable. In the end, everybody learns a valuable lesson about hubris.

In the television universe, production companies develop single episodes of new shows to try to sell a series to a network. This is called a pilot. In a typical year, about three pilots are developed for every show that airs.

An actual television pilot

In education reform, we only tend to pilot programs to which we have already committed, either through policy or contract (or both). That’s why teachers and administrators were excited this week to receive the following email from the SDE:

Roster Verification Coming Soon!In order to successfully collect data for the 35 percent quantitative portion of TLE, teachers will utilize a process called Roster Verification to properly link themselves to the students they teach.Why is Roster Verification important?  This process is important because no one is more knowledgeable about a teacher’s academic responsibility than the teacher of that classroom!  Rightfully so, teachers should have the opportunity to identify factors that affect their value-added results (e.g., student mobility and shared-teaching assignments).In order to assist teachers throughout this process, the Oklahoma State Department of Education (SDE) has partnered with Battelle for Kids (BFK), a non-profit school improvement organization. Together, SDE and BFK will provide teachers with an easy-to-use data collection instrument, Roster Verification training, and communication resources.During February, 2014 the Office of Educator Effectiveness is hosting webinars on Roster Verification.  The webinars will explain how to use the Batelle for Kids program to link students and teachers accurately.  Five sessions will be offered at various times.  We encourage administrators and/or data personnel to sign up for a session.  The same information will be covered at each session, and one session will be recorded and posted on the TLE Web page to access anytime. TLE Roster Verification Webinars Feb. 24, 1:00 – 2:30 PM;  Feb. 25, 9:00 – 10:30 AM;  Feb. 26, 3:00 – 4:30 PM; Feb. 27, 11:00 AM – 12:30 PM; Feb. 28, 9:00 – 10:30 AM To register for a webinar session, go to:  https://oksdetraining.webex.com and click on the “upcoming” tab.  Select one of the webinars titled TLE Roster Verification. As required by state statute, mandatory roster verification is scheduled for the spring of 2014 and should be completed by all districts. To learn more about roster verification, please access the following link: http://ok.gov/sde/tle-roster-verification

The catch is that 2013-14 is a pilot year. Districts must participate at all of their school sites, but they can select which teachers to use. They can use one teacher, one department, or the whole school. They are testing, more or less, how well the information tracks.

When Roster Verification is in full effect, we will eventually be able to calculate how much time each student spent with each teacher in each grade. That way, as the email suggests, we will know which teachers add the most value.

I’ve made my opinions on VAM clear before. We’re going to be making personnel decisions based on test scores. In some cases, these decisions will impact teachers in non-tested grades and subjects. Roster verification is a process by which we assign a percentage of responsibility to different teachers for a student’s growth. By responsibility, of course, I mean credit and blame.

If you’re a first grade teacher, eventually we will be able to tell you what percentage of the students you’ve taught passed the third grade reading test, took accelerated math classes in middle school, and graduated high school on time. We’ll also be able to tell you how many of your students were retained in third grade, struggled in math down the road, and dropped out.

To conduct Roster Verification (and VAM), the SDE has contracted with Batelle for Kids. Here’s how BFK describes themselves:

Battelle for Kids is a national, not-for-profit organization that provides counsel and solutions to advance the development of human capital systems, the use of strategic measures, practices for improving educator effectiveness, and communication with all stakeholders in schools.

Those who have read this blog for a while know I get twitchy around the words nonprofit or not-for-profit. Essentially, I loathe the idea that you can count as a charitable donation money you have given to an organization that really isn’t a charity.

Looking up their most recent tax form 990 on Guidestar, I found out a few interesting things about BFK. Here is some basic financial information from 2011:

Total Revenue	$21,398,999
Total Expenses	$18,761,469
Revenue Less Expenses	$2,637,530
Beginning Fund Balance	$8,896,988
Ending Fund Balance	$11,534,518

With such a healthy ending fund balance, I do hope they gave all of their employees a $2,000 raise!

This clearly is a non-profit on the rise. As I’ve said before, I don’t mind people making money. Profit is a good thing. I just abhor the doublespeak of non-profits making so much money. Where they make and spend their money is also interesting. They are heavily funded by the Bill and Melinda Gates Foundation. They have taken money from Race to the Top. Their top 11 executives all have six-figure salaries (as do two additional consultants).

Imagine the outrage if a school district in Oklahoma with a $21 million budget had 11 employees making over $100,000 (with the leader earning about $421,000). Imagine the outrage if Governor Fallin’s supporters knew that Oklahoma’s teacher evaluation system was entangled with Race to the Top, the hallmark of federal interference.

Roster verification does not benefit students. It does not give parents or teachers more information to make decisions. It simply creates additional work for already overextended teachers and principals while lining the pockets of out-of-state companies that are beholden to the corporate reform agenda.

Unfortunately, we know the network has picked up this pilot and bought several seasons worth of episodes.

Two Year Delay for TLE?

Yesterday, the SDE sent out a media release stating that they would be requesting a two year delay for full implementation of the Teacher/Leader Effectiveness system. The content of the release was linked to the SDE website, but is now down, probably due to a technical problem. The release reads as follows:

OKLAHOMA CITY (March 19, 2013) – State Superintendent Janet Barresi announced today that she will ask the authors of Senate Bill 426, Sen. John Ford and Rep. Earl Sears, to consider a two-year delay for full implementation of the Teacher and Leader Effectiveness reform that was originally enacted in 2010 in Senate Bill 2033.

In making the announcement, Superintendent Barresi stated, “After listening to input from teachers and superintendents across the state as well as teachers serving on working groups for the TLE Commission, I have concluded that this extra time is necessary to assure the entire TLE system is implemented with fidelity and to the high standards we expect of such a critical reform.”

“Nothing is more important than assuring that each child in our state has the opportunity to be taught by an effective teacher and school principal. We will continue to work with the TLE Commission and the State Board of Education to build a model program and quality technology infrastructure to support the program. I appreciate Gov. Fallin’s support in this decision and our work,” Barresi added.

Governor Mary Fallin said, “Studies show that the most important driver of student success in the classroom is high quality teachers. That’s why it’s so important that we get these reforms right. Giving Oklahoma schools adequate time to properly prepare for TLE implementation is in the best interest of everyone. I strongly support TLE and look forward to full implementation so we can utilize performance pay options and other compensation models tied to the system.”

Superintendent Barresi suggested the timeline for implementation of the qualitative or observational component of the system is currently being piloted this year and will be fully implemented in districts for the 2013-14 school year. If SB 426 passes, the Other Academic Measures portion of the quantitative component will be piloted next school year and implemented in 2014-15. The 35 percent accountability measures of the quantitative component will be implemented in the 2015-16 school year, making TLE complete.

Per state statute, the Oklahoma State Department of Education is working in conjunction with the TLE Commission and working groups of educators throughout the state to develop a robust professional growth tool known as the Teacher and Leader Effectiveness evaluation system. When fully implemented and utilized properly, TLE will identify the direct cause-and-effect relationship between teaching practices and student achievement using both qualitative and quantitative measures.

A teacher’s evaluation will be based on 50 percent of qualitative measures such as classroom observations and 50 percent quantitative measures. Of the quantitative measures, 35 percent will be based on student test scores for tested grades and subjects and the remaining 15 percent on Other Academic Measures as defined by the TLE Commission and educators. How teachers are evaluated in non-tested grades and subjects for the quantitative portion is still being discussed.

SDE staff have been very direct for months now that they would be asking for this delay, which will be appreciated by most teachers and administrators. If they need more time to work out the details, that’s fine. The cynic in me believes that something else is happening here, however.

Two things are missing at this time. First, as we heard back in January, the SDE is getting input from SAS to develop a model for creating a VAM model. (Yes, that was three acronyms in a single sentence.) Without a mathematical equation that includes factors both within and outside of a school’s control, there won’t be a value-added measure. This would make the recommendations of the various working groups meaningless at this time. The other piece in development is a more refined student data system – one that can effectively track where students were and for how long and which teachers impacted their learning and for what percentage of the time.

That brings us to roster verification, which I wrote about last week. This is a new experiment that the SDE wants to run before full implementation of TLE. If we start calculating the quantitative portion of the evaluation without these pieces, it will be harder to add them in.

A delay will ensure that VAM and roster verification will be a piece of the enacted system. It will also guarantee that we will have conversations like those taking place in Florida right now – ones in which successful teachers get low ratings because of the students they do not teach. Call me ungrateful, but rather than waiting to get it right, we should instead acknowledge that the entire concept is fatally flawed.

*****

Update: the SDE press release is back.

And Then There Was Roster Verification

I received an email last night with a 30-page document attached showing the different recommendations to the Teacher and Leader Effectiveness (TLE) Commission by the various working groups that developed them. In all, the Value Added/Student Growth Measures for Teachers of Non-Tested Grades/Subjects and Teachers Without a Teaching Assignment. In all, this document contains suggestions for how to quantify the effectiveness of 18 different classifications of certified teachers. These include:

CareerTechnology	Counselors
Early Childhood/PreK	Elementary (1-6) Non Tested
English Language Learners	Fine Arts
Gifted/Talented	Instructional Coaches
Library Media Specialists	Nurses
Physical Education	Reading Specialists/Response to Intervention
School Psychologists	Secondary: Non-Tested Subjects
Special Education	Speech Language Pathologists
Technology	World Languages

I am repulsed by the idea that we have to come up with some sort of a quantitative measure to evaluate some of these groups of teachers (and nurses, really!?!), but I decided to play along and read through the sections. What I found were some drastically disparate ways to calculate teacher effectiveness. All include some level of new training for both the teachers and the principals who would evaluate them. And most ask for more time to come up with a workable plan.

The different recommendations include some similar language that we rarely use when talking about education reform initiatives. We are going to have to learn these terms in the same ways that our students learn the academic vocabulary. I have already discussed Value-Added Measures (VAM) on this blog; I am not a fan. I do not believe that an agency incapable of developing a statistically-sound report card can develop VAM in a way that is fair to teachers. I’m not convinced that it is achievable in the first place.

Several of the proposals also call for some kind of a matrix, portfolio, or rubric to assess teachers. Principals would have to become familiar with all of these instruments. They also call at various times for different weights on the quantitative pieces of the evaluation. Imagine keeping track of all of that!

The newest term for educators, however, is Roster Verification. The only group that mentioned this process in the report to the Commission was Special Education. In an email to superintendents and principals this week, here’s how the SDE described Roster Verification:

Roster Verification – Voluntary, yet Valuable! The Oklahoma State Department of Education is offering Roster Verification as a service to school districts this spring. The OSDE will be completing value added analysis for all teachers of TESTED grades and subjects after testing occurs this spring. Value added analysis will be used for INFORMATIONAL purposes so that teachers and administrators have the opportunity to learn about the process and can use data to inform instructional practices during the 2013-2014 school year. This is a NO STAKES process meaning NONE of the value added calculations will be used in evaluations. Because there are so many different teaching scenarios that occur throughout the year, Roster Verification allows teachers to account for who they taught, for which months during the year, and for what percentage of the instructional time. For example, when I taught 5th grade, we were departmentalized. I was responsible for MANY students’ instruction in mathematics and science, but my team member was responsible for their instruction in reading and social studies. Without completing Roster Verification, my value added analysis would be based on my HOMEROOM roster (unless someone uploaded that information differently into the Wave.) As a teacher, I would want to be held accountable for the growth of the students I instructed in math and science, but I would want my partner to be responsible for their growth in reading and social studies. Roster Verification gives teachers the ability to account for such scenarios, therefore value added analysis reports are much more accurate for teachers who were able to complete the Roster Verification process.

The SDE provides even more detailed information on this flyer, which includes training dates, a shout out to the Gates foundation for funding, and a picture that would lead you to believe this is about children. One line even promises that Roster Verification will provide “much more accurate value added reports which will be extremely useful as a professional growth tool.”

This is not remotely about professional growth. This is about continuing down the path of assigning blame, and trying to find a mathematical formula for doing so. In ten years, we will be able to look at the students of two second-grade teachers and see which ones are better prepared for college. We will be able to assign partial credit/blame for the success/lack-thereof to all the teachers those students have ever had. Over time, we’ll have all kinds of data pointing back to that second grade classroom.

Think back to that bizarre oak tree analogy. The disembodied voice in the video tells us that countless factors go into two farmers raising their oak trees. It also tells us that some of those factors are out of the control of the farmers. (By the way, I still don’t know any oak tree farmers).  Removing all of those factors, the argument is that we can tell which farmer is adding more value. Extending the analogy, we can remove the factors out of the control of teachers and look at results, and ascertain value-added that way as well. This too makes me uncomfortable.

The factors out of a teacher’s control are too many to count. We will be assigning value-added sometimes ten or twelve years after the fact. Would you want part of your evaluation during your eleventh year in the profession to be based on something you did your first year?

Many districts have chosen not to participate in Roster Verification at this time. Others, for the sake of curiosity, are joining in the trial run. I understand both positions. While I can’t think of one school administrator who wants to see this happen, many want to see what the process looks like since it will happen eventually anyway.

The recommendations to the TLE Commission are non-binding. Commission members can act to accept, revise, or reject the proposals at a later date. Meanwhile, the SDE is wisely pushing for more time to implement the quantitative piece of the evaluation system. While they would be even wiser to scrap it altogether, that won’t happen. Too much money –taxpayer and corporate money – is invested at this point. The agency is philosophically entrenched in this process.