Should you vote for Opportunity School District? Depends on how much you trust state’s charter school record.

Jarod Apperson writes Grading Atlanta, a terrific education blog that goes deep into data and research.

In this new essay from his blog, Apperson discusses reasons to vote for and against Amendment 1, the Opportunity School District.

Apperson graduated from New York University with a B.S. in finance and accounting and is pursuing a Ph.D. in economics from Georgia State University. I think both opponents and proponents of the Opportunity School District will consider this a fair analysis. Please note I could not reproduce the cool interactive charts Apperson created so I have repeated the link to his blog whenever he cites the charts. Go to Grading Atlanta to check out his charts.

In alerting me to his analysis, Apperson said, “I’m sure you are probably suffering from OSD fatigue at this point.”

While I’ve been hearing about the OSD from advocates and opponents on the front lines, I haven’t heard many “regular” Georgians raise the issue until this weekend when a half-dozen people asked me about it. I believe the blitz of pro and con TV commercials has increased awareness.  I even saw several yard signs around metro Atlanta.

With that, here is Apperson’s commentary:

By Jarod Apperson

Depending on which ad you’ve seen, Gov. Nathan Deal’s Opportunity School District is either a white knight coming to save public education or a headless horseman coming to pillage the state’s most vulnerable communities.

Such simplistic appeals are inevitable when the general population is asked to vote on an issue that is complicated and requires a great deal of background knowledge to engage with substantively.

As someone with expertise in this area, I feel comfortable saying that frankly we don’t know how this endeavor might turn out if it is approved. There is a real possibility the OSD will improve education, there is a real possibility it will have little impact, and there is a real possibility it will do harm.  An informed vote for or against the OSD depends on which of those possibilities you think is most likely and the extent to which you believe the state should take a risk.  Below I give my take on several key questions and lay out the best available evidence.  I will leave it to readers to weigh the evidence, which points in different directions, and reach their own conclusions about the OSD’s prospects.

What will the OSD do?

The gist: Turn over the management of selected schools from the local school district’s central office to a charter operator selected by an appointee of the Governor.

The detail: Voters will approve or deny the OSD by voting on Amendment 1, appearing on ballots statewide with the following language:

Provides greater flexibility and state accountability to fix failing schools through increasing community involvement. Shall the Constitution of Georgia be amended to allow the state to intervene in chronically failing public schools in order to improve student performance?

Anyone with knowledge of the OSD will recognize this statement fails to paint a clear picture of what the initiative hopes to do.  Based on this description, one could be forgiven for believing Amendment 1 hoped to raise student achievement by encouraging more community bake sales.  That’s not the plan.  But the overly genial language alone doesn’t mean it is a bad idea.

The authorizing legislation spells out more clearly the tools the state will have at its disposal when intervening in schools. They include managing the school directly, stipulating changes the local school district must make, shutting the school down, and selecting a charter organization to operate the school.  It is clear the governor’s preferred course is to select charter organizations to operate the schools, a model used in Louisiana and Tennessee, states that inspired the proposal.

Now that we are clear on what the OSD hopes to do, the most pressing question comes down to whether OSD-eligible schools will be better off or worse off managed by charter organizations. I’ll come back to that discussion in a moment, but first I want to talk a bit about the identification of OSD-eligible schools.

Does the OSD do a good job of identifying low-quality schools?

The gist: Sort of, but it more consistently picks up high-poverty schools than low-quality schools.

The detail: Each year, the state puts out a score it calls College and Career Ready Performance Index (“CCRPI”), which is mostly based on crunching standardized test scores different ways.  This metric forms the basis for schools being selected for the OSD. Any school that scores below a 60 for three years in a row becomes eligible.  One reasonable  critique of CCRPI is it doesn’t do a very good job of comparing schools to their peers — other Georgia schools that serve similar students.

Instead, it systematically ranks schools with poor students low and schools with relatively rich students high.  In reality, there are low-quality and high-quality schools at all income levels. (See Apperson’s interactive chart here that presents a better measure of school quality and poverty for all schools in the state.[1]  Highlighted schools are schools that rank in the bottom 6% (the share of Georgia schools that are OSD-eligible) of student growth relative to peer schools.

It is clear the variation in quality at the high-income end is just as wide as the variation at the low-income end. Because the OSD relies on CCRPI rather than the school quality measure presented above, schools deemed eligible tend to systematically be poor schools, rather than schools that have achieved the lowest academic gains relative to their peers. I have created the same chart of quality and poverty, but highlighted the OSD-eligible schools rather than the schools in the bottom 6%.

There is no school without at least 35% of students in poverty that qualifies for the OSD. In contrast, one out of every three schools with more than 60% of students in poverty is on the OSD list. So having a sizable share of poor students is essentially a prerequisite for being selected.  Still, the schools chosen do tend to be below average quality. They may not be the worst schools in the state (and some even have high growth!), but they average around the 28th percentile.

This discussion so far about school quality – as measured by both the CCRPI and my own approach – relies on test scores.  But we ultimately care about whether schools prepare students for successful lives, not whether they can score well on a bubble test at the end of third grade. That brings me to the next question.

Are standardized test scores good metrics for measuring school quality?

The gist: Generally yes, but not always.

The detail: Over the past five years, the relationship between test score gains and long run outcomes has been a topic of great academic interest.  The most compelling evidence suggests teachers and schools that are able to achieve high growth on test scores cause their students to succeed later in life. However, it is also possible for schools to raise test scores using means that do not impart the skills necessary for later success.

There have been three major papers presenting high-quality evidence that schools and/or teachers who are able to raise test scores ultimately cause their students to have better long run outcomes.  Chetty et al. (2014) shows that high-growth New York City teachers reduce teen childbearing, increase college going, and increase earnings at age 28.  Dobbie & Fryer (2015)shows that a high-scoring Harlem school reduces teen pregnancy and incarceration rates. Angrist et al. (2016) finds that Boston charters able to raise test scores also increase four-year college going.

A fourth study finds more mixed evidence.  Dobbie & Fryer (2016) analyze Texas charter schools.  They find schools that negatively affect test scores also negatively affect four-year college enrollment and earnings (consistent with findings from the studies above). However, in contrast to the other evidence, schools that are able to raise test scores do not improve long-run outcomes.  One possible explanation the authors provide is that the high-scoring schools in the study may have focused too narrowly on tested skills, taking time away from the development of non-tested skills important for long-run success.

Collectively, these papers suggest test scores are a good proxy for whether schools and teachers are imparting the skills students will need to succeed; however, they also suggest it is possible for schools to achieve high scores without developing those skills.

If test scores are a meaningful measure of skill development and OSD-eligible schools do not now succeed at raising test scores (recall that they on average rank at the 28th percentile in quality), the logical next question is should we expect the schools to do any better if they were taken over by the OSD. Since the governor’s preferred intervention is to select charter operators, the answer hinges on the quality of those operators.

What is the evidence on how charter schools currently operating in Georgia affect standardized test scores?

The gist: Local charters are slightly above average, state charters are significantly below average, and within both groups there is a great deal of variation from school to school.

The detail: Georgia now has about 60 start-up charter schools that operate in grades tested annually (grades 3-8 take Milestones End of Grade Tests).  Before they opened, those schools were reviewed and approved by either the local school board (“Local Charters”) or the State Charter School Commission (“State Charters”).

On average, the charter schools now operating in Georgia are lower quality than traditional public schools. Much like traditional schools, the quality varies a great deal.  Some of the best schools in the state are charters.  Some of the worst schools in the state are charters. Go here to see the same chart of school quality and poverty we looked at before, but now local and state charters are highlighted.

If the OSD could ensure the charter operators it partnered with would achieve results similar to the four KIPP schools (all are 98 or above on the quality measure, compared to 28 for the OSD schools), voting for the amendment would be a no brainer.  But that is probably optimistic to say the least. Most charter applicants don’t come with a proven track record, making it tough for authorizers to ensure quality at the time charters are approved.

If instead, the OSD were to partner with schools similar to the average state-approved charter, schools taken over would likely end up achieving at even lower levels than they are today (State charters’ average quality is 12, even lower than the 28 for OSD schools).  For me, this uncertainty about quality is what causes the most skepticism of Amendment 1’s prospects.

Will the OSD charter operators be like the shining examples of what is possible (KIPP) or will they be subpar (like the average state charter)?

There are some reasons to believe the OSD charter partners will be more successful than state-approved charter schools. First, the funding will be higher. State approved charters are funded at a rate lower than most nearby traditional public schools, and they have to spend part of their funding on facilities. The OSD will fund schools like locally approved charters and give them facilities. Second, the OSD will be tasked with seeking out high-quality charter operators. Depending on how savvy the OSD leader is, he or she may find partners with proven track records elsewhere in the country.

On the other hand, there are reasons to believe the OSD charter partners will be of similar quality to the state-approved charter schools (i.e. worse than the OSD schools themselves). First, there is a limited pool of people capable of starting a high-quality charter.

If anything can be learned from the gap between the results from locally approved charters and state charters, it is probably that good charters tend to get approved locally. It takes an incredible amount of time and dedication to run a successful charter school.

My sense is the size of the high-quality charter school community is more constrained by the number of leaders capable of developing and implementing a strong plan than it is by local districts unfairly rejecting great proposals. If that’s indeed the case, the OSD will likely struggle to find great operators. Those out there are already opening local charter schools.  Second, it appears the OSD may be biting off more than it can chew.

The proposal would allow the OSD to take over up to 20 schools a year (the agency could elect to take over fewer schools). The scope of that potential undertaking is striking given there are only about 20 good charter schools in the whole state today and it took almost two decades to get here. The notion the OSD could open 20 schools of good quality in a single year seems tenuous. I would feel more comfortable if the plan was two per year, rather than 20.

At the end of the day, I think the governor has good intentions and wants to see the OSD-eligible schools improve for the kids who attend them. I don’t buy the narrative he is looking to exploit children to profit his friends (though I do think there are organizations out there who would like to profit from the initiative). I also believe there is plenty of room for improvement at OSD schools.

But I am less confident the OSD will partner with charter organizations capable of delivering that improvement.

If Georgia had a history of holding its charter schools to a high standard, I would feel more comfortable supporting Amendment 1. But with the mixed reality that exists today, supporting the amendment would require me to trust Georgia will raise the charter quality bar in the future, partnering with high-quality organizations.

If that is a risk you are willing to take, vote yes.

If instead you believe the state needs to demonstrate more consistent results from the charters already operating before taking on a new initiative, vote no.

[1] This measure of school quality is the three-year average Student Growth Percentile, with controls for observable characteristics of the students at the school. School performance on this measure is then used to rank schools by percentile. Percentile ranks are helpful for intuitively discussing one school relative to others; however, they may overstate differences around the center of the distribution. Schools between the 40th and th 60th percentile in the state probably differ from each other in less dramatic ways than schools between the 80th and 100th percentile.  If you want to see more about how this is calculated, you can access the data and the STATA code here.

 

Reader Comments 0

135 comments
Wascatlady
Wascatlady

I'd like to see Mr. Apperson's thoughts on the wild swings that schools' CCRPI scores have made over the last 4 years, and how that relates to validity of the measurement.  Can he be contacted?


For example, one school in my system posted scores from 2012-2015 of 73, 87, 77.3, and 65.9.  Given these scores, how on Earth can the CCRPI have any validity as a measure of school achievement?  BTW, this school serves EVERY public school student in those grades in the system, so there has been no redistricting, etc.


Given all the other plans for the OSD, how could ANYONE unbiased and knowledgable about metrics or policy analysis think this is a good idea?  

Ronnie
Ronnie

This is sOOO interesting - thanks AJC and Jarod Apperson...


EdJohnson
EdJohnson

In his very accessible book, Twenty Things You Need to Know, internationally acclaimed educator, author, data expert, and Deming protagonist Donald Wheeler, Ph.D., offers, in the section “How to Analyze Data:”

“To explain why a process behavior chart should always be the first step in the analysis of data from any Observational Study we need to first define the four basic questions of statistics.  These are the Description, Probability, Inference, and Homogeneity Questions. … When we find evidence of a changing universe in a situation where there should be only one universe we will be unable to learn anything from descriptive statistics.  When the universe is changing we cannot gain from statistical inference, nor can we make predictions using probability theory.  Any nonhomogeneity in our collection of values completely undermines the techniques developed to answer each of the first three questions.  The lack of homogeneity is a signal that unknown things are happening, and until we discover what is happening and remove its causes, we will continue to suffer the consequences.  … The primary tool for examining a collection of values for homogeneity is the process behavior chart.”

Putting CCRPI school-level average scale scores on process behavior charts quickly and efficiently reveals greatly homogeneous process behavior on the part of State of Georgia system of schools, including both public schools and charter schools, that perform within limits of variation centered on 500, the design average.  Even so, some nonhomogeneous behavior is powerful enough to show up.  On the one hand, a relatively few schools – around 20 -- perform above the upper limit of homogeneous process behavior variation.  These higher performing schools include Atlanta’s Morningside and Springdale Park and some other higher SES schools but all higher SES schools.  On the other hand, a relative few schools – less than ten – tend to perform below the lower limit of homogeneous process behavior variation.  These lower performing schools tend to be exclusively alternative schools, GNET schools, and youth detention centers: for example, Atlanta’s Forest Hills Academy and North Metro Psychoed, and Sumer’s Youth Development Center.

Critical learning that comes straightaway from performing “the first step in the analysis of data from any Observational Study” using process behavior charts include: 1) GaDOE should be about the business of learning to improve the whole Georgia public educational system, while 2) learning from the few higher performing schools but not messing with them, 3) intervening in and helping the few lower performing schools, 4) understanding charter schools bring nothing fundamentally different than public schools, and 5) it is irresponsible to persist with taking over any “chronically failing schools,” as the OSD would have it.

What 2015 Georgia Milestones End-of-Grade School-level Mean Scale Scores Say about Georgia Schools System offers some process behavior charts, and Atlanta school board and superintendent playing a zero-sum game offers some insight into how the CCRPI actually works to sustain academic gaps and give false rationale to claiming so-called chronically failing schools exist. 

EdJohnson
EdJohnson

A correction: These higher performing schools include Atlanta’s Morningside and Springdale Park and some other higher SES schools but [not] all higher SES schools.

EdJohnson
EdJohnson

Don't know what took him so long, but finally...


Tomorrow morning, Ambassador Andrew Young will speak out against the school takeover amendment[.]


Here are the details:

Tuesday, October 18, 10 a.m.The Gathering Spot
384 Northyards Blvd NW Atlanta, GA 30313 OR, join the press conference live on Facebook.com/BetterGeorgia

RoyalDawg
RoyalDawg

@EdJohnson Young, like most black political leaders (Black Caucus), are more concerned with the unions than the affected children.


They better be more careful- if they keep opposing issues that black parents understand will help their children, they may see more of their constituents get off of the democratic plantation.

AvgGeorgian
AvgGeorgian

OSD supporters are for a plan based on the current governor appointed, state run, chronically failing State Charter Commission School System. It has been failing for years, has no plan for turning around its schools, and hides all hiring, personnel, vendor payments and other spending from the taxpayer.


If the spending is not public, then neither is the school.


RoyalDawg
RoyalDawg

@AvgGeorgian Buddy, you do not understand the system but insist on constantly weighing in. The SCSC is not completely appointed by the governor, is not "chronically failing, and if it were it hasn't been for years, and the Amendment is only 4 years past. You people scream for local control- the "plan' is that the local Board exercises innovation and is only accountable for financial stability and academic performance. That is about a "local" as you can get.


The local innovation is the foundation of the plan; they perform or their charter is not renewed. It isn't rocket science. There are "local educators" in charge, even more local than the county Board. What else do you want?


You have become tiresome by refusing to acknowledge the fact, just repeating the same mantras. I wonder which district with failing schools employs you.

MaryElizabethSings
MaryElizabethSings

Very well presented. I, too, believe that Gov. Deal is well-intended with his OSD plan; however, after much consideration, I wil be voting, "No, " to Amendment 1. My primary reasoning? I have studied politics in this state and nation for many decades and I know that the Conservative Republican agenda has been to turn public schools to privately controlled as they have tried to turn prisons over to private management. Both are systematically bad ideas but fervently held by most Republican politicians. Read Jane Mayers' recently published book, "Dark Money," to learn surprising details.

Question? Are KIPP schools currently taking over entire school populations or only selected students in which parents have taken a lead in their particular children's enrollment?

colt07
colt07

Seems to me you found an academic 'pin head' who is against any new change in the school paradigm of poorly performing teachers/schools. I lived in a public school system that has above average students and below average school board. The board always seems to be changing things to justify their existence.

If you want schools to improve we must change the basic system. If anyone else might have a better idea I am open to suggestions. Just not from failing academia.

sneakpeakintoeducation
sneakpeakintoeducation

@colt07 So if someone from the academic community agrees with the OSD should we applaud them? Even when the evidence shows overwhelmingly that the OSD has failed wherever it has been tried. Should we institute a of privatizing our schools through charters even when it has been shown to, in almost every case, not produce better results? 


How do you suggests we change the system to improve it? Do we look at the things that might cause these students to do less well than their more affluent counterparts? Should we consider that their environment and surroundings might inhibit their ability to learn than their more affluent counterparts? Please tell me which parts should we ignore?

Astropig
Astropig

@sneakpeakintoeducation @colt07


Totally false premise,starting with "privatizing".Converting schools to charter status or direct intervention is not privatizing anything.They will still be public schools,open to all.The difference is that their systems of management will change to eliminate useless layers of non-teachers that soak up time and attention that is better devoted to actual classroom teaching.


You know this.I know this.Everybody knows this.Stop spreading union nonsense.School systems are not big city jobs programs.Their purpose is to educate kids-period.

ScubaSteve
ScubaSteve

Hmm, do I trust the state of GA?


Hahahahahaha. Vote NO.

class80olddog
class80olddog

@ScubaSteve @class80olddog  I have a bridge I inherited from my parents in New York City.  It is a great business venture and collects a fantastic amount of dollars in tolls each year.  I can sell it to you at a greatly reduced price because I need to raise some cash.  Are you interested?

ScubaSteve
ScubaSteve

@class80olddog No, nor am I interested in a state that has shown absolutely no ability to properly handle education issues or any desire to educate its citizens equally. 

ScubaSteve
ScubaSteve

@MaureenDowney @class80olddog @ScubaSteve It isn't "interesting" at all, Maureen. It's the same knee-jerk "Chicago!" retort we hear when trying to have intelligent discussions about crime, gun control, etc. That playbook doesn't change just because this is about education.

Astropig
Astropig

@MaureenDowney @class80olddog @ScubaSteve


He faced the same choice then as voters do now:Do nothing,turn away and let the status quo destroy the public school system to protect their own little kingdoms,or use the power that the voters entrusted to him to look out for all stakeholders and let the kids have a decent chance at a good education.If I recall,their accreditation was in jeopardy,which would have made it impossible for even good students to attend some universities,would have nuked property values (leading to a county of renters) and led to good teachers leaving for accredited systems.


Hard to see how things were going to get better before he acted. 

class80olddog
class80olddog

This from Wikipedia about DCSS: "The system educates more than 101,000 students at 137 schools with more than 14,000 full-time employees and 6,000 teachers. The student-to-teacher ratio is 23:1."  Now when I do the math, I get a student:teacher ratio of less than 17:1 (unless some of those teachers don't teach).  But look at the other figures: 6000 teachers and 8000 "other" employees!  Want to drop your student:teacher ratio?  Just get rid of some of those 8000 and hire more teachers!

ScubaSteve
ScubaSteve

@class80olddog WAY more than I trust the State. That is in no way a vote of confidence for DeKalb, and 100% intended as condemnation of the State.

AvgGeorgian
AvgGeorgian

@class80olddog The state legislature could pass a law requiring 85% of funding be spent in the classroom - they won't do it because the charter profiteers won't then be able to make a profit.

MaureenDowney
MaureenDowney moderator

@class80olddog @ScubaSteve The odd thing about bringing up DeKalb is that DeKalb is where state intervention has been the most extensive and far-reaching -- Gov. Deal removed the school board and handpicked the replacements. Those replacements chose the current leadership. 

It is interesting you keep citing DeKalb as a reason for state intervention when it is likely the district with the greatest state intervention in modern Georgia history. 

If you are not happy with how Deal's intervention altered DeKalb, I am not sure you will be happy with state intervention anywhere as the governor essentially remade DeKalb to his liking. 


elementary-pal
elementary-pal

@class80olddog We teach our fifth graders that anything on Wikipedia must be verified since it is editable information.  Just saying.  

class80olddog
class80olddog

In other words you can either vote Yes for the OSD or you can vote for a continuation of failing schools.  What will it be?

Astropig
Astropig

@class80olddog


Voting no is saying that 68,000 kids sentenced to poverty,jail and privation is an acceptable price to pay to keep a few educrats fat and happy.

EdJohnson
EdJohnson

@class80olddog In other words, you can either vote NO or you can vote for the status quo thinkers who are always trying to "fix" our public education systems and are disinclined to learn to actually do it.

class80olddog
class80olddog

@Astropig @class80olddog @EdJohnson  Yes, that is the only "root cause" that I have heard mentioned - not enough money.  Never do they mention issues like discipline, attendance, or social promotion, even though Wascatlady even admits that 99% of teachers would agree that these are serious issues.

EdJohnson
EdJohnson

@class80olddog @EdJohnson @Astropig The first step to solving a problem is to L-E-A-R-N about the problem and the context within which the problem manifests.  From such LEARNing it may prove advisable or ill-advisable to worry about root causes.  

EdJohnson
EdJohnson

@class80olddog @EdJohnson @Astropig Talk to firemen about root causes while your house is burning down.  Again, first, study the problem and the context within which it manifests and you just might learn something you did not already know or thought you knew.