APS releases GSU study on impact of cheating scandal on students. Finds ‘moderate’ impact.

For the first time, we have some data on the fate of children whose CRCT answer sheets were likely altered – wrong answers changed to right — in the APS cheating scandal.

Up to this point, there’s only been speculation on whether test tampering on the 2009 CRCT undermined student learning and, if so, to what extent.

Tonight Atlanta Public Schools released a report by Georgia State University that concludes “negative effects of the 2009 Atlanta Public Schools cheating are moderate and not uniform across students in classrooms identified as having irregular wrong-to-right test item changes in the spring of 2009.”

The study, “The Long-Run Effects of Teacher Cheating on Student Outcomes,” was completed by Dr. Tim R. Sass from GSU’s Andrew Young School of Policy Studies. and two doctoral students Jarod Apperson and Carycruz Bueno.  (Apperson writes the Grading Atlanta blog.)

Looking at the erasure data on wrong to right answers, the study found 7,064 students likely had answers doctored. More than half still attend APS schools.

The eight-page report cites a negative impact in mathematics learning for children who were in first and second grade in 2008-2009. In layman’s terms, the report says those children lost about the same amount of ground that comes from “having a rookie teacher rather than one with five years of experience.”

For other grades, the study found smaller effects on math performance. It is a different story for reading and English Language Arts.

The study states:

In contrast to math, in reading and ELA the researcher found that “being cheated had negative consequences for later student performance… The estimated impacts are in the range of one‐fourth to one‐half of the average annual achievement gain for a middle school student. This is equivalent to one to two times the difference between having a rookie teacher and one with 5 or more years of experience in a single year… There is little evidence that teacher cheating had any deleterious effects on subsequent student attendance or student behavior. Any impacts that may have occurred were very small.

The GSU researchers conclude:

The study clearly shows that the cheating disproportionately impacted black students. For example, while 75% of APS students are black, 98% of the students identified as having the largest number of erasures on their answer documents (10 or more) are black. The data also shows that the performance of students in this group is lower than the district as a whole and the student group with less erasures (5-9). In general, only small differences were found between the student group less impacted by the cheating (5-9 erasures) and the district as a whole.

However, it should be noted that it is difficult from these data to assign causes for the differences. For example, this summary did not attempt to control for other important variables such as teacher experience, teacher quality, and other school climate and culture variables that may well have impacted students in these schools.

To read what Atlanta is doing for these students, go here.

At the request of Superintendent Meria Carstarphen, APS asked GSU to attempt to identify students affected by the cheating scandal and assess the long-term impact.

At the request of Superintendent Meria Carstarphen, APS asked GSU to attempt to identify students affected by the cheating scandal and assess the long-term impact.

Here is the official statement on the study from APS:

Atlanta Public Schools today released an independent study on the effects of cheating on the 2009 Criterion Referenced Competency Test. APS commissioned the study before the start of the 2014-2015 school year after then incoming-superintendent, Dr. Meria J. Carstarphen, asked about students affected by teacher cheating.

The study, “The Long-Run Effects of Teacher Cheating on Student Outcomes,” was completed by Distinguished Professor Dr. Tim R. Sass from the Andrew Young School of Policy Studies at Georgia State University and provided to APS on May 5, 2015.

“When I joined Atlanta Public Schools last summer, I was committed to knowing exactly what APS was doing for those students still in the district and, if possible, even those who had left APS,” said Dr. Carstarphen. “We did not know which students may have been impacted – until now.”

The study will provide the baseline data needed to track each of these students through graduation.

According to APS records, during the 2009-2010 school year, the District implemented a 12-week accelerated academic recovery program for students scoring below proficient on the spring 2010 CRCT at the 58 schools identified by the Governor’s Office of Student Achievement. The program served 5,423 students.

In an attempt to restore credibility to the District’s student achievement data and provide reliable information that better serves students, APS, in spring 2011 implemented an online, nationally normed computer adaptive assessment system to provide independent validation of state and local test results.

After 2010, APS instituted additional programs and interventions to meet the needs of students who may have been impacted by teacher cheating. This included adoption of an Accelerated Intervention Plan (AIP) that comprises a mandatory school day component, afterschool enrichment, a Saturday academy and parent workshops.

Over the past school year, APS launched district-wide intervention programs in reading and mathematics, unit recovery programs for all students in grades 6 through 12, and flexible scheduling options. While these programs are designed for students who are not succeeding academically, or who are already out of school or are at risk of dropping out, they should also assist those students who may have been impacted by teacher cheating.

APS will make certain that the students directly impacted by the cheating are being served. Using the Georgia State University study, the District is identifying the students to track their progress, evaluate the services they have received, and to consider any other remediation program they need to keep them on the path to graduation.

“We continue to address the impacted students’ needs and ensure they are supported through graduation,” Carstarphen said. “This administration, from Day One, has supported a student-centered agenda and is focused on our mission to create a caring culture of trust and collaboration, where every student will graduate ready for college and career.”

The complete study, The Long-Run Effects of Teacher Cheating on Student Outcomes, is available online, as well as the District’s CRCT Erasure Analysis and Ongoing Remediation Update

Reader Comments 0

58 comments
eulb
eulb

@ Maureen Downey, is there any chance the authors might clarify their results?  Here are my main questions:

1) What does "moderate" mean in this report?  Some commenters here seem to think it means "minor" or "negligible", so they think this report is whitewashing or minimizing the degree of damage to students.  My own understanding of "moderate" is different.  I relate it to vehicle damage in a wreck, where moderate damage is somewhere between "minor/negligible" and "severe/totalled". Negligible damage is cosmetic and can be ignored.  A car with moderate damage needs repairs before it goes back on the road.  

2) Does a rookie teacher actually damage his/her students?   I tend to think of new teachers as being smart, energetic, and willing to devote countless hours outside the school day to devise ways to impart their knowledge to their students.  But this report uses them as a yardstick for estimating damage. That surprised me.

Cbpvsp
Cbpvsp

There are multiple opportunities provided kids who are struggling. After school tutoring, Saturday school, summer school, home work hotlines, paid private tutoring, ongoing credit recovery, etc.. I work with APS schools and the sad truth is that parents do not take advantage of these options - so kids suffer the consequences. Compare results of kids whose scores were changed against kids who were also at risk and you would see no difference. Both groups are placed in the next grade and both ignore help provided by school system.

algebra teacher
algebra teacher

I have one class of socially promoted students to whom I teach algebra even though they have little number sense and have failed math classes and standardized tests prior to this year.  Few students are retained in our schools and CRCT results seem to be of little consequence for students.  As much as we'd like to believe this scandal is about student education, the reality is that it's about money and power.  Alternative education is needed in lower grades for students who will not, for what ever reasons, meet academic standards.  We continue to debate the same issues, trying the same repackaged strategies, and expect different results. 

bu2
bu2

They are downplaying it, but the bottom line is this:


The study shows that the cheating scandal hurt students.  Period.



Wascatlady
Wascatlady

Not passing judgement on this study till I read more about how the data were treated.


However, GSU has produced some pretty "inelegant" research on the HOPE scholarship in the past--simple cross tabs, instead of substantive analysis.

straker
straker

As long as certain students are socially promoted and allowed to graduate while still functionally illiterate, I guess they really weren't hurt at all.

MaureenDowney
MaureenDowney moderator

@Folks, Not sure why you think the research team -- led by a nationally noted researcher who came to Georgia State a few years ago -- would manipulate results. Who would that help in 2015, six years after the cheating?


This was done at the request of a brand new superintendent who had no stake or benefit in minimizing the impact of the 2009 cheating. In fact, logic would suggest the current APS regime would benefit from just the opposite. It would enhance Dr. Carstarphen’s position if the study announced a long-lasting and devastating impact as it better sets her up to be seen as the district’s savior and salvation. (Every new boss likes to be viewed as the solution to a broken and inefficient system.)


To me, the report confirms what many of us thought -- these students whose test answers were changed were Atlanta's hardest-to-educate -- hence the motivation by educators to improve their lagging test scores. APS has a strong effort to move these kids forward but the challenge is reaching students who are now adolescents. There is not much evidence -- although KIPP is certainly the model with its longer day and year and its intense focus on core courses -- this can be done without incredible effort, investment and time.

popacorn
popacorn

@MaureenDowney

Someone else's motivation is difficult to determine. Maybe some are motivated to take the easy way out in everything they do. 

Gwinnetting
Gwinnetting

@OriginalProf

Job titles may impress you, but they won't dispell any of the sincere doubts others will have. Who paid for the study? And who's promoting its self-serving assertions?

OriginalProf
OriginalProf

@MaureenDowney 

Or, one could say, there were many other factors besides the cheating scandal that could have a negative impact on the students and prevent them from being diagnosed as needed remediation. Previous social promotion beyond their grade-levels, persistent grinding poverty in their home-lives, the schools' failure to enforce policies on attendance and student discipline-- all are factors that were present with these APS students.

In other words, the cheating scandal did not create the problems already present, although it did reveal the results: many illiterate, ignorant students being pushed along to the next grade level and out of the system.


Btw, many here are attacking the researcher... red herring, guys. The Professor is a "Distinguished Professor" at one of Ga.'s research universities, a recent category there that is one notch below a Regents Professor, in its School of Policy Studies.  This "school" or college offers majors in various aspects of public policy-making, so that it often is consulted by regional government agencies. No connections at all with APS or Georgia's public K-12 schools.

OriginalProf
OriginalProf

@Gwinnetting @OriginalProf 

I'm not even sure that anyone "paid for the study," since it was a public school (APS) that commissioned a state university (GSU) to prepare the study by one of its professors and 2 of his graduate assistants. It's common for universities to absorb the cost of such requests (leave-time for the professor and assistants), and then claim it as public service to the state. I assume that the three will now be able to publish it in a national journal as co-authors. That may not mean much to you, but it does to professors and grad students looking for post-doctoral employment.


As to who's promoting its conclusions, I note that "using the Georgia State University study, the District is identifying the students to track their progress, evaluate the services they have received, and to consider any other remediation program they need to keep them on the path to graduation."  So it's guiding present efforts to help the affected students.

Looking4truth
Looking4truth

@MaureenDowney  While I have no doubt that what you say is true (about the difficulties faced by these students), the cheating scandal indicates they didn't even try, so we'll never know.



OriginalProf
OriginalProf

@Looking4truth @MaureenDowney 

Or, what may be worse, they tried, thought they knew enough because their scores were passing (and so did their parents), and never found out differently till a few grades later when they couldn't read or couldn't do math. And then what could they do?

Gwinnetting
Gwinnetting

So the above personal attack from a habitual offender is allowed to remain up? 

Gwinnetting
Gwinnetting

@JBBrown1968 

The Internet doesn’t just offer opportunities for misogynistic abuse, you know. Penis enlargers can also be bought discreetly.

OriginalProf
OriginalProf

@Gwinnetting @OriginalProf 

According to today's AJC (Saturday), Professor Sass volunteered to Superintendent Carstarphen to do the report free of charge and she supplied him with the necessary data.

JBBrown1968
JBBrown1968

@Gwinnetting @JBBrown1968


Has the enlarger worked for you? Did you get a good deal? As far as I can tell you posted at me. Kitty! Here troll.....Here Troll......I mean PopAcorn!

Looking4truth
Looking4truth

I believe the conclusion of this awful report is that, somehow, these kids wound up in better classrooms with teachers who believed in them. Those teachers mitigated the damage caused by earlier teachers who'd rather cheat than teach.  The results would be different if the study were done earlier and entered into evidence. 


If the convicted teachers had bother to do their jobs to the best of their abilities, cheating would have been unnecessary. 

Mandingo
Mandingo

It is beyond me why parents with jobs that work for a living would voluntarily live in this school district and send their children to theses sh*tty schools run by people with such low morals. If you are living off of food stamps, section 8 etc for YEARS, YEARS and more YEARS and can't figure out how to obtain employment and improve your situation then this is what you and your children will always be stuck with.  What is the solution ?????? That is something parents in this school district will have to decide just like every other parent in the other 158 counties have to. 

redweather
redweather

This is a rather self-serving report.  While I would like to believe that the cheating had minimal long-term impact on students, it makes me wonder what else has been and still is going on in APS, as well as in other school systems for that matter.  The pressure to perform always seems to have unintended consequences.  Students plagiarize term papers, locker room attendants deflate footballs, etc.    

Astropig
Astropig

@redweather


"This is a rather self-serving report."


Sure is. That's the kind of report that big bureaucracies with turf and interests to protect generally produce.

popacorn
popacorn

So after a couple of years of exposure to the toxic radiation of crappy teachers, we have declared these kids symptom and disease free. Did the researchers themselves graduate from APS?

odelljones
odelljones

so that makes the cheating and the lying  better.....

I hope every person, (they don't deserve to be called educators any longer) loses their pension.

dcdcdc
dcdcdc

In other news, an Enron funded study has determined that cheating at Enron only had moderate impact on their stockholders as well....


What a crock.

Astropig
Astropig

Milquetoast reports like this and attempted whitewashes and coverups in this tragedy are what makes me even more glad that we're a nation of laws, and ,even though it's not perfect,the perpetrators of this episode faced justice and accountability for their crimes.All except the queen bee herself,of course. Beverly Hall will most likely face a different kind of justice if Andrew Young's imprecation to "let God judge her" is correct.

dreema
dreema

@Astropig Oh please. Report states that "those children lost about the same amount of ground that comes from “having a rookie teacher rather than one with five years of experience.” 

GCrouchback
GCrouchback

@Astropig 

So the Atlanta School District, in effect, evaluated its own monumental failure—and advises us that "there's nothing to see here."

Astropig
Astropig

@GCrouchback @Astropig


"So the Atlanta School District, in effect, evaluated its own monumental failure—and advises us that "there's nothing to see here."


I guess. I don't care. All I know is that the perps that did this are going to jail and faced justice (some subject to appeal). It's also a lesson to any other educator that might be tempted to do it in the future. I could give a toss what APS thinks.

ByteMe
ByteMe

Reading comprehension is not strong in your blog commenters, Maureen.


From APS' official reaction it sounds like this scandal did more for the kids than NCLB alone did... the kids are now getting the additional help they need and that appears to be an ongoing response.  I hope it's really true this time.

USrover
USrover

Finally we now know who is really at fault - those rookie or less than five years of experience teachers. And the beat goes on, nothing will change.  Truly sad for the students who will continue to under perform.

nola2atl
nola2atl

learning is a lifetime thing and is not limited to one teacher or the classroom. what a waste of money. what i would like the AJC to confirm is that none of the offenders will have licenses restored or be allowed to teach in the State of Georgia.

GB101
GB101

The authors of the report needed a better editor.  There is a misspelling in the very first sentence of the executive summary.  

Dasdguy
Dasdguy

If that's what Andy said;  it must be true.

MyThreeCents
MyThreeCents

Wow! All of that money for the trial was wasted. The teachers should have received a slap on the wrist and been judged by their superiors and all forgiven. The kids did not matter... how sad. 

dreema
dreema

@MyThreeCents The testing, and testing in general is overrated as a measure of student achievement. It's a flimsy, one-dimensional view that was (and is) given far more importance than it is worth. 

partlycloudy
partlycloudy

They are probably cheating on that report also.  Who would believe anything that is touched by APS?  Poor students.

TopSchoolAtlanta
TopSchoolAtlanta

Follow the money. How much money has been made off this cheating scandal? Lawyers, judges, a special investigation team, another study to figure this out. Buckhead (Bucket) loads of money have filled the pockets of all the wrong people. Follow the money. It all leads back to the pockets of those affluent Atlanta business leaders and their buddies. Yet, another money DEAL.

atlanta spirit
atlanta spirit

THE WORST PART OF THE CHEATING WAS THE ETHICS THEY TAUGHT THE CHILDREN.  CHILDREN WERE TAUGHT LYING AND CHEATING ARE OK BY THEIR TEACHERS.