Testing and technology in Georgia schools: Who will watch the watchers?

University of Georgia professor Peter Smagorinsky takes on the testing industry and the inaugural run of Georgia’s latest remake of its state exams, the Georgia Milestones. The Milestones came under heavy criticism this year for serious technology glitches that led the manufacturer to compensate the state.

Smagorinsky hits upon the main problem with the new tests in Georgia — the lack of time to pilot the tests before rolling them out statewide, especially given the move to online administration.

By Peter Smagorinsky.

Much has been made of the current trend toward constant standardized testing of students in school. These concerns include the mismatch between authentic teaching and standardized assessment, the encroachment of government into the process of schooling, the recognition that most government-imposed curriculum-and-assessment plans have little relation to research findings, the inattention to the role of poverty in educational achievement, and many other problems.

downey1213I’ve recently talked with teachers from two communities in Georgia that are, in many ways, typical of cities across our state. They are not too big—neither has minor league sports teams, as do relatively large Macon and Rome. They are not too small to be left off state maps. Especially for the people who live in them, they are just right: Communities neither especially rich nor especially poor, although with a wide enough range of parental affluence for teachers to find most of their students nonstandard.

My conversations with these teachers unearthed some remarkable aspects of the newest testing regimes: Georgia Milestones, End of Course Tests, graduation tests, and many more. In their schools, after spring break there is at least one test being administered to at least one group of students every day of school.

Some of the problems with their implementation are almost comically bizarre. EOCT data, for instance, aren’t returned until the following school year has begun, rendering them useless as EOCT evaluations. When satire is hard to distinguish from reality, it’s time to take a closer look and see what’s happening in this age of accountability—for students and teachers, that is. Those in charge of accountability apparently have no one to be accountable to, as this example demonstrates.

The fiasco of waiting until the following year for EOCT scores to be returned is just one of many absurdities afoot in this age of educational absurdity. Unfortunately, policymakers rarely ask teachers how the reforms are going. Rather, they just barge ahead with the next one without running a prototype with a pilot group to see how it works before going whole-scale.

A couple of years ago I met with one of Georgia’s leading legislative policymakers and a group of teachers to discuss public education. I was pleased to be invited, and primarily deferred to the teachers. My own limited contribution: Before you impose a high-stakes assessment system on the whole state, pilot it first with a volunteer district to see how it works under good conditions, then gradually expand the program as bugs are worked out.

But policymakers are much more concerned with rushing untested systems into place and holding the practitioners accountable for the results than holding themselves accountable for making the test reliable and valid before taking it to scale.

Here’s how that decision to charge forward with assessments such as the Georgia Milestones, such that they become the Georgia Millstones to the teachers charged with implementing it overnight, has worked in the real world.

Teachers in both systems that I have talked with recently say the same thing: the online testing infrastructure in their districts is insufficient to accommodate the extra layer of test-taking required by the state. In one district, the software itself failed, wasting a whole lot of instructional time with a test whose results were lost in the Ethernet.

In the other, the availability of bandwidth to support large groups of students taking online tests simultaneously brought the rest of the school to a halt when it came to using the Internet. This school is among the majority in Georgia whose budgets do not support running a full school year, resulting in 10-30 days struck from the calendar with teachers furloughed, that is, not paid, to align the funds in the budget with the salary commitment.

Schools that can’t afford to pay their teachers to teach a full year are in no position to invest heavily in Internet capacity so that students can reliably take tests online and, at the same time, allow teachers to send emails to parents, use their online gradebooks, and otherwise use web-based tools available to enhance their work. That capability is lost for weeks on end as students occupy bandwidth with testing, and even the availability for testing is fragile and spotty.

The state of Georgia has invested many tens of millions of dollars in developing web-based tests, enriching people from outside the schools, without investing in the schools themselves so that these tests may be taken with confidence that the scores will be available in a timely manner, or at all. Once again, the people at the policy level make the most vulnerable people in the system—teachers and their students—fully accountable for outcomes of their decisions, without providing them with the resources to do that work in good faith.

Nearly 2,000 years ago, the Roman poet Juvenal posed the question, Quis custodiet ipros custodes?, or Who will watch the watchers? The problem and challenge of making the authorities accountable remains today. In rushing untested policies into practice without providing the resources to support their implementation, and then blaming teachers for the results, they make a farce out of one of our nation’s most serious and important commitments: the education of our next generation of citizens.

For that, they should be ashamed. But I’m not counting on it.

 

Reader Comments 0

109 comments
cyadra
cyadra

I teach Economics, and EOCT course we give to our seniors. We always received our results about 5-7 days after we gave it.

living-in-outdated-ed
living-in-outdated-ed

It was an embarrassment for sure that McGraw Hill couldn't successfully implement an online standardized test, but having said that, we need to figure this out and get our technological infrastructure enhanced to support online tests.


Lets not forget the absolute embarrassment that the College Board faced earlier this month with a misprint in the test booklet!   It happens with paper tests too, on MANY occasions.   Seems the College Board reviewers were asleep at the wheel.


In the year 2015, we should be able to take an online test without major glitches.

taylor48
taylor48

Besides Internet bandwidth,what about the investment in hardware? We have three labs of about 27 computers (when they're all working). Each lab doesn't even accommodate ONE class, and the state wants all of the 3rd, 4th, and 5th grades to test on the computer? Our three labs can't fit one grade level at one time, so the testing will have to be staggered. So, ultimately, the schedule of an entire school will have to be in upheaval for two to three weeks to get this testing all done. (Even the non-testing grades have to switch lunch and specials in order to accommodate testing.). I get why the state wants to do this, and, in theory, it seems like a great idea, but there are a lot of hurdles that need to be overcome before this gets rolled out statewide. GCPS tried for a couple of years to do their quarterly benchmark testing online, and even that was difficult. Eventually, they scrapped it and went back to the paper and pencil.

jerryeads
jerryeads

Hi Pete. They should be far more than ashamed. While as we know I had the distinct relief to never have been employed by the Georgia DOE, I spent a while running testing for another state. I most certainly am ashamed of the things we did, which were (I hope) somewhat less egregious than the horrific negligence practiced these days. Not only do far too many of the people employed to operate testing programs in state departments of education have no training whatsoever in measurement, the one thing that struck me in my years downtown working for other agencies is the simple fact that the they absolutely DID NOT CARE ABOUT KIDS. NOTHING in my discussions with them EVER suggested helping kids learn was a priority. IT WASN'T EVEN ON THE LIST. (There was a guy some years ago running testing that was not only one of the brightest I'd ever known in the business but he cared. They fired him, of course.) Not only is this a function of who's hired to run testing, my guess is it's a pretty good indication of the priorities of the leadership.

I'd like some inkling of a suggestion that such is no longer the case, but all I see is evidence to the contrary.

Best bet: Parents getting on the no testing bandwagon. Next time your child is stuck with one of these abjectly worthless state tests that tell neither you nor your child's teacher anything at all and are thus of no use whatsoever to your child, keep 'em home. Enrollment in the state should be a tad over 1.7 million by now. If even a tenth of you keep your kids home next test, it goes down. And it won't hurt your kid's future, either. They wouldn't dare.

Scienceteacher - the technology is more complicated than that. For example, for some tests (not necessarily the low-bid relatively low-quality tests contracted for the state) difficulty of the questions is tailored so that the average student gets about 55% of them correct. Some low-bid minimum competency tests are set up differently, so that the average student gets most of the questions correct, and we of course learn absolutely nothing about above average students. Even worse, as we know from our state tests, some of them have over 95% of the students passing the first time, while another test in the same grade in another subject will fail half. This has NOTHING to do with how much kids are learning, and everything to do with the gross incompetence of those building the tests and arbitrarily and capriciously setting the cut scores.

class80olddog
class80olddog

@jerryeads "Best bet: Parents getting on the no testing bandwagon."

That is GREAT!  So all you parents - don't be shocked when your child gets a diploma, comes to work for me, and I fire him/her after two months because he/she doesn't have even the basic skills, but managed to get socially promoted right out of HS.  THANK GOD we don't have that pesky GHSGT, so now ANYONE can be a high school graduate!

class80olddog
class80olddog

@jerryeads "And it won't hurt your kid's future, either."

It will certainly hurt your kids' future if they are getting "A's" while learning nothing.  Eventually that catches up to them - either in college or in the real world - where made-up grades don't mean anything.

30092_
30092_

@jerryeads 

So, without any proof whatsoever, you suggest Georgia's DOE is guilty of doing the sort of nefarious things you did in your previous job?

And we should believe you ... why?

jaggar1
jaggar1

@class80olddog @jerryeads You have totally misunderstood what Jerry was suggesting. First, parents should be able to know if their child is relatively on grade level. If you don't, you shouldn't have kids! Education begins in the HOME,  you should be reading with your child EVERY night, and you should be practicing their math facts too. If every parents did this, they would have knowledge if their child knows the curriculum. Next, parents are keeping their children home during testing, because the testing in NOT a good indicator of what a child knows. Part of the reading test had two passages, each two pages long, and the kids had to read both passages, then answer questions to compare the passages. You lost the kids attention after the first page of reading. The results of Milestones won't be back until the Fall, so that doesn't help anyone. Unfortunately, politicians and business people, whom have never stepped into a classroom, have hijacked education. The executives in the districts have no clue what is going on, and we need to get back to basics. Reading and math are the two most important things in a child's early development to being successful in school. If we could focus on those two standards and everyone would stop cramming standards, which are NOT age appropriate, down our throats, education wouldn't be such a mess in Georgia!

ScienceTeacher671
ScienceTeacher671

@jerryeads "And it won't hurt your kid's future, either. They wouldn't dare."


Actually, if it's an EOC, the student won't get credit for the course.  That could hurt.

ScienceTeacher671
ScienceTeacher671

@jerryeads "Scienceteacher - the technology is more complicated than that...everything to do with the gross incompetence of those building the tests and arbitrarily and capriciously setting the cut scores.


Actually, it's the capriciously set cut scores I'm most concerned about.


With the old Physical Science EOCT, a student who got 43% of the questions correct would receive a 70 on the test.  A student who got 63% correct would receive a 90.  A student who got 85% of the questions correct would receive about a 94 or 95.  


We've been told that scores and passing rates will be much lower on the new tests because they're going to be "much more rigorous."  But if the questions are basically the same ("We've got plenty of field-tested questions from the OLD tests" per J-C. Aguilar), there's nothing to change but the cut scores.  


I've got no problem with actually expecting a student to pass (70%) a test to show that s/he has mastered the standards.  They'd need to on my teacher-created final exam, if we still gave those instead of EOCs.  I did have a problem with a student who got 63% of the questions correct getting essentially the same grade as a student who got 85% correct, though. 

ScienceTeacher671
ScienceTeacher671

@jerryeads  And I can see how, with a test like the ITBS, you'd want the average student to get maybe half of the questions correct, so that you could also identify the student who's working well above grade level. And that might also be useful for the elementary and middle grade Milestones tests, especially in reading and math.  Assuming competent test builders, of course.

Astropig
Astropig

@jerryeads


" Next time your child is stuck with one of these abjectly worthless state tests that tell neither you nor your child's teacher anything at all and are thus of no use whatsoever to your child, keep 'em home."


Did your kids ever take these worthless tests in public schools, Jerry? Now that your computer is "fixed", it would be nice to know if your're a "do as I say" educrat.


No answer means no-they did not.

MiltonMan
MiltonMan

Maybe we can hire the disgraced APS-teachers that were sacked for cheating.

ScienceTeacher671
ScienceTeacher671

Maureen Downey, have you asked Richard Woods to comment on any of this?


I'm also curious about cut scores.  Will EOCs be graded on a curve?  What percentage of questions will students need to get correct to indicate mastery of each subject?  

class80olddog
class80olddog

And what will happen to those who fail the test?

class80olddog
class80olddog

@ScienceTeacher671 "I'm also curious about cut scores."

Why would you have "cut scores" for something that only counts for 20% of your grade?

ScienceTeacher671
ScienceTeacher671

@class80olddog By current law, the test is 20% of the final grade. Originally I think students were supposed to have to pass it to get credit, but that got watered down.

ScienceTeacher671
ScienceTeacher671

@class80olddog @ScienceTeacher671 Yes, if they already had an A in the class.  Most students with A's wouldn't want to ruin their GPA like that, but it's possible. 


Some of our economics teachers have been a bit worried about the teacher evaluation part for this reason, though - a lot of the seniors in their last semester of school aren't particularly worried about how they do on the EOC if they know they're going to pass regardless.  And I've seen very few grades below 50, even for my kid who never even opened the test booklet. 

class80olddog
class80olddog

Obviously, requiring a test to be taken online when there is not the technology available in the school is pretty ludicrous.  But then again, testing a student and then not using the test results is pretty ludicrous, also.  And giving the test two weeks (or longer) before the end of the school year is the height of stupidity.

And people wonder why we voters keep questioning the educational system.

By the way, this is NOT teacher-bashing, this is ADMINISTRATOR and Legislature-bashing.

Point
Point

Administrators have no say in this either. Contact your representative!

class80olddog
class80olddog

@jaggar1 @class80olddog "all I care about is making my kids feel successful! "

Even if they are NOT successful!  But their self-esteem is good, even if they can't add and subtract!

jaggar1
jaggar1

@class80olddog The politicians have ruined education. The teachers AND administrators have no voice in any of this debacle. It is all thanks to the federal and state governments who run education. They have created one huge mess. We are losing great teachers and replacing them with new teachers who quit after 5-7 years. We are actually told what to teach, even if it is not developmentally appropriate, and then if a child doesn't understand and needs more time, we are told to tier them. We are on time deadlines for each standard, and screw the kids who need more time. Throw in the Hispanic students who are learning a new language, and you have a total mess. Hey- go ahead and tell me that you are going to base my salary on the progress of kids and the testing that doesn't work, because that makes total common sense. Politicians don't care, parents don't care, and all I care about is making my kids feel successful! 


MaryElizabethSings
MaryElizabethSings

This is only part of an on-going ideological agenda based on turning America's public institutions based on public service into private institutions based on profit.


I have just posted the following thoughts on Kyle Wingfield's blog, awaiting his approval.  For those who can see the connection to the political agenda mentioned in this article, please do:


"I believe public education should remain truly a public institution in our nation and state, paid for by public taxes on all citizens.  I do not believe that public education should be turned over to private enterprise which will, more than likely, be tempted to use public tax monies for private profit purposes, thereby turning students and teachers into pawns for profit.


I believe that the best and most sophisticated educational reform will come from within the public school systems themselves, not outside of them, with the support of public charter schools, aligned with traditional public schools.


I believe that unions have kept the working class alive with some degree of collective power in our democratic-republic in which the oligarchs in America could rule like the royalty of Europe did in another place and time.


I believe that Georgia should have authentic teachers' unions, not simply professional educational organizations."

popacorn
popacorn

Took a while for some folks to accept that the world is round. Some still think it is flat. Similar situation here with technology. It is so much easier to blame testing and technology rather than teachers, parents, and students. Clueless education students are the bread and butter of Smagorinski's existence - of course he is head apologist/enabler/cheerleader.

athensareateacher
athensareateacher

@popacorn 


Please know that this comes from a good place.  Tone is so difficult to determine in a post.


After reading your post I wondered if you experienced the Georgia Milestone test or any other high stakes tests from within the 4 walls of a school this year?  If not, you can only speculate and imagine how incredibly ridiculous the big picture really is.  We are building the plane in the sky and don't have all the parts of the plane or the tools/personnel/training to connect the pieces. 


I would suggest talking to your educator friends and talking to students about their experiences and really listening with an open mind. Think about how you would feel if you were put in a comparable situation at your job.  And ask questions.  Is this what is best for students?  Are we setting out students, teachers and public schools up for success?  Who is really benefiting politically and financially from the chaos created by this type of hastily planned rollout?  How can you act as a cheerleader with a goal of positively impacting education in our state?  What steps can you take to enable our students, teachers and public schools to reach their potential? 



popacorn
popacorn

@MaureenDowney @popacorn

If previous experience shows that GA does not have the technology to enable online testing, then get the technology and work around it. There are companies that will rent a whole bunch of computers for a short time. Figure it out! Anything done for the first time is rife with errors, head scratching, blaming, pain, anger, frustration etc. Just ask the early space program folks. Technology is not going anywhere folks, the transition is inevitable. Stop whining and deal with it. 

P.S. I am sorry for the stressful situations you describe. I hope you and the kids are ok. 

popacorn
popacorn

@MaureenDowney @popacorn

Wasn't referring to your kids, Maureen. Was referring to the 'test takers' in your post. 

A million kids at the exact same time? Wow. 

I guess I don't understand the Georgia Way. If it fails after a short time, abandon it. There is a way to get computers to these kids for a few days of testing. The problem is, it may take some creativity. 

athensareateacher
athensareateacher

@popacorn @MaureenDowney  I'm not seeing discussions of anyone suggesting we abandon having benchmarks or tests in place to monitor student progress. The issue we are discussing today is whether we have put resources in place to administer the test in a way that provides students an opportunity to show what they know.  Currently, the "Georgia Way" does not provide the technology infrastructure to create a positive testing environment.  Fixing this will take more than creativity.  It will take money and time.

MaureenDowney
MaureenDowney moderator

@popacorn @MaureenDowney Not sure why my kids came  up as I was referencing my own situations of filing on deadline from external  sites and encountering technology failures.

As to your comment, teachers can't fix or fund school technology. That is the state's role and it has not taken it seriously or funded it fully.

I do not think computer rentals are the answer to more than a million students taking state tests at the same time.

class80olddog
class80olddog

@Linda Lee Smith @popacorn @MaureenDowney Yes, I know nothing about testing:

Solve the problem 81/9=?

Answers: A) 8  B)9 C)10 D)11 E) All of the Above F) None of the above

Of course you have to have open-ended answers to demonstrate "higher thinking skills".

And for their self-esteem, if they get zero answers correct, you have to give them at least a 50.

And finally - if their test results show that they are two grades behind in a subject - PROMOTE THEM into regular classes where they are bound to fail.

popacorn
popacorn

@Linda Lee Smith

You obviously drank several glasses of the Edu flavored Kool-Aid, and I won't try to ruin your delusion. I will say that maybe its time to forget your research and use common sense. With the attitudes displayed by so many educators here, it is easy to see how students might pick up the fear, terror, and loathing inherent in their role-models. 

Maybe on display here are the attitudes of folks who themselves have proved to be mediocre, at best, on assessments, and they carry that baggage into the classroom? And then impart it to the students?

Astropig
Astropig

@popacorn @MaureenDowney


@ Pops-


ALL technology has the human factor to consider-New tech,old tech,it doesn't matter. Part of that human factor could possibly be teachers sabotaging this testing regime to discredit it.If you look at Jerry Eads comment above, he suggests sabotaging the testing in a more passive way by just not showing up,so there is legitimate doubt that the teachers are giving us their best effort here.And as usual, we're really only getting one side of the story,so I think there is a real question about whether this is a manufactured crisis.

ScienceTeacher671
ScienceTeacher671

@Astropig @popacorn @MaureenDowney So, Pig, you think it's just fine that the state (1) has spent piles of money on an unproven test that can't be compared with anything when they could have used something like the ITBS, and (2) is requiring the tests to be taken on computers when many counties can't afford a full school year, much less the computers and bandwidth to do the testing properly?

popacorn
popacorn

@Astropig

Sure seems that way. Instead of embracing technology and working to make it viable, criticize and sabotage it, thereby rendering the wreckage useless to gauge the effectiveness of teachers. Creating the Excuse, so to speak. 

Astropig
Astropig

@ScienceTeacher671 @Astropig @popacorn @MaureenDowney


I think teacher's hearts are not in this. They're like union employees trying to "make the plant run backwards". Eads gave the game away with his comment. He's irresponsibly advocating intentional absences or opting out for his own selfish ends (because he's opposed to testing).



Astropig
Astropig

@popacorn @Astropig


The organized outrage here tells me that there is an intention to throw sand in the gears of the Milestones tests. This is just too pat. There are people here that I've never seen before. It's about as organized as GAE members get,but it's organized.

athensareateacher
athensareateacher

@Astropig @ScienceTeacher671 @popacorn @MaureenDowney If "buy in" is one of the goals, stakeholders should be involved in the decision making process.  


The perception that teachers are sabotaging this process further illustrates my concerns with the way this was rolled out.  The perceptions held by the general public, teachers, students and administrators are impacted when new initiatives are hastily rolled out from the top down with no questions asked.  Involving stakeholders and increasing transparency would go a long way towards buy in of various groups.  Whether we are talking about a new set of standards, new testing regime or a new teacher evaluation system, involving stakeholders in the the decision making process seems not only logical but necessary.  


Personal thought:


Please know that this is coming from a good place.  For the sake of playing nice, could we avoid making blanket statements about an entire group of people?  To say that "teachers" believe something when one individual makes a comment is sensational.  Also, when addressing an individual, it would be nice if we could remain polite and stick to the topic rather than speculating on the reason another individual may have an opinion or stooping to describing folks with unflattering adjectives.  This is feels a little like name calling on the playground and it takes the "fun" out of what could be a healthy discussion about education. 


Thank you!

MaureenDowney
MaureenDowney moderator

@popacorn Your comment doesn't say anything about the issue. The issue isn't blaming technology; the issue is whether the state of Georgia has the technology needed to enable online testing. I cannot imagine the added stress on test takers when the technology fails -- I have been in a timed setting when the computers don't work and it is a horrible situation.

Here is what you are saying in effect: If you bought a brand new house and the lights didn't work or the AC went off and on, don't look to to blame the electrician or the HVAC company. Instead, look to the homeowner. 

Do you really believe that? 

Linda Lee Smith
Linda Lee Smith

@popacorn @MaureenDowney  Clearly you are not versed in the importance of research reliability, validity and how it impacts students.  If you were educated in this area you would understand that students in Georgia (and many other states) are losing valuable time taking high-stakes tests that have not been proven effective in measuring student learning.  (In fact in most instances -there are a few isolated exceptions - such testing is detrimental),


Also, your understanding of technology seems a little short.  (Shorter than mine and I am definitely not a techno-wizard).  You can have a computer for every student, but if the internet band width isn't sufficient, renting "a whole bunch of computers" isn't going to correct anything.  Further, faulty technology is just one additional component of a failed system of reform.  Before our students' long-term future is affected by potentially damaging multiple high-stakes testing sessions and insufficient internet service, perhaps forums of educators and policy makers should have collaborated to develop educational reform based on documented, replicated research findings.    


Just to be clear, the debate is over what is best for our children.  I believe I speak for many parents and educational professionals when I say for most of us the concern is whether what is best for our children is even a passing thought for the government officials DICTATING policy.  


And finally, I'm sure that NASA conquered many, many difficulties when advancing the space program; however, in all social-science research it is unethical to involve humans, in this case (our children), in any scientific exploration that may cause them harm.  Using unvetted tests implemented without integrity would fall into that category.  

straker
straker

"who will watch the watchers"


Just follow the money.

SouthGATeacher
SouthGATeacher

It's about the teacher evaluations and the vendor.