After your kids take the Milestones, who grades them? You might be surprised.

With the Milestones scores coming in this week, I thought this issue was worth revisiting.

Along with thousands of other Georgia high school students, my twins are taking the Georgia Milestones this week. I never thought much about who grades these state tests until several parents and teachers reported seeing ads in local publications for graders placed by Data Recognition Corporation, Georgia’s testing contractor.

The ad states:

Data Recognition Corporation is a national leader in educational testing and we are seeking temporary Test Scorers to assist us in our busy assessment season. You will be scoring standardized tests that children take to comply with the No Child Left Behind Act. These tests are administered to elementary through high school grades in the subjects of reading, writing and math. You must have a four-year degree to qualify for this position. The pay is $12.00 per hour plus eligibility to earn an attendance bonus, which would bring your rate to $13.25 hour. We offer paid training and a pleasant work atmosphere.

We wondered about hiring temporary graders so my AJC colleague Ty Tagami consulted testing expert Greg Cizek at the University of North Carolina at Chapel Hill. Cizek was an expert witness in the Atlanta Public Schools cheating trial.

Tagami says:

Dr. Cizek told me the testing company that placed ad is well-known and well-reputed. He also told me it would be standard practice to employ part-time scorers with undergraduate degrees and training by the company – so long as their degrees are in a subject related to the tested material.

A Fulton mother went well beyond just reading the ad; she applied for one of the jobs to score the student responses that cannot be graded electronically, the short answers and essays. Alarmed by her experience, the parent shared a detailed account with the state Department of Education and legislators. And now all of you.

Here is her account, slightly edited for space:

I was made aware this past week that DRC is in charge of scoring the Milestones tests. They have put ads out on Craigslist and other sites. I wanted to know what the qualifications were to be hired to be a test scorer. I submitted my resume on April 18. I received an email back that they received my resume and I needed to attend one of the job fairs at DRC  in Smyrna.

They asked me to bring proof of my degree by bringing a copy of the original diploma, official transcripts or an unexpired teaching license. At the job fair, they said the pay was $13/hour with a potential of a $50 bonus, so it would equal $14.25/hour. (Note from Maureen: Her bachelor’s degree is in fashion merchandising.)

There were about 15 of us and they brought us to a room, handed us the application, a form stating the information we submitted was accurate, a math and English test that we had to take and two other optional forms to fill out. After sitting down, I got my phone out and took pictures of the math and English test. There were two girls who handed everything out and they were not paying attention so they never said anything.

I started to complete the four math questions and, honestly, did not know how to answer one. It asked if you had dough to fill a 10 inch x 14 inch pan, what size circular pan would you need to fill the same area and thickness. It’s been awhile since I have done equations so I was unsure. I got my phone out and Googled the answer. No one said anything to me. They did a five-minute presentation about DRC and shared what the position would entail.

My takeaway from the presentation was that they are looking for people during the day from 9:30 to 4 with one unpaid 30-minute break and two 15-minute paid breaks. They were also looking for people in the evening from 5:30 to 9:30 and you would receive one 15-minute paid break. They said we would all be in a room together and sitting closely. They also said you could receive a potential $50 bonus if you showed up for your shift every day and that they would assess you on the quantity you graded.

From my understanding, you have to score/grade a certain number of tests within the time-frame that you work. In the presentation, it said you will be assessed on quantity. I then had a one-on-one interview in a break room where people were able to come in and out to get drinks and snacks. In that five-minute interview, I was asked about my previous employment and why I thought I was qualified for the job.

I was also asked if I had ever received criticism from a supervisor and how I handled it. I was then asked if I would be comfortable being in “close quarters” with the other test scorers. They also asked if I would be able to grade the same question over and over and over and over again.

I was told I would receive a letter in the mail letting me know if they would offer me the position. Please note this was on Thursday and the training begins next Wednesday. So, in six days, I assume they will look over my resume, check my references, verify I have a college degree and grade my math and English test and be able to get a letter in the mail in enough time to start training next Wednesday.

When they asked everyone for a copy of their college diplomas (or an official transcript or an unexpired teaching license), only about 6 of the 15 people had brought it. The employees at DRC told them not to worry that they would just need to see a copy before the training starts.

With all that being said, I am very concerned about the quality of the potential test scorers. For the Georgia Milestones to be such a high-stakes test and for all the work and time the teachers and administration have put into getting the children prepared and this is how they are going to interview potential test scorers.

I have never contacted anyone at the Department of Education but I feel so strongly about this that I could not just sit back.

The parent would have been grading tests from Ohio and Pennsylvania. The Georgia tests are graded in Indiana, Ohio and Wisconsin. Still, the parent said, “It concerns me the same thing is happening at another of their scoring sites where it will be graded.”

After the mom shared her experiences with Melissa Fincher, DOE deputy superintendent for assessment and accountability, Fincher responded with an email saying she planned to talk to DRC, adding, “…it concerns me as well which is why I have escalated your report to the president.”

In another email, Fincher said:

DRC has opened a temporary site in Georgia but this site will not score our state assessment.  Although this particular site will not score our state assessments, I certainly can understand the concerns you express based on your experiences.  It concerns me as well.

I will share your email with the President of DRC directly as well as those responsible for their hand-scoring operation.  The experience you describe is not acceptable.

Please note that our staff monitors, closely, the scoring process of our state assessment – given the stakes you mention.  We travel to our scoring sites to observe scoring and monitor the scoring metrics (which are based on accuracy, not quantity!) each and every day.  In addition, we are able to read behind scorers to ensure accuracy.

In response to questions from legislators this week about the qualifications of the temps hired to score the Milestones, DOE is sending out this statement; it differs a bit from the experiences of the Fulton parent.

The scoring process does utilize temporary workers but there is a detailed screening process that includes verifying college degrees (and they do match related majors to subject matter to be scored).  Scorers must demonstrate competency in the subject matter through completion of a content area test/task.  Scorers are trained using student responses that have been scored by committees of Georgia teachers (during a process called range finding).  Scorers must demonstrate that they can apply the rubric as our teachers would.  Essentially, the scorers score sets of student responses (called qualifying sets – this serves as a qualifying test for employment) and must assign the same rubric score as our teachers did. They can only begin scoring if they qualify.

Once scoring is underway, there are many checks that are conducted to ensure scorers maintain fidelity to the scoring rubric.  Metrics are closely monitored at both the item and individual scorer level.  We (GaDOE) receive daily reports.  We also visit the scoring sites to monitor scoring.  Georgia follows industry standards – and this type of scoring has gone on for years in Georgia with our former writing assessments. This process is not new.

Many of the scorers are retired professionals – many of whom are retired or former educators.  This isn’t the case of just anyone can score….there is a very detailed process that is followed from start to finish.

In talking to parents, many assumed former teachers graded all tests. Do the graders have to be educators? Does it matter to you?

 

 

Reader Comments 0

67 comments
mensa_dropout
mensa_dropout

Please note that our staff monitors, closely, the scoring process of our state assessment – given the stakes you mention.

The fact that she is using an adverb as an absolute seals my discomfort with her choosing people to grade essays. 

Reba Smith3253
Reba Smith3253

Who is scoring definitely matters for all 11th graders who wrote online essays.  Who is qualified to read and grade essays?  At 12.00 an hour?  Seems like a long process whomever they hire.  Does the student get to challenge the grade?  If you don't teach how to write essays how can you grade them?


I hate this!

AnotherMom
AnotherMom

@Maureen - when will the results be posted?

Starik
Starik

Maybe it's time to go to Common Core?

MaryWalker
MaryWalker

What matters to me is that we are assuming that Data Recognition Corporation has ANY idea what progress a student has made in any given year. The whole idea that a test created by an industrial testing complex should determine anything about either our students or teachers is patently false. I am not so sure that our schools are failing, given the enormous challenges that some of our children face, but I am absolutely sure that fixing any issues that are present has nothing to do with a standardized test. It has everything to do with empowered teachers, caring administrators, and engaged parents. Because many state legislatures are in the thrall of national groups such as ALEC which encourage this kind of testing to "prove " that schools are failing so that they may be privatized, parents are the one stakeholder group that can stop this. Refuse to let your child be a pawn in this rigged game by refusing to let him or her be tested. There is a national Opt-out movement which provides information on how to do this. 

The grading of the tests by low-paid non-professionals is only a symptom of the pernicious nature of this process. Stop high-stakes testing and start letting teachers spend time designing and assessing a meaningful education for every child. 

Lori Evoy Reaves
Lori Evoy Reaves

A Cobb County friend (teacher) told me that the scorers only had to have a HS diploma?

Anne Hendricks Childress
Anne Hendricks Childress

I was a grader -scorer - last year on the PARCC. I applied several times via the Pearson website for two years and finally got the job. I scored high school essay. We had a strict rubric - well done - and I took time to bone up on the specific themes to best remember the works of literature and what the kids were needing to do when the essays were to come. Now, that was my personal conviction. I can't say what I graded, but I will say 19th century American poets of a well known group. I already knew them well - but I had some extra things I wanted to know. Because, anything I do - I do well. My NBCT didn't come out of Cracker Jack box.

I worked from April to June. The money was not the best, but I worked as often as I wanted. Yes, if you graded faster, it was good. I didn't. I took my time. Because at the end of each PARCC test was a kid. I saw excellent, I saw bad, and I saw even one girl do a political theme with anti-testing. She was the very best one I read. I told her so in the feedback.

One odd thing - when I had handwritten essays I thought to be dysgraphia, I sent it to the leader with an alert. Because I know what a dysgraphic child writes like. I teach one, my #1 learner, every day. I felt that conviction - because I am a teacher.

For teachers who are nervous about scorers, I found my experience rewarding. My fellow team were everything from retired to new college graduates to stay at home mothers (all with degrees - we all were English or Communications for what we were doing). I had many homeschooling moms (all certified teachers I encountered), like me, who were anti-testing but liked the extra income and the opportunity to help in our field, and learn some things too as we do our convictions but may return to the government school scene.

As for recruitment: I've seen bad locations such as Craigslist and I've seen great locations like LinkedIn, Indeed, and a few others. End here: I did tests in Northern states, I taught for 16 years as a school librarian, and thank you for my two cents.


jerryeads
jerryeads

Love it when this comes up. with measurement/research/statistics Ph.D. I wandered into working with one of the major testing company efforts to figure out writing assessment back in 1981. Interesting stuff. I have NEVER darkened the front of a P-12 classroom. Even back then, the company grabbed anybody with a warm bachelor's degree to score papers. 

That said, two of my later colleagues during another enterprise started their own testing company; their main effort was toward scoring writing assessment. Got to watch (a bit) the trials and tribulations of their doing the doing the job well (and yet efficiently enough to be competitive so that they survived). I also learned from them in observing a few other companies' efforts how it can so easily NOT be done well.

I can't know how the present contractor is scoring Millstones. In fairness, if they are doing it well, it really does not matter from which state the scorers are drawn. We DO know how to do this. I also do not know who they're finding to score the tests, but my guess is they won't be all teachers with 20 years experience.

SOME of the error variance will be due to variation in scorer experience. But I WILL guarantee you that the error variance is so huge that for those folks anywhere near the "cut score" that you might as well be flipping coins to decide who goes on and (in reality) who drops out to become a further burden on society. The system is egregiously inaccurate and ensures that the rich get richer and the poor get poorer. Sorry, it is what it is.

Dorothy Lozano
Dorothy Lozano

Wish i could read it free without having to subscribe to AJC

Starik
Starik

I wish these freeloading facebookers would go away.

Another comment
Another comment

What is really important to note is that almost all of these legislators and politicians exempt their own children from these testing requirements. They also avoided Kathy Cox's (r) Math 1,2, 3 fiasco the same way. Their children and grandchildren attend private and Christian schools who aren't required and don't give these test.

I know for a fact that Johnnie Isacksons children who are 15- 16 year old range have been at Woodward since Prek. Even the high performing east Cobb schools aren't acceptable. They have been joined down their by DA Paul Howard's Children. Then the biggest slap is his wife has been drawing a high paid administrators check from Atlanta Public Schools, but they could not send their children. Of course, their is Earl Erhart who is the head of one of those Private Christian school scholarships, that he sponsored the legislation for. He had to make sure he and his deserving part-time legislators get the scholarships ( he is one of the same guys who cut the amount of child support mothers can get, too. Counting the Fulltime custodial mothers income against her child support payment. The fathers just have to claim in court I am going to see my kid this percentage of time. Then the mothers income gets used. The father never bothers to pick up the kids, too busy dating or with wife 2, 3, 4. They know you can only get a child support modification every 3 years, and with self employment they can zero out income).

Marist, Pace, Lovett, Westminster, Holy Innocents, North Cobb Christian, All the Catholic Schools, Greater Atlanta Christian etc.. Do not ad mister any of these tests to their students. They only give the IOWA tests. Then at the 7th and 9th grade level most require new admissions that did not start in Pre K or K to take the SSAT for admission. They then require their students to take the PSAT to see where they are at followed by the SAT or ACT. They then brag about 100% college acceptance. These are the schools in the metro areas who get the students admitted to the IVies. They get the most admissions to GT and Georgia.

eulb
eulb

Thanks for publishing this Fulton parent's report about how the Ga Milestones tests are scored and how the scorers are selected.  I had assumed all mandatory Georgia tests are scored in Georgia by educators who are accustomed to evaluating students' work.  I'm really surprised that neither assumption is true.  


I remember the anti-Common Core camp's demand that our children's tests be "Georgia grown and Georgia owned." Are they ok with scoring done in Indiana, Ohio, and Wisconsin?

I'd love to read an update on whether this fashion merchandising major is offered a test-scoring job.  If the company really does match applicants' degrees to test content area, I would think her application went straight to the "unqualified" stack before she ever started taking the test.  Also, I would assume the testers saw her use her phone while answering taking the test, so she should be weeded out on those grounds, too.


I'm not surprised that they treated her cordially and respectfully during the application process, will be surprised if a job offer is forthcoming.  Would love to know how it turns out. 



Denise McNeil
Denise McNeil

The most disgusting part as a teacher, we never know what a great nor poor essay looks like. We have 1000', literally 1000's of topics we can touch on for a 70 question test. Shoot in the dark and hope we, as teachers and students, hit the target. It's a crap shoot!

proudparent01
proudparent01

The real story is about this insane parent. Mom was so concerned about Georgia Milestones that she applied to score the tests. This is taking helicopter parenting to hypercopter parenting. 

mensa_dropout
mensa_dropout

@proudparent01 No, this is a parent who sees what a sham the testing is and wants to let folks know about it. I find her to be a hero, actually. 

mensa_dropout
mensa_dropout

So, lemme get this straight:

Teachers' jobs are on the line to ensure that standards are taught.

Teachers' pay is directly impacted by how well the students do on the tests. 

Teachers in the great state of Georgia are required to have a certification in the field they teach.

And someone off the street with a degree in Parks and Rec (no offense...I love Park Rangers) is qualified to grade the tests that directly impact teacher pay.


Students' class grade averages are 20% of the EOC Milestones. 

Students who take AP classes also take the EOC.

Students' overall GPA is directly impacted by the scores, thereby affecting their HOPE Scholarship.

And someone off the street with a degree in music (no offense...music soothes the soul) is qualified to grade these tests that directly impact a student's GPA and HOPE Scholarship?


What did I miss?


Oh, wait.  I missed the fact that this is insulting to all parties concerned:  students, teachers, and parents. 

It is disgusting, it is immoral, and it is beyond the pale for all educators, teachers, and parents. 

SHAME on you DOE.  Shame on you GOSA.  Shame on you every single legislator who is involved in the sham you call Education Committees.  


And for those who think that charter schools do better, I don't disaggree, but those kids have to take the test, too.  Whatcha gonna do when YOUR kid in YOUR charter school fails the test because some yahoo with a BA in Philosphy decides that the essay isn't up to par?


Test the kids!  Fine! But don't you DARE say that the tests are valid or that the grading is accurate.  Don't you DARE say that your third grader sitting in a room for two hours a day is being assessed in a manner that is conducive to his or her learning. Don't you DARE say that judging a teacher's pay on some nebulous grading from someone who is NOT trained in pedagogy or content is good for education.  None of it is true, and no matter how you slice it, you can't say otherwise. 


I'm pissed beyond belief. 

Beach Bound2020
Beach Bound2020

@mensa_dropout Sorry but you left out two large groups who should harbor the most shame as they are the only ones who can change this. First, shame on the hundreds of thousands of parents who allow their children to be held captive for three to four weeks a year in a building with no instruction for the sole purpose of taking these ridiculous tests.  Again parents - go visit your schools anytime after Spring Break.  Not enough of you are even close to doing so or there would be an outrage.


Second, shame on the millions of voters who allows billions to be spent on the GADOE who implements this and allows your tax dollars not go to teach our children, but go to creating factories of test takers who have yet in the last decade shown enough knowledge to get us about 45th in public education in this country.


These are the ONLY two groups who can make a change in public education. 


You are pissed? Go have a conversation with a high school student who has taken SLOs, EOCs and for some AP tests.

Moreofthesame
Moreofthesame

@Beach Bound2020 @mensa_dropout  My high school sophomore is taking all those tests, it's absurd.  Yes, I know, the AP exam is "optional", but quite frankly, it's the only one that is worth it.  On one of the EOC's the teacher didn't even cover all of the material, yet it's a mandatory 20% of the grade.  Let's hold my child accountable for material that was never taught!  And, my child still has to take a final exam in that class.  I am disgusted, but my child loves the school and doesn't want to switch.  I, am glad we only have 2 more years of this...

Tammy Lollar
Tammy Lollar

I don't know?? When I taught, we didn't take that test. Sorry

Another comment
Another comment

$12 per hr is inadequate pay, especially in the Smyrna Vinings area. My daughter who just completed her junior year of college makes between $15-18 hr. Babysitting in Buckhead and Sandy springs. She has made at least $12 hr babysitting since she was a junior in high school in Smyrna and Dunwoody area. When she worked at a national coffee chain even 4-5 years ago the girls with the UGA drama and Lit degrees who couldn't find a real job were stolen from the coffee shop for full time Nanny jobs of $15-18/hr off the books.

$12 /hr or a maybe $50 bonus isn't enough to drop everything and come in never miss a minute, for a job that is short term and offers no carrot of a long term. It also locks you out of the prime time start of summer hiring season.

Right now you can pick up tutoring jobs for $40/hr. Degrees and teaching certificates aren't even required, just subject matter knowledge.

taylor48
taylor48

I know everyone wants the scores back as quickly as possibly, but having teachers grade the tests in the summer wouldn't be a bad idea.  I don't have time in the spring, but I'd be willing to earn some extra money in the summer time.

redweather
redweather

@taylor48 I don't know if this would be such a good idea in light of the APS cheating scandal. 

Milo
Milo

Look on the bright side, teachers. Add this to your list of things/people to blame. 

bu22
bu22

I wondered how they would get the essay portion graded in any rational length of time.

Amy Blafer
Amy Blafer

The person should not have to be a teacher, but a degree in math to grade a math test is important. There are multiple representations for ever answers. A person with limited understanding of mathematics will not understand that more than one line of thought is a valid explanation on an open ended question. The state has chosen to tie 50% of teacher pay to these results and 70% of administrator pay to these tests. Scoring quality is important as neither a committee of teachers or administrators are allowed to verify a randomly selected subset of tests upon which their pay depends.

redweather
redweather

This column doesn't explain what scorers will be required to do. Are they evaluating essays, for example, or simply checking multiple choice answer sheets?

HollyJones
HollyJones

@redweather The multiple choice are graded by computer, These scorers are grading the free response sections- essays and short answers. 

As a comparison, AP exams have free response sections that are hand graded by teachers in that field (e.g. English teachers grade the American Lit exam). More than one scorer grades each question using the AP rubric, which is fairly clear and easy to apply (I used it when I taught and AP class),  and if there is a vast discrepancy, those scorers discuss their scores to come to an agreement on the grade. I believe the scorers gather for about a week and work all day to grade.  I never heard of a quantity quota, but you are talking about a significantly lower number of exams to grade, so the number graded each day is likely not as crucial.  Sounds  a bit more thorough than the DRC process, IMO. 

Vincent M Smith
Vincent M Smith

We get it AJC, you are staunchly opposed to the Georgia Milestones Assessment. There's been non-stop propaganda for over a week!

AJC  Get Schooled
AJC Get Schooled

Wanted to add the cost of these tests: Georgia taxpayers are paying $107.8-million for a five-year contract with test makers.

mensa_dropout
mensa_dropout

And if the elemetary schools call before 9:00 Indiana time, they are told to call back when tech is there. Meanwhile, back at the ranch, the kid(s) can't test and have to come back later so that the techs in Indiana can have that last cup of coffee.

AND the weekend before the test, it was closed all day on a Saturday, keeping the coordinators from working on them.  107.8 mill should mean that the site is looked at over night, like from Midnight to 8am on the weekend.  NOPE!  Gotta save some money. 

AJC  Get Schooled
AJC Get Schooled

Not opposed but this is only year 2 of the new tests Georgia created because it decided it could create its own tests cheaper than a national consortium of states. Many experts warned the state needed a longer pilot and roll-out before attaching high-stakes consequences to tests. It appears those experts made that point because new tests and a statewide move to online testing come fraught with problems. The AJC is bound to report those problems as they are consequential. These tests, while cheaper than the PARCC tests, cost millions and taxpayers are footing the bill. Student grades and teacher evals hinge on them -- problems have to be reported. And we have been reporting this for a week because the tests are happening now so the news is happening now. The fact kids are going into their third day of repeat testing because of computer failures is news. Our old tests were graded electronically because they were multiple choice. We have now expanded to essay and short answers -- and have to figure out how to grade these more involved responses. Most parents believed teachers graded their children's work, not hourly workers. So, this is valid news and important news.

Susan Culbreth
Susan Culbreth

This is a travesty! So many children worried sick over this useless test. Then who grades them? NONteachers! The powerbrokers who encourage standardized tests should be ashamed for what they are doing to children.

Milo
Milo

If you're afraid of the results, by all means opt out. Ignorance is bliss. Just ask your kid. 

gapeach101
gapeach101

Great article Ms. Downey.  

I wish I was surprised. Not to say that I knew this.  I just wish it was a surprise. 

dcdcdc
dcdcdc

Wow, the eduacracy is obviously scared to death to have the value that they deliver to students actually be objectively measured. 


And they have their "AJC champion" bashing away at any student testing that might show how badly they are failing - in spite of the tripling - 300% - in per student funding over the past 30 years.


Keep at it.  I'm sure parents will agree that it's "the test's fault".  Not the approach that the eduacracy is forcing on teachers - don't discipline students, especially if they are "minority", advance them even if they can't do the work, and if you fail them then the admin will step in and change their grades so the kids pass, and can "feel good about themselves".


Meanwhile, other orgs will continue to start schools that actually enforce discipline, ensure learning, and don't let kids skate by and not learn.   And parents who give a darn about their kids lives will move their kids to these establishments.


That is clearly incredibly scary to the eduacracy, which clearly thinks our public school system is really in place to benefit.

gapeach101
gapeach101

@dcdcdc

Minimum wage has increase 220% in the past 30 years.  The responsibilities dumped on the schools has increased, I don't know, 4 fold in the past 30 years.  So, what's your complaint about the tripling of funding in the last 30 years?

Wascatlady
Wascatlady

@dcdcdc Another hit piece by dcdc, who seems to be unable to read anything without seeing a conspiracy.

dcdcdc
dcdcdc

@dcdcdc Right, because our schools are doing such an amazing job now.  Said no one who looks at the actual data - ever.  Which obviously is why objective data so scares the eduacracy.


Personally, I look forward to a day when actually effective teachers can form a school, and run it w/out eduacracy interference - focus on discipline, learning, and performance.


But that clearly won't come from the "eduacracy apologists" that flood this "education blog" (cough cough - I mean, this "the important thing in public schools is that the adults who control it, don't actually teach in the classroom,  and make a living from it are never actually held accountable)

redweather
redweather

@dcdcdc Climb down off that hobby horse every once in a while and feel what it's like to have your feet on the ground.

elementary-pal
elementary-pal

@dcdcdc  I can't decide if I should laugh or cry at your comments.  I can only say that you need to come spend a day in our school and get an idea of what REALLY happens every day. 

Milo
Milo

@dcdcdc Nice comment, fun to watch this bunch circle the wagons.