Amended Nationals Appeals Policy

User avatar
Unome
Moderator
Moderator
Posts: 4030
Joined: January 26th, 2014, 12:48 pm
Division: Grad
State: GA
Location: somewhere in the sciolyverse

Amended Nationals Appeals Policy

Postby Unome » April 23rd, 2018, 4:26 pm

soinc.org wrote:At the end of the Science Olympiad National Tournament Awards Ceremony each team’s head coach will be provided one copy of the final scores. Within one hour after the ceremony is completed the head coach may submit compelling evidence of a scoring inconsistency using this Team Ranking Inquiry Form (will be a live link at the 2018 National Tournament). If the evidence is verified, the appropriate points, medals and trophies will be awarded for that team only. Thinking that your team "did better" than scores reflect is not considered compelling evidence. Teams will not be asked to return awards or to relinquish any points. Scores for the Science Olympiad National Tournament are not official until they are posted on the soinc.org website. For more information about Science Olympiad Scoring, please visit our Policies section.


The obvious question is of course "what prompted this?", and the answer likely has to do with the second-to-last and last bolded sections. The relinquishing of points part looks unusual at first glance but appears to be consistent with past scoring corrections (e.g. 2013 with Booth and 2014 with SAA).
Userpage
Chattahoochee High School Class of 2018
Georgia Tech Class of 2022

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.

CMS AC
Member
Member
Posts: 151
Joined: January 10th, 2017, 3:54 pm
Division: B
State: NJ

Re: Amended Nationals Appeals Policy

Postby CMS AC » April 23rd, 2018, 5:57 pm

What would be an example of "compelling evidence?"

User avatar
daydreamer0023
Member
Member
Posts: 194
Joined: January 29th, 2015, 5:44 pm
Division: Grad
State: -
Location: I have no idea where I am, but I can tell you exactly how fast I'm going ;)

Re: Amended Nationals Appeals Policy

Postby daydreamer0023 » April 23rd, 2018, 6:00 pm

CMS AC wrote:What would be an example of "compelling evidence?"


Knowing another team had a visibly worse build run than you, yet they placed much, much higher point wise.
"I am among those who think that science has great beauty. A scientist in his laboratory is not only a technician: he is also a child placed before natural phenomena which impress him like a fairy tale." - Marie Curie

User avatar
dxu46
Exalted Member
Exalted Member
Posts: 774
Joined: April 11th, 2017, 6:55 pm
Division: C
State: MO
Location: telling young innocent middle schoolers to sleep while they can

Re: Amended Nationals Appeals Policy

Postby dxu46 » April 23rd, 2018, 6:09 pm

daydreamer0023 wrote:
CMS AC wrote:What would be an example of "compelling evidence?"


Knowing another team had a visibly worse build run than you, yet they placed much, much higher point wise.

Doesn't that qualify as "thinking your team did better?"
Ladue Science Olympiad!
tHe YeAr Of ThE gRiNd Is HeRe
"Beer is proof that God loves us and wants us to be happy." - Benjamin Franklin
Summary of 2019 season: :cry: :lol: :x :oops: :( 8-) :geek: :D :!: :cry:

User avatar
daydreamer0023
Member
Member
Posts: 194
Joined: January 29th, 2015, 5:44 pm
Division: Grad
State: -
Location: I have no idea where I am, but I can tell you exactly how fast I'm going ;)

Re: Amended Nationals Appeals Policy

Postby daydreamer0023 » April 23rd, 2018, 6:30 pm

dxu46 wrote:
daydreamer0023 wrote:
CMS AC wrote:What would be an example of "compelling evidence?"


Knowing another team had a visibly worse build run than you, yet they placed much, much higher point wise.

Doesn't that qualify as "thinking your team did better?"


Not necessarily. For example, if you see your Helicopter get 2 min in the air and know you got all the bonuses...then saw someone's Heli get 14 seconds with no bonuses (which you can notice fairly easily based on watching their run)...and they get 10 places higher then you...that is grounds for argument.
"I am among those who think that science has great beauty. A scientist in his laboratory is not only a technician: he is also a child placed before natural phenomena which impress him like a fairy tale." - Marie Curie

hippo9
Member
Member
Posts: 242
Joined: March 12th, 2018, 9:35 am
Division: C
State: IN

Re: Amended Nationals Appeals Policy

Postby hippo9 » April 23rd, 2018, 6:31 pm

daydreamer0023 wrote:
dxu46 wrote:
daydreamer0023 wrote:
Knowing another team had a visibly worse build run than you, yet they placed much, much higher point wise.

Doesn't that qualify as "thinking your team did better?"


Not necessarily. For example, if you see your Helicopter get 2 min in the air and know you got all the bonuses...then saw someone's Heli get 14 seconds with no bonuses (which you can notice fairly easily based on watching their run)...and they get 10 places higher then you...that is grounds for argument.

Also, it could be something like you could tell a team got tiered just by visibly looking at their device, but they still somehow beat you.
"If we knew what we were doing, it wouldn’t be called research." - Albert Einstein

2018: Battery Buggy, Road Scholar, Roller Coaster
2019: Chemistry Lab, Codebusters, Disease Detectives, Fossils, Geologic Mapping, Sounds of Music

User avatar
dxu46
Exalted Member
Exalted Member
Posts: 774
Joined: April 11th, 2017, 6:55 pm
Division: C
State: MO
Location: telling young innocent middle schoolers to sleep while they can

Re: Amended Nationals Appeals Policy

Postby dxu46 » April 23rd, 2018, 6:34 pm

hippo9 wrote:
daydreamer0023 wrote:
dxu46 wrote:Doesn't that qualify as "thinking your team did better?"


Not necessarily. For example, if you see your Helicopter get 2 min in the air and know you got all the bonuses...then saw someone's Heli get 14 seconds with no bonuses (which you can notice fairly easily based on watching their run)...and they get 10 places higher then you...that is grounds for argument.

Also, it could be something like you could tell a team got tiered just by visibly looking at their device, but they still somehow beat you.

Right, I forgot that most build events are open to the public.
Ladue Science Olympiad!
tHe YeAr Of ThE gRiNd Is HeRe
"Beer is proof that God loves us and wants us to be happy." - Benjamin Franklin
Summary of 2019 season: :cry: :lol: :x :oops: :( 8-) :geek: :D :!: :cry:

User avatar
rfscoach
Coach
Coach
Posts: 586
Joined: July 7th, 2008, 4:58 pm
Division: B
State: GA

Re: Amended Nationals Appeals Policy

Postby rfscoach » April 23rd, 2018, 6:43 pm

CMS AC wrote:What would be an example of "compelling evidence?"


In the case if JC Booth in 2013, it was being scored as a No Show in an event they participated in.
I am the Lorax. I speak for the trees. I speak for the trees, for the trees have no tongues.

SciNerd42
Member
Member
Posts: 75
Joined: March 15th, 2017, 6:14 am
State: -

Re: Amended Nationals Appeals Policy

Postby SciNerd42 » April 23rd, 2018, 8:22 pm

The problem (that has no solution), is that teams can tell if a mistake was made with building events (you knew you had a higher time, score, but were ranked lower) but with academic events, you have no evidence, and you just have to assume everyone was scored right.

Skink
Exalted Member
Exalted Member
Posts: 947
Joined: February 8th, 2009, 12:23 pm
Division: C
State: IL

Re: Amended Nationals Appeals Policy

Postby Skink » April 23rd, 2018, 8:27 pm

SciNerd42 wrote:The problem (that has no solution), is that teams can tell if a mistake was made with building events (you knew you had a higher time, score, but were ranked lower) but with academic events, you have no evidence, and you just have to assume everyone was scored right.

Yeah. And, this is a feature, not a bug. There's a beast of a slippery slope the moment they open up any subjectively scored events to prying eyes.

nicholasmaurer
Coach
Coach
Posts: 354
Joined: May 19th, 2017, 10:55 am
Division: Grad
State: OH
Location: Solon, OH

Re: Amended Nationals Appeals Policy

Postby nicholasmaurer » April 23rd, 2018, 8:53 pm

Skink wrote:
SciNerd42 wrote:The problem (that has no solution), is that teams can tell if a mistake was made with building events (you knew you had a higher time, score, but were ranked lower) but with academic events, you have no evidence, and you just have to assume everyone was scored right.

Yeah. And, this is a feature, not a bug. There's a beast of a slippery slope the moment they open up any subjectively scored events to prying eyes.


Yes and no. There are many valid reasons for wanting to keep the scoring process - particularly of test events - as a black box. If, for example, tests were passed back or raw scores were released, competitors would no doubt find numerous mistakes in answer keys, point totals, and rubrics. Some of the ensuing criticisms would be entirely valid. Others would likely be nit-picky, unreasonable, or plain wishful thinking.

However, I did have an experience at a National Tournament where I placed far worse (think 30+ places lower) in a test event than was reasonably possible. There is, of course, no "compelling evidence" to support this claim; it truly does rely on what the arbitration policy specifically excludes: "thinking that your team 'did better'." The difference in my performance from both that year's Ohio State Tournament, and the previous National Tournament, was far too wide to be entirely explained by a bad day or careless errors on my part. Most likely, a page of my exam was lost during scoring, or a math error was made when totaling points. As an ES, I know these mistakes happen more often than anyone would like.

Unfortunately, the current system offers no redress, or even closure/understanding, for competitors in this situation. I am not necessarily sure there is a better option available. When, as an ES or Tournament Director, you are handling thousands (or hundreds of thousands) of data points, many will doubtless be in error. However, it does concern me that the opacity of the scoring for test events leaves little incentive for quality grading or careful totaling. I'm sure the overwhelming majority of supervisors are diligent and careful, and I am sure all have good intentions. But the skeptic in me is always wary of a system whose accountability relies on one principle: trust us.
Assistant Coach and Alumni ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.

User avatar
windu34
Moderator
Moderator
Posts: 1287
Joined: April 19th, 2015, 6:37 pm
Division: Grad
State: FL
Location: Gainesville, Florida

Re: Amended Nationals Appeals Policy

Postby windu34 » April 23rd, 2018, 9:46 pm

nicholasmaurer wrote:
Yes and no. There are many valid reasons for wanting to keep the scoring process - particularly of test events - as a black box. If, for example, tests were passed back or raw scores were released, competitors would no doubt find numerous mistakes in answer keys, point totals, and rubrics. Some of the ensuing criticisms would be entirely valid. Others would likely be nit-picky, unreasonable, or plain wishful thinking.

However, I did have an experience at a National Tournament where I placed far worse (think 30+ places lower) in a test event than was reasonably possible. There is, of course, no "compelling evidence" to support this claim; it truly does rely on what the arbitration policy specifically excludes: "thinking that your team 'did better'." The difference in my performance from both that year's Ohio State Tournament, and the previous National Tournament, was far too wide to be entirely explained by a bad day or careless errors on my part. Most likely, a page of my exam was lost during scoring, or a math error was made when totaling points. As an ES, I know these mistakes happen more often than anyone would like.

Unfortunately, the current system offers no redress, or even closure/understanding, for competitors in this situation. I am not necessarily sure there is a better option available. When, as an ES or Tournament Director, you are handling thousands (or hundreds of thousands) of data points, many will doubtless be in error. However, it does concern me that the opacity of the scoring for test events leaves little incentive for quality grading or careful totaling. I'm sure the overwhelming majority of supervisors are diligent and careful, and I am sure all have good intentions. But the skeptic in me is always wary of a system whose accountability relies on one principle: trust us.

Definitely agree with this. I have certainly made grading mistakes on tests and I don't know how many went un-caught by me or the rest of my team. I have personally my own system when it comes to scoring tests for a study event: I grade all of the FRQ/Math/Subjective questions, volunteer #1 grades multiple choice and short answer, volunteer #2 double checks volunteer #1's grading and totals the points, volunteer #3 checks volunteer #2's point totaling. I believe this is the most thorough possible way to score tests to avoid as many mistakes as possible given a limited number of volunteers because every objective answer has 2 people look at it and any subjective answer has only one person looking at it to maintain consistency in partial credit. I don't personally know how other supervisors choose to grade their tests and I feel it is not as much of an "assembly line".

The grading process for tests does need to be a black box by necessity, but that doesn't mean the scoring process cannot be standardized/optimized to minimize mistakes as much as possible.
President of Science Olympiad at the University of Florida || Boca Raton Community High School Alumni
kevin@floridascienceolympiad.org
windu34's Userpage

Event Supervisor for 2019:
MIT - Mission Possible
Harvard - Sounds of Music
Princeton - Herpetology
Nationals - Circuit Lab

Ender1982
Member
Member
Posts: 40
Joined: February 4th, 2018, 8:10 pm
Division: C
State: IN

Re: Amended Nationals Appeals Policy

Postby Ender1982 » April 24th, 2018, 7:43 am

Ideally, for every event (at all competitions, not just nationals), there are 2 people in the room, one scorer, and one double-checker. We all make mistakes, but the truth is that a lot of us have spent hours/days/months preparing for competition, and we are owed the chance to have our scores be accurate. I totally get that giving tests back will be a slippery slope, so the solution is just to have everything double checked per event before it goes to scoring, then we as competitors can "trust the process"

Now for this to happen, perhaps an official "Event supervisors" rule book. Many tournaments say the 1-2 pgs are the official rules, and things we discuss on this board, don't get included in the rules. Science Olympiad puts out a ton of materials for Event Supervisors, but in general they are not read. Requiring the "Event supervisors follow guidelines produced by Science Olympiad" rule would help to minimize the incorrect scoring, bad tests, limited questions, etc. This is especially important because we don't get the tests back, and we need to "trust the process"

Yes, I'm aware that even then, some event supervisors will do what they want (volunteers and all) but it will help minimize at least state/national test discrepancies and help us all "trust the process"

User avatar
Unome
Moderator
Moderator
Posts: 4030
Joined: January 26th, 2014, 12:48 pm
Division: Grad
State: GA
Location: somewhere in the sciolyverse

Re: Amended Nationals Appeals Policy

Postby Unome » April 24th, 2018, 7:49 am

Ender1982 wrote:Now for this to happen, perhaps an official "Event supervisors" rule book. Many tournaments say the 1-2 pgs are the official rules, and things we discuss on this board, don't get included in the rules. Science Olympiad puts out a ton of materials for Event Supervisors, but in general they are not read. Requiring the "Event supervisors follow guidelines produced by Science Olympiad" rule would help to minimize the incorrect scoring, bad tests, limited questions, etc. This is especially important because we don't get the tests back, and we need to "trust the process"

This assumes that such a book can be produced at good quality. Based on my experiences with the Event Logistics manual produced by NSO, I personally would take any such NSO resource with several grains of salt.
Userpage
Chattahoochee High School Class of 2018
Georgia Tech Class of 2022

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.

Ender1982
Member
Member
Posts: 40
Joined: February 4th, 2018, 8:10 pm
Division: C
State: IN

Re: Amended Nationals Appeals Policy

Postby Ender1982 » April 24th, 2018, 7:55 am

This assumes that such a book can be produced at good quality. Based on my experiences with the Event Logistics manual produced by NSO, I personally would take any such NSO resource with several grains of salt.


But at least it would be something "official" that can be critiqued, and make sure that all supervisors follow, and if they don't, proper arbitration can happen.


Return to “2018 Nationals”

Who is online

Users browsing this forum: No registered users and 1 guest