Amended Nationals Appeals Policy

hippo9
Member
Member
Posts: 271
Joined: March 12th, 2018, 9:35 am
Division: C
State: IN
Has thanked: 1 time
Been thanked: 6 times

Re: Amended Nationals Appeals Policy

Post by hippo9 »

I think he just wants a system to be there to allow for a possible correction in study events because currently there is none.
2018: Battery Buggy, Road Scholar, Roller Coaster
2019: Chem Lab, Code, Disease, Fossils, Geo Maps, Sounds
2020 and 2021: Astro, Chem Lab, Code, Fossils, Geo Maps, Sounds

When you miss nats twice by a combined two points :|
User avatar
Unome
Moderator
Moderator
Posts: 4338
Joined: January 26th, 2014, 12:48 pm
Division: Grad
State: GA
Has thanked: 235 times
Been thanked: 85 times

Re: Amended Nationals Appeals Policy

Post by Unome »

Ender1982 wrote:
This assumes that such a book can be produced at good quality. Based on my experiences with the Event Logistics manual produced by NSO, I personally would take any such NSO resource with several grains of salt.
But at least it would be something "official" that can be critiqued, and make sure that all supervisors follow, and if they don't, proper arbitration can happen.
I intended my statement to be a little stronger than that. There are parts in the Event Logistics manual that I would consider outright counterproductive to running a good tournament at times (or at least, there were last year - I haven't looked at this year's manual).

Edit: Probably should be more specific. I refer in particular to the info on number of helpers, which is often significantly overstated for the vast majority of tournaments.
Userpage

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
chalker
Member
Member
Posts: 2107
Joined: January 9th, 2009, 7:30 pm
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 56 times

Re: Amended Nationals Appeals Policy

Post by chalker »

Unome wrote:The obvious question is of course "what prompted this?"
Other than the fact we have a webform now, nothing is really changing with the process. We are spelling out certain things more explicitly, but everything follows the same criteria and processes we have followed from year to year. I was the one who pushed for this to happen because of 2 reasons: 1. we didn't have a defined post-ceremony appeal process (which meant many coaches who were 'in the know' would run me down after the ceremony to file an appeal since they mostly filtered through me and 2. many of the appeals we got we rejected because there was no evidence or they were of the type that someone 'thought' they should have done better.

Student Alumni
National Event Supervisor
National Physical Sciences Rules Committee Chair
nicholasmaurer
Coach
Coach
Posts: 422
Joined: May 19th, 2017, 10:55 am
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 22 times

Re: Amended Nationals Appeals Policy

Post by nicholasmaurer »

chalker wrote:2. many of the appeals we got we rejected because there was no evidence or they were of the type that someone 'thought' they should have done better.
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).

However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
chalker
Member
Member
Posts: 2107
Joined: January 9th, 2009, 7:30 pm
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 56 times

Re: Amended Nationals Appeals Policy

Post by chalker »

nicholasmaurer wrote:
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).

However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
We do indeed look for major issues like reverse sort order, unexpected No Shows, etc. However looking for more nuanced outliers like a top 10 team placing low in an event is much harder. For example, if you look at the Div C nationals result from last year, you'll see that:

1. The gold medal team had 2 events they placed in the 20's
2. The silver medal team have 2 events they placed in the 30's
3. The 6th place team had 3 events in the 20's, 1 in the 30's and 1 in the 50's
4. The 8th place teams had 2 events in the 20's and 3 in the 40's
5. Conversely, the 38th place team got a silver medal in one event

I don't see an obvious algorithm we can implement beyond our normal process of carefully checking things multiple times. Of course if someone has a good idea, I'd be happy to try to implement it.

Student Alumni
National Event Supervisor
National Physical Sciences Rules Committee Chair
nicholasmaurer
Coach
Coach
Posts: 422
Joined: May 19th, 2017, 10:55 am
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 22 times

Re: Amended Nationals Appeals Policy

Post by nicholasmaurer »

chalker wrote:
nicholasmaurer wrote:
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).

However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
We do indeed look for major issues like reverse sort order, unexpected No Shows, etc. However looking for more nuanced outliers like a top 10 team placing low in an event is much harder. For example, if you look at the Div C nationals result from last year, you'll see that:

1. The gold medal team had 2 events they placed in the 20's
2. The silver medal team have 2 events they placed in the 30's
3. The 6th place team had 3 events in the 20's, 1 in the 30's and 1 in the 50's
4. The 8th place teams had 2 events in the 20's and 3 in the 40's
5. Conversely, the 38th place team got a silver medal in one event

I don't see an obvious algorithm we can implement beyond our normal process of carefully checking things multiple times. Of course if someone has a good idea, I'd be happy to try to implement it.
I would have to test out some different approaches to try and find an algorithm that doesn't turn up too many false positives, because that rapidly becomes too resource intensive for the scoring staff. Intuitively, I would start by looking for situations where dropping a team's worst event significantly reduces the standard deviation of their placements. It would likely have to be refined from there.
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
nicholasmaurer
Coach
Coach
Posts: 422
Joined: May 19th, 2017, 10:55 am
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 22 times

Re: Amended Nationals Appeals Policy

Post by nicholasmaurer »

chalker wrote:
nicholasmaurer wrote:
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).

However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
We do indeed look for major issues like reverse sort order, unexpected No Shows, etc. However looking for more nuanced outliers like a top 10 team placing low in an event is much harder. For example, if you look at the Div C nationals result from last year, you'll see that:

1. The gold medal team had 2 events they placed in the 20's
2. The silver medal team have 2 events they placed in the 30's
3. The 6th place team had 3 events in the 20's, 1 in the 30's and 1 in the 50's
4. The 8th place teams had 2 events in the 20's and 3 in the 40's
5. Conversely, the 38th place team got a silver medal in one event

I don't see an obvious algorithm we can implement beyond our normal process of carefully checking things multiple times. Of course if someone has a good idea, I'd be happy to try to implement it.
I ran some quick calculations using the placings from the 2017 National Tournament. For each event placing, I calculated the percent change in that team's overall standard deviation if you excluded that event.

You can set various thresholds, but if you find all cases where a team's standard deviation falls by more than 15% because you exclude that one event placing, you identify 16 cases for extra review. Nine of these cases are teams who placed substantially better than their mean placing; I'd be inclined to ignore these. This leaves seven outliers where a team did unusually poorly. One of these is a no-show, which you are already verifying. That would leave six individual scores for the scoring team to double-check.
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
nicholasmaurer
Coach
Coach
Posts: 422
Joined: May 19th, 2017, 10:55 am
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 22 times

Re: Amended Nationals Appeals Policy

Post by nicholasmaurer »

For those who care, the seven specific events my algorithm flagged were:

#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational

Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
DarthBuilder
Member
Member
Posts: 288
Joined: August 1st, 2017, 8:02 am
Division: Grad
State: IL
Has thanked: 0
Been thanked: 0

Re: Amended Nationals Appeals Policy

Post by DarthBuilder »

nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:

#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
Do we know the specific reasons for each flagged?
Deleted
chalker
Member
Member
Posts: 2107
Joined: January 9th, 2009, 7:30 pm
Division: Grad
State: OH
Has thanked: 1 time
Been thanked: 56 times

Re: Amended Nationals Appeals Policy

Post by chalker »

nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:

#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
This is very interesting... I'll have to think about how I'd actually go about potentially implementing it to automatically check within excel - any idea what the formula might be to accomplish this? Of course one issue is that it will only catch single event mistakes of a certain threshold... Of the list I previously provided, I think it only flagged 1 of the teams. However even incremental improvements are still improvements!

Student Alumni
National Event Supervisor
National Physical Sciences Rules Committee Chair
Locked

Return to “2018 Nationals”

Who is online

Users browsing this forum: No registered users and 2 guests