I would have to test out some different approaches to try and find an algorithm that doesn't turn up too many false positives, because that rapidly becomes too resource intensive for the scoring staff. Intuitively, I would start by looking for situations where dropping a team's worst event significantly reduces the standard deviation of their placements. It would likely have to be refined from there.chalker wrote:We do indeed look for major issues like reverse sort order, unexpected No Shows, etc. However looking for more nuanced outliers like a top 10 team placing low in an event is much harder. For example, if you look at the Div C nationals result from last year, you'll see that:nicholasmaurer wrote:
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).
However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
1. The gold medal team had 2 events they placed in the 20's
2. The silver medal team have 2 events they placed in the 30's
3. The 6th place team had 3 events in the 20's, 1 in the 30's and 1 in the 50's
4. The 8th place teams had 2 events in the 20's and 3 in the 40's
5. Conversely, the 38th place team got a silver medal in one event
I don't see an obvious algorithm we can implement beyond our normal process of carefully checking things multiple times. Of course if someone has a good idea, I'd be happy to try to implement it.
Amended Nationals Appeals Policy
-
- Coach
- Posts: 422
- Joined: May 19th, 2017, 10:55 am
- Division: Grad
- State: OH
- Has thanked: 1 time
- Been thanked: 22 times
Re: Amended Nationals Appeals Policy
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
-
- Coach
- Posts: 422
- Joined: May 19th, 2017, 10:55 am
- Division: Grad
- State: OH
- Has thanked: 1 time
- Been thanked: 22 times
Re: Amended Nationals Appeals Policy
I ran some quick calculations using the placings from the 2017 National Tournament. For each event placing, I calculated the percent change in that team's overall standard deviation if you excluded that event.chalker wrote:We do indeed look for major issues like reverse sort order, unexpected No Shows, etc. However looking for more nuanced outliers like a top 10 team placing low in an event is much harder. For example, if you look at the Div C nationals result from last year, you'll see that:nicholasmaurer wrote:
I do wish there was a better process for handling these types of situations. If I remember correctly, your scoring system does include some useful metrics than can help spot outlier data (e.g. if an event should be ranked with scores low-to-high, but is accidentally entered as the reverse).
However, I wonder if some additional review could be done if, for example, a team that is otherwise placing consistently in the top 10 has one event entered where they placed 50th. Is it possible they genuinely did that poorly? Sure. Anyone can have an off day. But I would think that result would be something worth reviewing to make sure there wasn't a clerical error along the way. This is less practical at invitationals, but since State and National Tournament awards are typically at a set time later in the evening, it is more possible there. Just a thought - maybe this already happens behind the scenes and I'm just not aware!
1. The gold medal team had 2 events they placed in the 20's
2. The silver medal team have 2 events they placed in the 30's
3. The 6th place team had 3 events in the 20's, 1 in the 30's and 1 in the 50's
4. The 8th place teams had 2 events in the 20's and 3 in the 40's
5. Conversely, the 38th place team got a silver medal in one event
I don't see an obvious algorithm we can implement beyond our normal process of carefully checking things multiple times. Of course if someone has a good idea, I'd be happy to try to implement it.
You can set various thresholds, but if you find all cases where a team's standard deviation falls by more than 15% because you exclude that one event placing, you identify 16 cases for extra review. Nine of these cases are teams who placed substantially better than their mean placing; I'd be inclined to ignore these. This leaves seven outliers where a team did unusually poorly. One of these is a no-show, which you are already verifying. That would leave six individual scores for the scoring team to double-check.
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
-
- Coach
- Posts: 422
- Joined: May 19th, 2017, 10:55 am
- Division: Grad
- State: OH
- Has thanked: 1 time
- Been thanked: 22 times
Re: Amended Nationals Appeals Policy
For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
-
- Member
- Posts: 288
- Joined: August 1st, 2017, 8:02 am
- Division: Grad
- State: IL
- Has thanked: 0
- Been thanked: 0
Re: Amended Nationals Appeals Policy
Do we know the specific reasons for each flagged?nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
Deleted
-
- Member
- Posts: 2107
- Joined: January 9th, 2009, 7:30 pm
- Division: Grad
- State: OH
- Has thanked: 1 time
- Been thanked: 56 times
Re: Amended Nationals Appeals Policy
This is very interesting... I'll have to think about how I'd actually go about potentially implementing it to automatically check within excel - any idea what the formula might be to accomplish this? Of course one issue is that it will only catch single event mistakes of a certain threshold... Of the list I previously provided, I think it only flagged 1 of the teams. However even incremental improvements are still improvements!nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
Student Alumni
National Event Supervisor
National Physical Sciences Rules Committee Chair
- Unome
- Moderator
- Posts: 4336
- Joined: January 26th, 2014, 12:48 pm
- Division: Grad
- State: GA
- Has thanked: 235 times
- Been thanked: 85 times
Re: Amended Nationals Appeals Policy
Just dropping this link here, in case anyone finds it useful. I came up with this specifically because I found standard deviation to be unhelpful for SO purposes.
- Ashernoel
- Member
- Posts: 345
- Joined: January 27th, 2017, 1:31 pm
- Division: Grad
- State: IL
- Has thanked: 0
- Been thanked: 2 times
Re: Amended Nationals Appeals Policy
nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
I can confirm that the 53rd in EV was due to a "construction violation" that was appealed but rejected.
The ruling was disgusting, relying not on proof but the word of the ES, and was based on an incredibly unclear section of the rules that had gone uncalled all year.
The run was also one of their best all year, and would have resulted in New Trier placing 6th or 7th overall.
The arbitration committee completely missed the ball and the ruling shows a failure in the national arbitration system.
Last edited by Ashernoel on April 24th, 2018, 7:34 pm, edited 1 time in total.
NT '19
Harvard '23
Harvard '23
- dxu46
- Exalted Member
- Posts: 809
- Joined: April 11th, 2017, 6:55 pm
- Division: C
- State: MO
- Pronouns: He/Him/His
- Has thanked: 2 times
- Been thanked: 7 times
Re: Amended Nationals Appeals Policy
What about #21 Ladue - 39th Electric Vehicle?nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
- Name
- Member
- Posts: 434
- Joined: January 21st, 2018, 4:41 pm
- Division: C
- State: NY
- Pronouns: He/Him/His
- Has thanked: 49 times
- Been thanked: 46 times
Re: Amended Nationals Appeals Policy
The only 30somethings in the list is northville harriton, 2nd and 4th overall. Thier high ranking is probably why they were caught and ladue wasn't high enough for a 39th to countdxu46 wrote:What about #21 Ladue - 39th Electric Vehicle?nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
South Woods MS, Syosset HS '21
BirdSO TD/ES
Past Events: Microbe, Invasive, Matsci, Fermi, Astro, Code, Fossils
BirdSO TD/ES
Past Events: Microbe, Invasive, Matsci, Fermi, Astro, Code, Fossils
1st place MIT Codebusters 2019-2020 1st place NYS Fermi Questions (2019), Astronomy and Codebusters (2021) Science Olympiad Founder's Scholarship winner
-
- Coach
- Posts: 422
- Joined: May 19th, 2017, 10:55 am
- Division: Grad
- State: OH
- Has thanked: 1 time
- Been thanked: 22 times
Re: Amended Nationals Appeals Policy
From the list you started with, it flagged the silver medal team placing 31st and the 6th place team placing 50th. It also technically flagged the 38th team placing second, but I excluded this as one of the places where a team did unexpectedly better. That list of seven is only teams where the outlier was worse than their mean.chalker wrote:This is very interesting... I'll have to think about how I'd actually go about potentially implementing it to automatically check within excel - any idea what the formula might be to accomplish this? Of course one issue is that it will only catch single event mistakes of a certain threshold... Of the list I previously provided, I think it only flagged 1 of the teams. However even incremental improvements are still improvements!nicholasmaurer wrote:For those who care, the seven specific events my algorithm flagged were:
#7 Northville - 38th Ecology
#9 New Trier - 53rd Electric Vehicle
#11 Harriton - 31st Experimental Design
#20 Clements - 50th Write It, Do It
#24 Munster - 48th Optics
#38 ABRHS - 60th Electric Vehicle
#50 Clark - 54th Write It, Do It
In terms of calculation, I did all of my work in Excel. The formula is relatively straightforward. Here is a link to the math.
Standard deviation may not be the best measure, but it's reasonable for this application and less arcane than the alternatives.Unome wrote:Just dropping this link here, in case anyone finds it useful. I came up with this specifically because I found standard deviation to be unhelpful for SO purposes.
Removing this event only lowered their standard deviation by 8%, so it didn't cross the 85% threshold I used. This is likely because they had many other events where they placed in the 20s and 30s.dxu46 wrote: What about #21 Ladue - 39th Electric Vehicle?
Assistant Coach and Alumnus ('14) - Solon High School Science Olympiad
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
Tournament Director - Northeast Ohio Regional Tournament
Tournament Director - Solon High School Science Olympiad Invitational
Opinions expressed on this site are not official; the only place for official rules changes and FAQs is soinc.org.
Who is online
Users browsing this forum: No registered users and 2 guests