Musings on Test Length

Alumni make some of the best volunteers!
knightmoves
Member
Member
Posts: 319
Joined: April 26th, 2018, 6:40 pm
Has thanked: 2 times
Been thanked: 27 times

Re: Musings on Test Length

Post by knightmoves » December 17th, 2020, 2:05 pm

Unome wrote:
December 17th, 2020, 12:50 pm
The difficulty is that tiebreakers have to effectively tiebreak evenly matched teams across the entire spectrum, from top teams to the very bottom. Just choosing in-depth questions can easily backfire, since often they'll end up being unanswered by the teams toward the bottom of the stack.
I know SO policy is to break ties, but I actually think you don't need to. You can make a case for tiebreaking for medal places (apart from anything else, if you've just got the right number of medals, you can't give out second place to two teams, or you'll run out), but other than that, I think keeping the tie is a more accurate reflection of the performance of the teams involved.

Keep the tie, assign the mean points for the tie (so if there's a two-way tie for 10th place, you assign 10.5 points to each team. If there's a three-way tie for 8th, each team gets 9 points).

But that would need a SO policy change...

User avatar
builderguy135
Exalted Member
Exalted Member
Posts: 723
Joined: September 8th, 2018, 12:24 pm
Division: C
State: NJ
Pronouns: He/Him/His
Has thanked: 138 times
Been thanked: 96 times
Contact:

Re: Musings on Test Length

Post by builderguy135 » December 18th, 2020, 12:31 am

Thoughts on estimation questions used to break ties? If a question was placed near the front of the test and asked all teams to input an answer, you could "curve" the estimate on a scale of, say, 0~0.1 points, which would effectively break ties without impacting the actual score itself significantly. I've implemented this on a few tests I've written so far, and it seems to break ties pretty well. As long as a question is pertinent to the test and easy enough such that everyone is able to understand the question and take a guess, it might be a possible solution for competitions with a lot of teams.

Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.
These users thanked the author builderguy135 for the post:
Unome (December 19th, 2020, 4:16 pm)
West Windsor - Plainsboro High School North '22
PPP/Sounds/Wright/Code/Cyber/DigiStruc
7x National Medalist, 2x MIT SoM Champion
Test Writer and Event Supervisor, BirdSO Co-Director
My Userpage

knightmoves
Member
Member
Posts: 319
Joined: April 26th, 2018, 6:40 pm
Has thanked: 2 times
Been thanked: 27 times

Re: Musings on Test Length

Post by knightmoves » December 18th, 2020, 7:31 am

builderguy135 wrote:
December 18th, 2020, 12:31 am
Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.
This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.
These users thanked the author knightmoves for the post:
Unome (December 19th, 2020, 4:17 pm)

User avatar
SilverBreeze
Exalted Member
Exalted Member
Posts: 298
Joined: November 28th, 2019, 3:42 pm
Division: C
State: CA
Pronouns: She/Her/Hers
Has thanked: 143 times
Been thanked: 258 times

Re: Musings on Test Length

Post by SilverBreeze » December 18th, 2020, 10:29 pm

knightmoves wrote:
December 18th, 2020, 7:31 am
builderguy135 wrote:
December 18th, 2020, 12:31 am
Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.
This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.
Perhaps the issue is, past a certain number of teams, many of those teams will be at almost identical levels. Maybe if you have too many teams, there is no way to break ties meaningfully instead of RNG or amplifying existing noise.
These users thanked the author SilverBreeze for the post (total 3):
Unome (December 19th, 2020, 4:17 pm) • gz839918 (December 21st, 2020, 11:23 am) • BennyTheJett (December 24th, 2020, 8:45 am)
Troy SciOly 2019 - now
Suzanne SciOly 2016 - 2019
Events this season: Water Quality, Forensics, Ornithology, (some experience in DP)

The team may be gone, but the program lives on...

User avatar
jaggie34
Member
Member
Posts: 129
Joined: November 30th, 2018, 10:40 am
Division: Grad
State: GA
Has thanked: 21 times
Been thanked: 77 times

Re: Musings on Test Length

Post by jaggie34 » December 18th, 2020, 10:57 pm

SilverBreeze wrote:
December 18th, 2020, 10:29 pm
knightmoves wrote:
December 18th, 2020, 7:31 am
builderguy135 wrote:
December 18th, 2020, 12:31 am
Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.
This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.
Perhaps the issue is, past a certain number of teams, many of those teams will be at almost identical levels. Maybe if you have too many teams, there is no way to break ties meaningfully instead of RNG or amplifying existing noise.
That's what happened for circuit lab at the rickards invite, lots of ties (even in a 240pt test)
Boca Raton High School -> Georgia Tech
Circuit Lab writer/co-writer: UT Austin, Rickards, Yosemite, Mira Loma, Northern Florida Regionals, Yale, Case Western, Golden Gate, Princeton, Georgia States
Florida Game On C and Fermi Questions C champion!
and Circuit Lab too I guess

User avatar
CPScienceDude
Exalted Member
Exalted Member
Posts: 356
Joined: December 12th, 2018, 2:40 pm
Division: C
State: IN
Pronouns: He/Him/His
Has thanked: 139 times
Been thanked: 80 times

Re: Musings on Test Length

Post by CPScienceDude » December 24th, 2020, 8:36 am

In the one test I've written and supervised, out of 252 points and 12 teams, there were 3 two way ties at 52 points, 58 points, and 59 points. I'll attribute this mostly to my test being, um how do I put this, not the greatest lol, but it also helped me understand that longer tests with harder questions and more points don't work the greatest for separating lower level teams. While longer tests with harder questions definitely help separate the higher tier teams, in my limited experience, it's important to keep the less experienced teams in mind and have a good spread of easier and medium level questions too.
These users thanked the author CPScienceDude for the post (total 2):
Giantpants (December 28th, 2020, 1:19 am) • Mr.Epithelium (December 30th, 2020, 11:12 am)
Captain of CPSO

Assassinator 139 and 147
2021 events: Astronomy, Chem Lab, Ornithology, Machines, Gravity Vehicle, and Forensics

About Me!
Image

ScoutViolet
Member
Member
Posts: 8
Joined: August 9th, 2020, 5:59 pm
Division: B
State: WA
Pronouns: He/Him/His
Has thanked: 2 times
Been thanked: 0

Re: Musings on Test Length

Post by ScoutViolet » January 1st, 2021, 9:18 am

I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted

User avatar
SilverBreeze
Exalted Member
Exalted Member
Posts: 298
Joined: November 28th, 2019, 3:42 pm
Division: C
State: CA
Pronouns: She/Her/Hers
Has thanked: 143 times
Been thanked: 258 times

Re: Musings on Test Length

Post by SilverBreeze » January 1st, 2021, 10:31 am

ScoutViolet wrote:
January 1st, 2021, 9:18 am
I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted
Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)
Troy SciOly 2019 - now
Suzanne SciOly 2016 - 2019
Events this season: Water Quality, Forensics, Ornithology, (some experience in DP)

The team may be gone, but the program lives on...

User avatar
Umaroth
Exalted Member
Exalted Member
Posts: 337
Joined: February 10th, 2018, 8:51 pm
Division: C
State: CA
Pronouns: He/Him/His
Has thanked: 114 times
Been thanked: 229 times

Re: Musings on Test Length

Post by Umaroth » January 1st, 2021, 11:00 am

SilverBreeze wrote:
January 1st, 2021, 10:31 am
ScoutViolet wrote:
January 1st, 2021, 9:18 am
I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted
Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)
I agree, there are two different types of invitationals, those that are meant to mimic regionals and those that are meant to mimic states/nationals, and most strong teams attend both and expect good tests. Regionals is meant to be the lowest level of competition, it wouldn't make sense for invitationals to be lower than them when for many teams the entire point of going to invitationals is to practice for nationals. In a normal year there are a sufficient amount of smaller scale invitationals that powerhouse teams don't attend that satisfy the needs of the local teams prepping for regionals and hopefully states, but this year a lot of those tournaments have vanished and all that's left are the tournaments who cater to the national audience.
These users thanked the author Umaroth for the post (total 2):
SilverBreeze (January 1st, 2021, 12:52 pm) • Unome (January 4th, 2021, 6:22 am)
Troy SciOly 2021 Co-Captain
Proud Padre of the Evola SciOly Program 2018-now
Dank Memes Area Homeschool Juggernaut 2018-now
Sierra Vista SciOly Co-Head Coach 2020-now
2021 Tryouts: Circuit, Code, Detector, DP, GeoMaps, Machines

The team may be gone, but the program lives on...
Umaroth's Userpage

ScoutViolet
Member
Member
Posts: 8
Joined: August 9th, 2020, 5:59 pm
Division: B
State: WA
Pronouns: He/Him/His
Has thanked: 2 times
Been thanked: 0

Re: Musings on Test Length

Post by ScoutViolet » January 2nd, 2021, 7:57 pm

Umaroth wrote:
January 1st, 2021, 11:00 am
SilverBreeze wrote:
January 1st, 2021, 10:31 am
ScoutViolet wrote:
January 1st, 2021, 9:18 am
I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted
Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)
I agree, there are two different types of invitationals, those that are meant to mimic regionals and those that are meant to mimic states/nationals, and most strong teams attend both and expect good tests. Regionals is meant to be the lowest level of competition, it wouldn't make sense for invitationals to be lower than them when for many teams the entire point of going to invitationals is to practice for nationals. In a normal year there are a sufficient amount of smaller scale invitationals that powerhouse teams don't attend that satisfy the needs of the local teams prepping for regionals and hopefully states, but this year a lot of those tournaments have vanished and all that's left are the tournaments who cater to the national audience.
Yes, I do agree with that. However, me being a 6th grader and all, I had practice tests with regular probably state/regionals level. So, I went to Rickards, and boom... 8th graders at my school say it was like nationals. Me: That would explain. But I think that for some people who got like 1 place below me, 23.50 for example on machines out of 286 may be pretty discouraging for them. This could cause them to quit science olympiad and/or get really sad (hopefully none actually happened). I think you shouldn't start of like easy easy, but maybe for the time being at like state levels for invitationals. My regionals is like in February, so there would be plenty of time to go to National Level Invitationals.

Post Reply

Return to “Alumni”

Who is online

Users browsing this forum: No registered users and 0 guests