Test Set Exchange Update

[Schrodingerscat]

I'm going to conclude with my final thoughts that SOINC should step in and require all invitationals register with SOINC and release tests and keys promptly to centralize these resources. The recent addition of "Invitational Ruleset" has been a foray into this but I hope it becomes more useful later. I applaud Princeton for being the first high-level invitational to publicly release tests and set off the next wave of invitationals to publically release tests (MIT, SOUP, etc.). Kudos to the aforementioned invitationals. Please make all tests public. It would save teams countless hours not worrying about tests to get and get rid of this issue once and for all.

Personally from a national policy perspective I'd rather see a "right to redistribute" rather than an "obligation to distribute" even if the latter would result in more tests available. In effect to say that if a sanctioned tournament decides to release tests to teams, any single recipient of the tests has the right to upload the tests to a site like this, and that sites like us could return to the old days of not worrying so much about permission to host invite tests (at least to my knowledge then). I think it might be a step too far to require tournament directors do anything specific to distribute tests though.

[Raleway]

@Unome: Unfortunately, the consequence of NSO not being involved in this and soliciting advice and/or adding people to the steering committee with adequate knowledge is exactly the status quo. And it's not pretty. Sure, the first year or even subsequent years could be tough as NSO and invitationals work to get a balance, but in the end, it will be much more beneficial. There is no other oversight anything that has the power besides NSO. The problem with NSO being distant is also a larger issue in of itself. It just means they are not staying current, which is a dire problem that needs to be solved as soon as possible.

@Schrodingerscat: The issue with the right to redistribute is that what is currently happening occurs. Like in game theory, there is a much stronger incentive to not allow sharing of tests. Obviously, the incentives and intangibles at play have caused the current status quo; an external force will be required to create a paradigm shift (namely, NSO stepping in). I had a hope of Princeton, MIT, and SOUP being leading examples to other invitationals to publicly release tests, but it seems it is not the case (Cornell this year and whichever other invitationals).

It's unfortunate that things are like this, but I know teams that have dedicated test traders and individuals spending countless hours trying to get tests and make deals. That shouldn't be a thing in Science Olympiad.

[Unome]

@Unome: Unfortunately, the consequence of NSO not being involved in this and soliciting advice and/or adding people to the steering committee with adequate knowledge is exactly the status quo. And it's not pretty. Sure, the first year or even subsequent years could be tough as NSO and invitationals work to get a balance, but in the end, it will be much more beneficial. There is no other oversight anything that has the power besides NSO. The problem with NSO being distant is also a larger issue in of itself. It just means they are not staying current, which is a dire problem that needs to be solved as soon as possible.

@Schrodingerscat: The issue with the right to redistribute is that what is currently happening occurs. Like in game theory, there is a much stronger incentive to not allow sharing of tests. Obviously, the incentives and intangibles at play have caused the current status quo; an external force will be required to create a paradigm shift (namely, NSO stepping in). I had a hope of Princeton, MIT, and SOUP being leading examples to other invitationals to publicly release tests, but it seems it is not the case (Cornell this year and whichever other invitationals).

It's unfortunate that things are like this, but I know teams that have dedicated test traders and individuals spending countless hours trying to get tests and make deals. That shouldn't be a thing in Science Olympiad.

Consider how many invitationals do you think would lose significant numbers of teams and/or just stop running altogether as a result of something like what you propose.

[drcubbin]

I could be wrong, but I thought (or maybe it's just old age setting in) that when we began almost 5 years ago, someone told us that there are certain test-writers who were adamantly opposed to releasing their tests and that you never saw their tests. And I vaguely remember never seeing those tests. But I suppose, like most teams, we do have somewhat of a small "repository" of tests that we use as a means to see the direction an event will take during competition. That being said, I believe the overall "pros" of test trading (getting a glimpse into what to study) are far overshadowed by the "cons", such as a) students taking the same test from last year (since test-writers have the same access to past tests), b) students using valuable study time to accumulate past tests (which in spite of this practice, currently appears to be a necessary evil) or at the very least c) getting into a rut of studying to the test (which I think we can all agree is not good). The bottom line is... there are a mountain of past tests currently in circulation. The question is, "What do we do now?", because as test-writers and event supervisors - and educated adults - I think we can all agree something needs to be done... and rather quickly, either yea or nay.

Most people do not know, but I had a previous - very successful and highly lucrative - former life as a chiropractor treating more than 300 patients/week (that's a LOT!) for almost 20 years. And if it weren't for my desire to teach, I might still be there, but with a crippling "SciOly void" in my life. I graduated in 86' (pre-computer lunacy) and had to study for everything I learned. While you might have seen some SciOly-level test questions (particularly A&P or Thermo) on our National and State boards, those examinations were infinitely more difficult. The teachers taught, we took notes, read books and took tests. No "previous tests" were ever available (though we did hear urban legends that ONE of Dr. Bogin's Physiology finals were floating around somewhere, but I never saw it). There were no "SAT prep" classes using previous tests either. We studied, learned a boatload of information and were set loose into the world. Trying to think back, I cannot even imagine what it would have been like if we had access to prior exams. My guess is that many - not all - would have been studying to the tests, and studying the information less. Like where we are now, it is just the nature of the beast. This is why I am in favor of halting the release of all future exams - a total moratorium on releasing any and all tests. Here's why.

There is currently a massive amount of "competitive direction and support" being given through a variety of venues for students to amply prepare for the rigors of competition: SciOly Wiki, scioly.org forums, etc... In fact, I would guess that many competitors have read old tests more completely than they have read the Rules manual for their events or the Wiki site (and this is often how they end up with construction violations). I believe these resources could be further developed so competitors would have even more resources at their disposal - but no more tests.

For those students complaining that event supervisors often include questions that are "way out there", they are right. We often do stretch the limits of test writing because so many of the reasonable and excellent questions have been rehashed over and over again and then given to students - through test-trading - to memorize. Imagine if there were no "previous" tests? You would never again hear a competitor say, "We took 3rd Place because it was the same test from two years ago". Is this really what we want? To the test-writers who write multiple tests for the same event, imagine being able to write one kick-butt test and simply modify it between events, instead of having to come up with unique and deal-breaker questions that even the best students are often only guessing at? On that note, thank you Alice Kasten for never releasing any NYC Regional exams. And while I am sure the NYC test-writers do write new tests every year, I am sure it would be nice to "rehash" test great questions or labs that students should be expected to know and not just because they have seen them on a dozen previous test versions. What test writer would love to use certain questions again - because they were well-developed and caused the competitors to think - but did not because the kids already had them in their hands?

It seems to me we are at a crossroads. Do we create a widespread underground/open-market for past exams for students to value over the actual information they need to know? (*Note - I do believe the schools that are crushing it at competitions really do know the information, are studying diligently and are very well prepared, but some in the middle range may be weighing the value of tests too heavily). Or do we stop the practice of test-trading here and now. Create an "across the board" end to releasing tests (leaving those tests that are out there to do what they may) and possibly create a more challenging approach to competitive events that the competitors may actually like. Having spent this past summer with so many great National Supervisors at the SOSI, I am guessing most would be in favor of the latter.

Disclaimer - the opinions of this contributor are sure to raise the ire of some, but he does not really care. He loves SciOly too much and believes that if the US is to ever claw its way back to the top of international strength in science SciOly is the only viable means to that end.

[lumosityfan]

I could be wrong, but I thought (or maybe it's just old age setting in) that when we began almost 5 years ago, someone told us that there are certain test-writers who were adamantly opposed to releasing their tests and that you never saw their tests. And I vaguely remember never seeing those tests. But I suppose, like most teams, we do have somewhat of a small "repository" of tests that we use as a means to see the direction an event will take during competition. That being said, I believe the overall "pros" of test trading (getting a glimpse into what to study) are far overshadowed by the "cons", such as a) students taking the same test from last year (since test-writers have the same access to past tests), b) students using valuable study time to accumulate past tests (which in spite of this practice, currently appears to be a necessary evil) or at the very least c) getting into a rut of studying to the test (which I think we can all agree is not good). The bottom line is... there are a mountain of past tests currently in circulation. The question is, "What do we do now?", because as test-writers and event supervisors - and educated adults - I think we can all agree something needs to be done... and rather quickly, either yea or nay.

Most people do not know, but I had a previous - very successful and highly lucrative - former life as a chiropractor treating more than 300 patients/week (that's a LOT!) for almost 20 years. And if it weren't for my desire to teach, I might still be there, but with a crippling "SciOly void" in my life. I graduated in 86' (pre-computer lunacy) and had to study for everything I learned. While you might have seen some SciOly-level test questions (particularly A&P or Thermo) on our National and State boards, those examinations were infinitely more difficult. The teachers taught, we took notes, read books and took tests. No "previous tests" were ever available (though we did hear urban legends that ONE of Dr. Bogin's Physiology finals were floating around somewhere, but I never saw it). There were no "SAT prep" classes using previous tests either. We studied, learned a boatload of information and were set loose into the world. Trying to think back, I cannot even imagine what it would have been like if we had access to prior exams. My guess is that many - not all - would have been studying to the tests, and studying the information less. Like where we are now, it is just the nature of the beast. This is why I am in favor of halting the release of all future exams - a total moratorium on releasing any and all tests. Here's why.

There is currently a massive amount of "competitive direction and support" being given through a variety of venues for students to amply prepare for the rigors of competition: SciOly Wiki, scioly.org forums, etc... In fact, I would guess that many competitors have read old tests more completely than they have read the Rules manual for their events or the Wiki site (and this is often how they end up with construction violations). I believe these resources could be further developed so competitors would have even more resources at their disposal - but no more tests.

For those students complaining that event supervisors often include questions that are "way out there", they are right. We often do stretch the limits of test writing because so many of the reasonable and excellent questions have been rehashed over and over again and then given to students - through test-trading - to memorize. Imagine if there were no "previous" tests? You would never again hear a competitor say, "We took 3rd Place because it was the same test from two years ago". Is this really what we want? To the test-writers who write multiple tests for the same event, imagine being able to write one kick-butt test and simply modify it between events, instead of having to come up with unique and deal-breaker questions that even the best students are often only guessing at? On that note, thank you Alice Kasten for never releasing any NYC Regional exams. And while I am sure the NYC test-writers do write new tests every year, I am sure it would be nice to "rehash" test great questions or labs that students should be expected to know and not just because they have seen them on a dozen previous test versions. What test writer would love to use certain questions again - because they were well-developed and caused the competitors to think - but did not because the kids already had them in their hands?

It seems to me we are at a crossroads. Do we create a widespread underground/open-market for past exams for students to value over the actual information they need to know? (*Note - I do believe the schools that are crushing it at competitions really do know the information, are studying diligently and are very well prepared, but some in the middle range may be weighing the value of tests too heavily). Or do we stop the practice of test-trading here and now. Create an "across the board" end to releasing tests (leaving those tests that are out there to do what they may) and possibly create a more challenging approach to competitive events that the competitors may actually like. Having spent this past summer with so many great National Supervisors at the SOSI, I am guessing most would be in favor of the latter.

Disclaimer - the opinions of this contributor are sure to raise the ire of some, but he does not really care. He loves SciOly too much and believes that if the US is to ever claw its way back to the top of international strength in science SciOly is the only viable means to that end.

While in theory having no access to any prior resources would be nice for SciOly, the problem is that people aren't perfect. Competitors will naturally ask supervisors for old tests and supervisors will inevitably feel compelled to give the tests to them, despite their best efforts. Also the reason why competitions like USABO, the psychology tests, and others are not released is because they're propriety. SciOly tests aren't propriety, meaning there's nothing inherently wrong with releasing tests. Also if we are to allow event supervisors to repeat tests, that gives so much more advantages to competitors who have actually competed in previous year's event. That prevents new teams from having an even keel when it comes to understanding event supervisors' tendencies (and tests should not be taken based on knowing event supervisors' tendencies - why I feel irked by the fact that those habits make a difference). Also it provides no new accountability as there would be no way of figuring out if an event supervisor reuses a test.

[lumosityfan]

@Unome: Unfortunately, the consequence of NSO not being involved in this and soliciting advice and/or adding people to the steering committee with adequate knowledge is exactly the status quo. And it's not pretty. Sure, the first year or even subsequent years could be tough as NSO and invitationals work to get a balance, but in the end, it will be much more beneficial. There is no other oversight anything that has the power besides NSO. The problem with NSO being distant is also a larger issue in of itself. It just means they are not staying current, which is a dire problem that needs to be solved as soon as possible.

@Schrodingerscat: The issue with the right to redistribute is that what is currently happening occurs. Like in game theory, there is a much stronger incentive to not allow sharing of tests. Obviously, the incentives and intangibles at play have caused the current status quo; an external force will be required to create a paradigm shift (namely, NSO stepping in). I had a hope of Princeton, MIT, and SOUP being leading examples to other invitationals to publicly release tests, but it seems it is not the case (Cornell this year and whichever other invitationals).

It's unfortunate that things are like this, but I know teams that have dedicated test traders and individuals spending countless hours trying to get tests and make deals. That shouldn't be a thing in Science Olympiad.

Consider how many invitationals do you think would lose significant numbers of teams and/or just stop running altogether as a result of something like what you propose.

The thing is though competitions like Princeton, MIT, and UT-Austin do it and as far as I know, their attendance has not suffered. (Indeed Princeton's attendance has gone up and UT-Austin and MIT have stayed in-demand as ever.)

[syo_astro]

The thing is though competitions like Princeton, MIT, and UT-Austin do it and as far as I know, their attendance has not suffered. (Indeed Princeton's attendance has gone up and UT-Austin and MIT have stayed in-demand as ever.)

And UMSO;P. And Golden Gate I thought? And Dartmouth now! Anyway, Lumo, that wasn't the point of what Unome was saying. Can you name non-university invites that stayed in demand and compare it with the other mass amount of invites? I think cat has the most reasonable approach in line with how scioly usually runs things from the ground up anyway. If tests are even just allowed to be more openly shared, maybe we'll realize how this all shouldn't be such a big deal...

To drcubbin: Seems like it would be interesting discussing with you! (Nobody answered my question on the last page *why* people expect tests to "not be confidential" and are okay with test trading with the context and questions that you can see last page...would like it if someone could...). My opinion summarized in response to your issues: If we are indeed a science competition, then our questions should have plenty of concepts, math, etc. These questions SHOULD be constantly modified and also emphasize applying the basics. The events all change in different ways to reflect this, which is one thing I love! In fact, I think this is the best thing, science questions CAN be modified, which means you have to connect ideas from studies to do well. Some factual questions don't need to be, that's not the point of those. I know I don't have infinite experience, but I also find my past tests need significant modifying anyway based on answers I get (I sometimes even use invites to test question ideas I have...). Therefore, the issues of test content and test release can be separated a little more.

Now para by para (not quoting that all...you also seem to have a lot of different issues here that I try to separate):
1) You also have a repository of tests...so having some tests is a necessary evil? Just something to keep in mind. In response to your cons...point (a)...test writers reusing tests sort of goes against my opinion. I know it takes more time, but I think the better tests are usually rewritten anyway. Is your issue that we don't have enough writers? Why not take from someone who is writing in another state, which I imagine has to be done anyway? As for students taking study time to accumulate tests (b), yup fully agree, a waste of time, but I don't think we can totally control that. The internet exists, they'll figure out a way to make "test banks". Is it worse to be more open with people (students, coaches), not release essential or regionals test (fair enough), and only release practice tests (invites)? Your point (c) I also agree with. But the mountain of tests aren't all very good...it sounds like most I've talked to respect that 60 tests are a waste of time anyway.

2) Useful to know where you're coming from. Basically: What Lumo said. From my perspective, you're comparing different tests in different contexts. The internet exists, and the onus is on us to make use of it, not to say "just ignore it". I'm not saying the way you're talking about learning is "wrong". But the teachers I learned best from made their own questions that connected ideas you just can't google and don't really want to anyway. Your last few sentences I'd agree with. From here you assert whys!

On posted resources and reading the rules:
3) Perhaps competitors read tests more than the rules manual. Is that the fault of the tests, though? You said there's already so many online that we can't do anything about...won't they just keep paying more attention to those then? I know this isn't your point, but should the question marathons go too since they're easy to reuse questions? If you're saying we should update the site more, why not give better sources for practice with tests too? Practice tests (though I scorn tests) do have value for learning. If you have issue with students not reading the rules, then isn't that a more complicated and separate issue? For example, some may not read the rules because of last minute preparations as opposed to pure ignorance. Posting tests or not that can happen anyway, no?

On types of questions and question reuse:
4) This depends on the type of "out there" question. There are those "out there" questions that aren't justified by the rules. Other times students rightly dislike getting tested on literally everything (especially trivia), but people are more okay with that since it's in the rules. On the other hand, those questions may do us writers no justice: it makes it seem like the event should focus on memorizing or cramming as much into your notes as possible. A few of those questions can be fun or give a test quirks, though, so I don't see this as a huge deal except for writers that do it way too much.

Here's where I really get confused. "So many of the reasonable and excellent questions have been rehashed over and over again through test trading"...huh? Is this the main reason? All my tests are posted online, but nobody seems to do well on them:P. Maybe this is more specific to certain events than a universal issue? This just goes back to my prior opinion that updating questions or writing new questions can be (and certainly is!) a good thing. If a team can easily get 3rd place because it was the same test as two years ago, how can you fault the students? Wouldn't this happen anyway? Of many examples, one I like: they could have written the questions down on paper just after the test. I know this because I used to do that...more because a few questions bothered the HECK out of me. Even ignoring the internet, there's plenty of ways to circumvent regionals not getting posted.

Also, I don't know how much you write tests, but I need to write multiple tests anyway because rules constantly change and invite, regional, and state exams should NOT look the same. That has nothing to do with my tests being posted. I need to adjust the difficulties and question types anyway...if it's a question so trivial it could be answered with notes, isn't that my fault for giving a literally trivial question? There is a place for having some of those, and I think test writers (or at least I) reuse some of those questions anyway (yet students still mess them up...-_-).

5) This already exists. We can't undo that. If we give a system that can be gamed, students (people in general) WILL game it difficult or not. I can come up with a long list of ways to game testing if you like, but I hope you get the point. I also don't get what you mean by that last part: aren't making tests (or rules) more difficult and releasing tests not mutually exclusive?

Disclaimer - the opinions of this contributor are sure to raise the ire of some, but he does not really care. He loves SciOly too much and believes that if the US is to ever claw its way back to the top of international strength in science SciOly is the only viable means to that end.

Disclaimer: I'm not mad, thanks for sharing...know that others love scioly too;). I also feel like there are fundamental problems with education that need to be addressed beyond just scioly, but that's a story for another day and not this thread...

[drcubbin]

I appreciate the response syo_astro, but please understand that not being a child of the digital age, I find corresponding over the internet always loses something in translation, which it obviously did, and which is why I hesitated posting my thoughts in the first place as I had a feeling they would be "examined" under a microscope. I am sure if we were to sit and chat face to face, we would get much more accomplished. I paint with a broad brush, and though verbose, my intentions were not complicated, so allow me a little clarifying:

a) I believe we all love SciOly. That is why we are here. I never meant otherwise.

b) When I first learned about old tests being used for studying purposes, my first instinct was that it was somehow wrong. I still feel that way. Yes, we (our teams) do have tests. Yes, we use them to prepare. Call me a hypocrite, but no one is perfect. But I'm sure most would agree, our top students can live without these tests, so long as they are provided a little more information on what can, should and will be asked of them in Invitationals, Regionals and Nationals (I am just guessing about the Nationals part). Many students are not using tests to find out the basics. They can find this anywhere. They are looking to see how far astray the upcoming test will go. Take Potions and Poisons. I have no problem holding teams accountable for knowing molarity or even intermediate dilution calculations. But when the manual calls for no more than "balancing simple chemical equations", and the invy test asks for detailed understanding understanding of molarity... well then, yes, having that test to show them that it might come up sometime in the future is a definite asset - in spite of what limits the manual states.

c) I completely agree. Tests should always be modified. I never said we should make a test and use it over and over. I think if the test parameters could be expanded - and explained in the manual - to include higher level chemistry (for example with P&P), it would negate the need for reviewing tests, and require more learning take place. It would also make writing the test easier. Just one simple man's opinion, so please do not dissect this idea.

d) Yes, there are lots and lots of online resources. And it never ceases to amaze me how difficult it is for "children of the computer age" to find all of them, but easy to want to review another test. I like question marathons and I love the Summer Test exchange. Students making student tests for review is an absolute winner.

e) I apologize for the confusion behind "So many of the reasonable and excellent questions have been rehashed over and over again through test trading". What I am saying is that students are not using tests to study the basics, but to find the exception, rather than the rule. I think this is best illustrated by Section 7, page 7, question #52 of the 2019 Cornell DD test. Tell me the number of students who answered this correctly and I will admit I am wrong. In hindsight, I can see what they were looking for, but after speaking with other schools, many lost all points on this. There is so much more relevant information on microbiology and public health that can be asked. Again, having studied this for years, it is just my opinion.

f) You write, "I don't know how much you write tests, but I need to write multiple tests anyway because rules constantly change and invite, regional, and state exams should NOT look the same." Come on. I know that, but there is so much to ask in so many areas that when the competitors are given such limited parameters, the result is sure to make them look anywhere to find what will be asked that is not being told they will be asked.

g) "If a team can easily get 3rd place because it was the same test as two years ago, how can you fault the students?" I am not faulting the students. They did not write the test, but I thought my point here was obvious.

I am just sending this off in a flurry, so if I have not responded to all points, so be it. It is, after all, just a forum for ideas. And I will leave with a line from luminosityfan, "While in theory having no access to any prior resources would be nice for SciOly, the problem is that people aren't perfect". Yes, it would be nice.

[syo_astro]

...I had a feeling they would be "examined" under a microscope. I am sure if we were to sit and chat face to face, we would get much more accomplished. I paint with a broad brush, and though verbose, my intentions were not complicated, so allow me a little clarifying:
...a) I believe we all love SciOly. That is why we are here. I never meant otherwise.
...I am just sending this off in a flurry, so if I have not responded to all points, so be it. It is, after all, just a forum for ideas.

Probably not worth extending that discussion then, but just wanted to say thanks for the thoughts and clarifications (really, I mean that). Sorry, that was a lot, I'm just trying to get at what people mean exactly, which I agree confuses me too on the internet (why I try to toss in lots of questions and examples). @a) If I said or implied something that indicated someone didn't love scioly, really didn't mean to...I'm aware we're on a forum dedicated to this...

[Unome]

I appreciate the response syo_astro, but please understand that not being a child of the digital age, I find corresponding over the internet always loses something in translation, which it obviously did, and which is why I hesitated posting my thoughts in the first place as I had a feeling they would be "examined" under a microscope. I am sure if we were to sit and chat face to face, we would get much more accomplished. I paint with a broad brush, and though verbose, my intentions were not complicated, so allow me a little clarifying:

a) I believe we all love SciOly. That is why we are here. I never meant otherwise.

b) When I first learned about old tests being used for studying purposes, my first instinct was that it was somehow wrong. I still feel that way. Yes, we (our teams) do have tests. Yes, we use them to prepare. Call me a hypocrite, but no one is perfect. But I'm sure most would agree, our top students can live without these tests, so long as they are provided a little more information on what can, should and will be asked of them in Invitationals, Regionals and Nationals (I am just guessing about the Nationals part). Many students are not using tests to find out the basics. They can find this anywhere. They are looking to see how far astray the upcoming test will go. Take Potions and Poisons. I have no problem holding teams accountable for knowing molarity or even intermediate dilution calculations. But when the manual calls for no more than "balancing simple chemical equations", and the invy test asks for detailed understanding understanding of molarity... well then, yes, having that test to show them that it might come up sometime in the future is a definite asset - in spite of what limits the manual states.

c) I completely agree. Tests should always be modified. I never said we should make a test and use it over and over. I think if the test parameters could be expanded - and explained in the manual - to include higher level chemistry (for example with P&P), it would negate the need for reviewing tests, and require more learning take place. It would also make writing the test easier. Just one simple man's opinion, so please do not dissect this idea.

d) Yes, there are lots and lots of online resources. And it never ceases to amaze me how difficult it is for "children of the computer age" to find all of them, but easy to want to review another test. I like question marathons and I love the Summer Test exchange. Students making student tests for review is an absolute winner.

e) I apologize for the confusion behind "So many of the reasonable and excellent questions have been rehashed over and over again through test trading". What I am saying is that students are not using tests to study the basics, but to find the exception, rather than the rule. I think this is best illustrated by Section 7, page 7, question #52 of the 2019 Cornell DD test. Tell me the number of students who answered this correctly and I will admit I am wrong. In hindsight, I can see what they were looking for, but after speaking with other schools, many lost all points on this. There is so much more relevant information on microbiology and public health that can be asked. Again, having studied this for years, it is just my opinion.

f) You write, "I don't know how much you write tests, but I need to write multiple tests anyway because rules constantly change and invite, regional, and state exams should NOT look the same." Come on. I know that, but there is so much to ask in so many areas that when the competitors are given such limited parameters, the result is sure to make them look anywhere to find what will be asked that is not being told they will be asked.

g) "If a team can easily get 3rd place because it was the same test as two years ago, how can you fault the students?" I am not faulting the students. They did not write the test, but I thought my point here was obvious.

I am just sending this off in a flurry, so if I have not responded to all points, so be it. It is, after all, just a forum for ideas. And I will leave with a line from luminosityfan, "While in theory having no access to any prior resources would be nice for SciOly, the problem is that people aren't perfect". Yes, it would be nice.

b and e) Interesting point on the main reason for tes-gathering being to identify areas not explicitly outlined in the rules but frequently covered on tests. I rather prefer that competitors have access to past tests though, because otherwise this just means that competitors with more competition experience have a vastly better understanding of what is likely to be asked, which is not currently the case to such a large degree (or at least, does not have to be the case for those willing to put in some effort).

c) I think most of us agree on this point. Some of the events, Div B in particular, are woefully restricted to the point where it's very difficult to make new questions (Density Lab comes to mind).