Balancing Difficulty and Accessibility in Test Writing

[Unome]

late response, but I'm a division C participant, and our school made us write tests for division B students. I was in charge of a Dynamic Planet Glaciers test, and I tried to base my test on some tests I've seen. My test was around 40 mc and 15 frq questions with multiple parts. The thing is, I based it off of Division C tests since Division B and C dynamic have the same content. I personally am not the greatest at Dynamic, so I expected my test to be somewhat easy, but the average score was around 25%, and the highest score was Longfellow at 50%. Was this a good idea? Or did I screw up my difficulty? I could share the test as resources, but I don't know if it was worth taking tbh.

50% is a little low, but I would say still a reasonable top score, provided you had enough room to distinguish teams without a bunch of ties (i.e. more than about 1-2 ties for every dozen or so teams is probably too many). That said, I tend to aim for a 60% top score when I write tests.

[pikachu4919]

Most of my test writing experience is in forensics, in which the event in general in itself by design is quite difficult to write for, seeing as you need to write not only a test, but also a story and characters to accompany the test, and all the parts must be well integrated with each other.

That being said, my philosophy for forensics is that even though my tests (even with other alumni collaborating with me) definitely push the limit on length and difficulty, per the rules, the lab ID portions and the crime analysis are the bread and butter of any forensics exam. As such, they make up the bulk of the points on my tests and are considered to, more or less, be my form of “easy” points - if you can’t do either of those portions on a forensics test, then ...well, I’m not sure what to tell ya. Then all the supplementals thrown in to test the limits of your knowledge of forensics or properties of the ID materials beyond the crime scene of the test itself are literally and figuratively like the harder questions of normal tests. As such, those that can finish the lab ID and crime analysis will be the ones that get the fundamental stuff and can then earn additional points for being able to answer to the more think-out-of-the-box content.

That’s more of my writing philosophy on a specific event, but it does show thought processes on spread of difficulty/topics/etc and also how each event’s definition of such can vary.

Many of the other tests I've done, I've written in stations because they're better at testing who really knows their stuff and who just relies on their notes. I always do a bunch and then I have the stations range in difficulty from really easy to really hard. For example, I recently wrote a Herpetology test, and I had a couple stations where the specimen was really easy, which was my form of easy questions, or a station that was entirely ID, as it is a fundamental skill to Herpetology. And then there were either entire stations that were difficult or just difficult questions at other stations. Honestly, point is, stations are also not a bad way to easily see how spread the difficulty is, since you can set a certain number of your stations to be easy, medium, or hard. The drawback is that the stations format doesn't work for all the events.

[Alex-RCHS]

In terms of difficulty, my personal philosophy is simple: Have a broad range of questions, the easiest one being "everyone will get this" and the hardest one being "nobody will get this", with an even distribution in between. In theory, this results in a mean score of 50%, which is pretty close to ideal IMO.

As for content, I like to emphasize understanding more than memorization. However, some test writers make the mistake (IMO) of making questions that are meant to test your understanding actually test your interpretation (the difference between the two being that your understanding reflects the concepts you've learned specific to the event, and interpretation abilities just reflects your general ability to read graphs, analyze data, etc., regardless of your preparation for the event).

I agree with most people that keeping questions of the same topic together is best for competitors, and organizing it into "case studies" is a great way to do this. Give some information (data, graphs, paragraphs, etc.) and then ask questions relating to the data. Calculations, interpretations, predictions, etc. are all good types of questions to ask. It also makes the test feel more "real" and interesting than just a stream of random, unrelated questions.

I personally prefer a stations format, as it more thoroughly constrains the time teams are allowed for each section of the test. This way, I can emphasize a more thorough and even understanding of the material, rather than knowing a lot about one topic and only a little about another.

[primitivepolonium]

As for content, I like to emphasize understanding more than memorization. However, some test writers make the mistake (IMO) of making questions that are meant to test your understanding actually test your interpretation (the difference between the two being that your understanding reflects the concepts you've learned specific to the event, and interpretation abilities just reflects your general ability to read graphs, analyze data, etc., regardless of your preparation for the event).

I agree with most people that keeping questions of the same topic together is best for competitors, and organizing it into "case studies" is a great way to do this. Give some information (data, graphs, paragraphs, etc.) and then ask questions relating to the data. Calculations, interpretations, predictions, etc. are all good types of questions to ask. It also makes the test feel more "real" and interesting than just a stream of random, unrelated questions.

Agreed on the interpretation part; kinda like how the Science section on the ACT tests your ability to read stuff quickly rather than you know, science. Like they're important skills but they're no substitute for knowing the event. Another mistake that I've seen is when "baiting", or when a test writer is looking for a specific answer with the intention of testing "understanding" or "problem-solving", so the question is phrased in such a way to try and push the competitor towards that specific explanation. Sometimes it works, but often it results in weirdly phrased or ambiguous stuff. If the kid happens to not read it the intended way, they'll get it wrong even if they really understand the concepts behind the question. I definitely find myself guilty of doing this sometimes, but any ambiguous "leading questions" should be fixed before the final draft.

As for keeping questions of a certain topic together--it doesn't work as well for some events. For Chem Lab, it makes sense for there to be some concept mingling on FRQ questions because chemistry is often interdisciplinary. For instance, if you're analyzing its substance, both its acidity and its physical properties are of great importance. I always aim to have a cohesive "theme" or "case" for each set of questions though, because as you said, it feels a lot more "real".