Musings on Test Length

[knightmoves]

The difficulty is that tiebreakers have to effectively tiebreak evenly matched teams across the entire spectrum, from top teams to the very bottom. Just choosing in-depth questions can easily backfire, since often they'll end up being unanswered by the teams toward the bottom of the stack.

I know SO policy is to break ties, but I actually think you don't need to. You can make a case for tiebreaking for medal places (apart from anything else, if you've just got the right number of medals, you can't give out second place to two teams, or you'll run out), but other than that, I think keeping the tie is a more accurate reflection of the performance of the teams involved.

Keep the tie, assign the mean points for the tie (so if there's a two-way tie for 10th place, you assign 10.5 points to each team. If there's a three-way tie for 8th, each team gets 9 points).

But that would need a SO policy change...

[builderguy135]

Thoughts on estimation questions used to break ties? If a question was placed near the front of the test and asked all teams to input an answer, you could "curve" the estimate on a scale of, say, 0~0.1 points, which would effectively break ties without impacting the actual score itself significantly. I've implemented this on a few tests I've written so far, and it seems to break ties pretty well. As long as a question is pertinent to the test and easy enough such that everyone is able to understand the question and take a guess, it might be a possible solution for competitions with a lot of teams.

Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.

[knightmoves]

Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.

This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.

[SilverBreeze]

Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.

This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.

Perhaps the issue is, past a certain number of teams, many of those teams will be at almost identical levels. Maybe if you have too many teams, there is no way to break ties meaningfully instead of RNG or amplifying existing noise.

[jaggie34]

Inflating point values may also work. Although the test length would be the same, it would be out of more points, which would spread out the teams more in terms of raw point range.

This only really helps if you award partial credit, or if you have more points available for more difficult questions - otherwise you have the same number of discrete total scores available. And awarding more points for harder questions doesn't get you as much extra spread as you think, because everyone tends to find the same questions easy and the same ones hard.

Perhaps the issue is, past a certain number of teams, many of those teams will be at almost identical levels. Maybe if you have too many teams, there is no way to break ties meaningfully instead of RNG or amplifying existing noise.

That's what happened for circuit lab at the rickards invite, lots of ties (even in a 240pt test)

[Giantpants]

In the one test I've written and supervised, out of 252 points and 12 teams, there were 3 two way ties at 52 points, 58 points, and 59 points. I'll attribute this mostly to my test being, um how do I put this, not the greatest lol, but it also helped me understand that longer tests with harder questions and more points don't work the greatest for separating lower level teams. While longer tests with harder questions definitely help separate the higher tier teams, in my limited experience, it's important to keep the less experienced teams in mind and have a good spread of easier and medium level questions too.

[ScoutViolet]

I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted

[SilverBreeze]

I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted

Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)

[Umaroth]

I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted

Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)

I agree, there are two different types of invitationals, those that are meant to mimic regionals and those that are meant to mimic states/nationals, and most strong teams attend both and expect good tests. Regionals is meant to be the lowest level of competition, it wouldn't make sense for invitationals to be lower than them when for many teams the entire point of going to invitationals is to practice for nationals. In a normal year there are a sufficient amount of smaller scale invitationals that powerhouse teams don't attend that satisfy the needs of the local teams prepping for regionals and hopefully states, but this year a lot of those tournaments have vanished and all that's left are the tournaments who cater to the national audience.

[ScoutViolet]

I think that I personally prefer shorter tests for invitationals, and gradually increase questions for higher levels. For example, I did one SO a couple of weeks ago (Rickards Invitational). I took one look at it and I was like... but the practice tests were like half this long!!! Then my parents got really happy because we're 6th graders and I got like 14th/45 teams in both Road Scholar and Water Quality and 15th/45 teams in Machines. I, of course, being a realllly eager student, wanted to see my exact scores.
Here they are: (prepare for cringe)
Machines: 24.50 out of 286.00
Road Scholar: 23.00 out of 229.00
Water Quality: 36.00 out of 160.00
I mean, really, we got 15th, 14th and 14th respectively out of 45 teams. I think the tests should be a lot shorter, as we had an average of like 79 questions... And we got busted

Hmmmm this is interesting. As someone who likes insanely long tests, I've always thought that, from a competitor standpoint, a test that's just long enough is the same as an impossibly long test, simply because you're going to do the same number of questions either way. The only difference is, once someone writes past what any person could finish in 50 minutes, the test writer is wasting their own time.

Is it that a really long test is discouraging to people who feel expected to finish? (especially if they carry over the same expectations as a test for, say, science class)

A lot of more difficult invitationals - generally ones who can get alumni test writers who write for the fun of it - are actually expected to be more difficult than regionals, although I do respect teams that use invitationals as regionals/state practice. I know at my middle school, we generally took regionals a lot more seriously than the one invitational we attended - we had a decent chance of medalling at regionals, but Kraemer/JT/etc. at Mesa Robles Invitational made medalling really difficult. I really enjoyed the invitational, though. Harder tests and powerhouse teams gave me more "room" to test how well I had prepared, if that makes sense.
A smaller or new team would probably not benefit as much from attending a very hard invitational and feel discouraged. In the same thread, a team that had poured in a lot of time would feel disappointed by an invitational with shorter test.
I think it's best that invitationals exist at a wide variety of levels, and that schools choose the ones that fit their needs the best. (I understand transportation costs are a pain, though)

I agree, there are two different types of invitationals, those that are meant to mimic regionals and those that are meant to mimic states/nationals, and most strong teams attend both and expect good tests. Regionals is meant to be the lowest level of competition, it wouldn't make sense for invitationals to be lower than them when for many teams the entire point of going to invitationals is to practice for nationals. In a normal year there are a sufficient amount of smaller scale invitationals that powerhouse teams don't attend that satisfy the needs of the local teams prepping for regionals and hopefully states, but this year a lot of those tournaments have vanished and all that's left are the tournaments who cater to the national audience.

Yes, I do agree with that. However, me being a 6th grader and all, I had practice tests with regular probably state/regionals level. So, I went to Rickards, and boom... 8th graders at my school say it was like nationals. Me: That would explain. But I think that for some people who got like 1 place below me, 23.50 for example on machines out of 286 may be pretty discouraging for them. This could cause them to quit science olympiad and/or get really sad (hopefully none actually happened). I think you shouldn't start of like easy easy, but maybe for the time being at like state levels for invitationals. My regionals is like in February, so there would be plenty of time to go to National Level Invitationals.