Musings on Test Length

[knightmoves]

I have noticed that smaller invitationals are beginning to disappear with online, and that some powerhouse teams are attending pretty much every invitational.

Being online means you don't have any travel constraints - no parents dropping kids at the school at 4am to get the bus to an invite, no hotel stays, and no transport costs. Just the bus to take the team to a physical competition costs significantly more than the entry fee. So combine the fact that tournaments are easier and cheaper remotely with the fact that there's fewer other things going on, because of Covid, and it's much easier for keen teams to compete every weekend. Plus, of course, there's no problem with attending a competition on the other side of the country remotely (unless you're a west-coaster having to get up for east-coast start times )

What happens if you were to write a full-length "easy" test aimed at lesser competitors, and a full-length "hard" test aimed at powerhouse teams, concatenate them, and award maybe 1 point per question for the easy ones, and 5 points per question for the hard ones?

[sneepity]

Wow! That's a really nice test format (especially with reminding students to skip questions). Yeah, that's basically what I like- the hardness is distributed evenly, making sure I can get the most credit I can. This knocks out hard tests out of my preference- most students won't be able to finish all of them anyway, so it's better for them to try their best and spend their time answering all the possible questions they can get points for.
And yeah, it's really important to have a healthy mindset when you go to any competition, not just scioly- you should expect the worst and the best placement for the amount of effort you put in. Likewise, knowing that sometimes we can do really bad, and that we can progress and get better is something to keep in mind as well!
And yeah, it's true- test writers write in a variety of ways, and you just have to get through bad tests to have experience with them. The test writers can't really cater to each students' needs, they have to learn to adjust.

[sneepity]

What happens if you were to write a full-length "easy" test aimed at lesser competitors, and a full-length "hard" test aimed at powerhouse teams, concatenate them, and award maybe 1 point per question for the easy ones, and 5 points per question for the hard ones?

I feel like easy and hard is objective to events, but that seems like a nice idea. That would also mean more test writers, more labor, and more work on scoring.
But I don't get how the point system you described works- if they were the same length, wouldn't the teams who took the harder tests get more points? Or you could multiple the easy teams' score by 5. Sorry if I'm not understanding!

[knightmoves]

What happens if you were to write a full-length "easy" test aimed at lesser competitors, and a full-length "hard" test aimed at powerhouse teams, concatenate them, and award maybe 1 point per question for the easy ones, and 5 points per question for the hard ones?

I feel like easy and hard is objective to events, but that seems like a nice idea. That would also mean more test writers, more labor, and more work on scoring.
But I don't get how the point system you described works- if they were the same length, wouldn't the teams who took the harder tests get more points? Or you could multiple the easy teams' score by 5. Sorry if I'm not understanding!

Yeah, that's the point. It's a way of making a test that can separate teams at the top end and also teams lower down. Experts will do mostly the hard questions, and get many points. Lesser competitors will do the easy questions, with maybe a couple of the harder ones thrown in.

Basically the idea is that the overall test is much longer than anyone can feasibly complete, but has questions with a range of difficulties worth a range of points, so you can choose to do hard questions that score more, or easy questions that score less.

You're right - it's more work writing the test. It shouldn't be much more work scoring, because nobody will answer all the questions. 5 points was a guess to try to make it not worthwhile for a top team to spend its time speeding through easy questions. You can probably tell I haven't thought this through completely.

[RiverWalker88]

I also might add that it's really hard for test writers to know what to expect especially now. Many have no idea if they are writing for less experienced teams or for powerhouse teams, or a combination of both (I couldn't tell you for Socorro if I tried). Even if a powerhouse school is signed up, are they sending their A or D team? And we can't expect test writers to recognize most of the powerhouse schools (I certainly can't). This is tough to gauge in a normal competition, but then throw teams from all over the nation in it, and top that off with cheating potential, and now test writers (especially early test writers) are essentially shooting into the dark. While blindfolded. And handcuffed. And falling. On Mars.

One of Socorro's event supervisors put it well, "We, too, are curious to see how this turns out. Not sure if we are shooting too high or too low with our expectations ... hopefully, somewhere in the middle."

Additionally, while I'm posting, long tests (especially for invitationals) aren't a bad idea. Granted, if they're so long that you're going to be lucky to finish Section 1 of 3 50 minutes, that's probably too long, but if an invitational keeps their test a little on the long side so that they can get a good distribution of content, I would say that's probably better, as it homogenizes the practice you get. Of course, as was stated above, if you are looking more for something similar to a regional competition, you might try looking for easier comps (which, as was also stated above, have pretty much vaporized or have >1 powerhouse schools attending them).

[sneepity]

What happens if you were to write a full-length "easy" test aimed at lesser competitors, and a full-length "hard" test aimed at powerhouse teams, concatenate them, and award maybe 1 point per question for the easy ones, and 5 points per question for the hard ones?

I feel like easy and hard is objective to events, but that seems like a nice idea. That would also mean more test writers, more labor, and more work on scoring.
But I don't get how the point system you described works- if they were the same length, wouldn't the teams who took the harder tests get more points? Or you could multiple the easy teams' score by 5. Sorry if I'm not understanding!

Yeah, that's the point. It's a way of making a test that can separate teams at the top end and also teams lower down. Experts will do mostly the hard questions, and get many points. Lesser competitors will do the easy questions, with maybe a couple of the harder ones thrown in.

Basically the idea is that the overall test is much longer than anyone can feasibly complete, but has questions with a range of difficulties worth a range of points, so you can choose to do hard questions that score more, or easy questions that score less.

You're right - it's more work writing the test. It shouldn't be much more work scoring, because nobody will answer all the questions. 5 points was a guess to try to make it not worthwhile for a top team to spend its time speeding through easy questions. You can probably tell I haven't thought this through completely.

Oh okay, I see. I think that a test should only have as many questions as someone can answer to make sure someone gets all the points they can, but that seems okay too. But to be fair, I think the score they get would just tell you have well they answered the questions they got up to, instead of how much knowledge they had that was being asked on the test.

[sneepity]

Oh okay, I see. I think that a test should only have as many questions as someone can answer to make sure someone gets all the points they can, but that seems okay too.

(this isn't really related to knightmoves' idea, just talking about long vs. short in general)

Hmmm that's interesting to hear. One thing with being a test writer is that you don't actually know how prepared your participants will be. As a result, many of us err on the harder side, as every team leaving some unfinished questions is better than many teams finishing and getting very close scores near the top. Furthermore, a longer test discourages cheating - cheaters will work more slowly and be unable to answer as many questions as teams who know the information. I do agree that passing a certain length is overkill, though.

But to be fair, I think the score they get would just tell you have well they answered the questions they got up to, instead of how much knowledge they had that was being asked on the test.

Is this not true of a shorter test as well? Accuracy is very important in answering questions. Since I'm not sure I'm reading your point(s) correctly, forgive me if I guess incorrectly:

I agree that it could create an emphasis on speed, which is what I believe you are trying to say. However, if you finish the entire normal test, the test also only tells you "how well they answered the questions they got up to" - the only difference is that there is a limit on how far they can get.
Consider two teams of the same knowledge level, but Team A is faster than Team B. With a shorter test, they would both finish and get the same score. With a longer test, A would have a higher score. Shorter tests benefit slower teams, but only by creating more ties/close scores.

I agree that shorter tests may force students to attempt problems they didn't know instead of just skipping them (which is what I think you are also trying to say). However, I also see this as a safeguard against accidental bad questions - if you write a very trivia-based or poorly-worded question, it will affect placements far less because teams can afford to leave it blank and do more well-written questions instead.

I agree most teams will not be tested on the questions near the end of the test - how is this different from not including them at all?
This could create issues if the front of the test is one topic and the back is another - this can be avoided by evenly spreading questions of different topics throughout the test. (or putting easy questions at the beginning of sections and telling teams to skip to those if they get stuck)

I do realize that insanely, absurdly long tests can unintentionally allow a new strategy - studying one topic in detail and neglecting the other aspects of the event. I believe this is avoidable with any test of somewhat reasonable length, even if it is written to be far longer than any team can finish. Thank you for opening my eyes to this potential disadvantage.

[sneepity]

Oh okay, I see. I think that a test should only have as many questions as someone can answer to make sure someone gets all the points they can, but that seems okay too.

(this isn't really related to knightmoves' idea, just talking about long vs. short in general)

Hmmm that's interesting to hear. One thing with being a test writer is that you don't actually know how prepared your participants will be. As a result, many of us err on the harder side, as every team leaving some unfinished questions is better than many teams finishing and getting very close scores near the top. Furthermore, a longer test discourages cheating - cheaters will work more slowly and be unable to answer as many questions as teams who know the information. I do agree that passing a certain length is overkill, though.

But to be fair, I think the score they get would just tell you have well they answered the questions they got up to, instead of how much knowledge they had that was being asked on the test.

Is this not true of a shorter test as well? Accuracy is very important in answering questions. Since I'm not sure I'm reading your point(s) correctly, forgive me if I guess incorrectly:

I agree that it could create an emphasis on speed, which is what I believe you are trying to say. However, if you finish the entire normal test, the test also only tells you "how well they answered the questions they got up to" - the only difference is that there is a limit on how far they can get.
Consider two teams of the same knowledge level, but Team A is faster than Team B. With a shorter test, they would both finish and get the same score. With a longer test, A would have a higher score. Shorter tests benefit slower teams, but only by creating more ties/close scores.

I agree that shorter tests may force students to attempt problems they didn't know instead of just skipping them (which is what I think you are also trying to say). However, I also see this as a safeguard against accidental bad questions - if you write a very trivia-based or poorly-worded question, it will affect placements far less because teams can afford to leave it blank and do more well-written questions instead.

I agree most teams will not be tested on the questions near the end of the test - how is this different from not including them at all?
This could create issues if the front of the test is one topic and the back is another - this can be avoided by evenly spreading questions of different topics throughout the test. (or putting easy questions at the beginning of sections and telling teams to skip to those if they get stuck)

I do realize that insanely, absurdly long tests can unintentionally allow a new strategy - studying one topic in detail and neglecting the other aspects of the event. I believe this is avoidable with any test of somewhat reasonable length, even if it is written to be far longer than any team can finish. Thank you for opening my eyes to this potential disadvantage.

I see what you mean now! I now think that long tests can give teams more opportunities to get points for their knowledge, but only if they get to the end, right? But yeah, there's the problem of badly written questions, and skipping them can give retribution to teams.
Yeah! so like I mentioned, I feel that tests that are good quality have consistent hardness all through the beginning to the end to try to prevent kids from potentially being at an advantage because they couldn't collect those points at the end, and they have a really nice mix of questions instead of "topic" sections. Which I noticed happens most of the time in anatomy- there's a section for each system. What do you think about this? Would it put some teams at an advantage or a disadvantage?

[sneepity]

I see what you mean now! I now think that long tests can give teams more opportunities to get points for their knowledge, but only if they get to the end, right?

Not really; with two tests of very different lengths, usually the longer one is better for separating teams, even if no one finishes the longer test. An unfinishable test is not the same as a short test, because teams still get to do more questions than on the short test.

Yeah! so like I mentioned, I feel that tests that are good quality have consistent hardness all through the beginning to the end to try to prevent kids from potentially being at an advantage because they couldn't collect those points at the end, and they have a really nice mix of questions instead of "topic" sections.

I think a mix of question difficulties in roughly increasing order would meet the needs of a wide range of competitors better, but I agree on trying to limit the advantage that "test taking skills" would grant.

Which I noticed happens most of the time in anatomy- there's a section for each system. What do you think about this? Would it put some teams at an advantage or a disadvantage?

From an ES perspective, writing by section is far easier for many events.

A few technicalities:
Some events, like Crime Busters, have a strict requirement for what percent each section should be for the total score. Writing these events by mixed topic would be a nightmare - extra work making sure each topic has the right fraction of points, students being confused because most teams split Crime Busters by topic.
ID events, like Ornithology, are best written with questions grouped bird by bird. (I dislike the double jeopardy of missing an entire group/station if you miss an ID and have some ideas for countering this) This makes a lot more sense, and makes things far easier for participants, who would otherwise have to flip to a new binder page each question.

But I don't think those are what you meant. I'm going to use Water Quality as an example. The rules give three sections that must be worth 30% each. (bringing and testing the hydrometer is the last 10%)
I write Water Quality tests in three sections, with questions in increasing order within each section. I tell students to skip to section beginnings if they get stuck. This is mainly for me - I know exactly how many points each section should be, relative to the rest of the test. If I take out a question or allot it more points, I know to add or remove points from other questions in that section. This also prevents me from making the whole test on organism ID, for example.
But to get around that, I could always merge the sections when I finish writing. I think my main reason for not doing so is that students tend to split the event by topic (which I'm guessing happens in Anatomy and other events as well). Maybe I like ecology better, and my partner just loves fish and reef organisms. So I do section 1, my partner does section 2, and we do section 3 together. This is far more efficient and enjoyable for us if the test is in sections, since we don't have to keep skipping back and forth, leaving questions blank, or overriding each other's answers.
What if I studied all the sections equally, and so did my partner? In that case, sections and mixed are really the same to us - I could do a section that's just on water testing as easily as a bunch of mixed questions.

I think using test instructions to remind students to skip to easy questions and making it clear that there are sections suffices. What are your thoughts?

[sneepity]

Making sure kids know (I think these are what you mentioned, please correct me if I'm wrong) that you can skip questions, and that they can split up the test evenly is a huge relief in a test taker's point of view. If an ES urged me to do this, and there were enough questions in the packet- my partner and I would be glad we were given some pointers. Your perspective as an (awesome if I didn't mention!) ES makes a lot of sense, since you want to be able to give students the chance to perform to their fullest!

Skipping easy questions or hard ones- in my opinion. if I were taking a test, I wouldn't skip the easy questions. They can still earn you points, even though they may be easy. After all, the question is *easy* because you studied for it. So why not just take a second and complete it, you know!

What you say about water quality does make sense, I can see it being easier for test writers to grade and for test takers to split it up according to abilities and interests. Splitting it up in sections seems like the best idea for tests that have clear topic divides. I'm gonna quote something you said-
"I agree most teams will not be tested on the questions near the end of the test - how is this different from not including them at all?
This could create issues if the front of the test is one topic and the back is another - this can be avoided by evenly spreading questions of different topics throughout the test. (or putting easy questions at the beginning of sections and telling teams to skip to those if they get stuck)"
I think a good compromise for this is the second option you mentioned- going from easy to hard in sections as well so kids have the chance to skip hard questions if they divided the test. However, if partners were to split the studying itself, I guess you can say both partners aren't "fully" learned in their event and are just very invested in their topic (so they can effectively split the test sections). Is this a common practice? If both partners don't study all aspects of an event, I don't really consider them proficient at the event ( of course apart from interests).

And I think you're right when you said that question difficulty should increase as you go- it's less likely that students can answer those questions, so they can afford to not do them. Kind of like bonus points, right?

But I'm still debated on what the most "accurate" way there is to properly test a student's knowledge (or their full potential), not just smart test taking strategies like skipping questions, partners dividing it into sections (although this is a fairly common type of strategy given the time constraints and the number of questions). What's your opinion on this?

Edits: adding more stuf