National Test Discussion

[Uber]

==double post sorry mods==

You know the exam was poorly written when there was a SEVEN WAY tie. How did the proctor even break the tie when there were only 3 very easy tie breakers (3! is 6 and 6 is less than 7)? Did she just flip a coin?

Errr... not to be that guy but you'd want to use 2^3 = 8 > 7 (because you can get each question right or wrong)

Tiebreakers are ranked in order 1-3, so it's actually 2*3=6

[fizzle]

You know the exam was poorly written when there was a SEVEN WAY tie. How did the proctor even break the tie when there were only 3 very easy tie breakers (3! is 6 and 6 is less than 7)? Did she just flip a coin?

==double post sorry mods==
Errr... not to be that guy but you'd want to use 2^3 = 8 > 7 (because you can get each question right or wrong)

Tiebreakers are ranked in order 1-3, so it's actually 2*3=6

Think of it like a binary number - using three bits, or questions, there are eight different rankings you can get, in this order. 000, 001, 010, 011, 100, 101, 110, 111
Though I find it hard to believe that a seven-way tie could be completely broken this way - things would have to fall perfectly into place. How did they rank teams when they had the same tiebreak record? An arbitrarily chosen question? Alphabetically??

[Skink]

Biology events at any level should involve less factual recall and more interpretive stuff. Timing-out teams is fine at larger tournaments like this provided it's, still, not a pile of facts copied off the cheat sheet or binder (and this is why I'm concerned about Herp having a binder like Invasive). Now:

just a large amount of easy Pre-AP Biology level questions on the water cycle.

The problem is possibly that:
1. The ES is on the biology committee, and the committee doesn't have that high of expectations for competitors. Look at the rules and training handouts. See point 5.

2. See, events are designed as if participants haven't any special background. An ES is technically not supposed to assume participants have an AP (or equivalent! Even though self-studying is a thing...and the backbone of this program) background and have only started studying from square one. It's why it's mean (albeit fair game...maybe) to put a lot of chemistry on a bio test or excessive mechanics on an Astro test, for example. Those may be prereq skill sets, but events don't have prerequisites.

3. I don't follow this, but the recommendation in my state (which is likely the recommendation elsewhere) is to have a third of the test be easy, medium, and hard each. Consequently...in order to be doable by, I dunno, Montana (sorry Montanites! Montanians? Monts?), there's gotta be easy stuff that some of you have seen at four competitions this season already...according to that system. That takes up time and space and hurts the high-end teams, yes.

Devil's advocate, though:
4. Sometimes, participants feel they perform well in an event due to content mastery when they really perform well due to MC (which is the standard test format) leading them along as opposed to them demonstrating "actual" mastery in free response questions. That's probably not the case here because it sounds like a typical ran-out-of-time test (though this ES does value higher-order thinking in case studies and free response questions), but this is a thing, and I've seen it time and time again.

5. When push comes to shove...there is a point where most teams do crash and burn in C division, and it's kind of hard to gauge where it is at times.
It's easy for me to slap down point 1 above as if SO doesn't expect much from participants, but I believe this comes, in part, anyway, from the experience that participants don't handle brutal tests very well (not to mention them being hard to grade or whatever). They, just, don't perform! I've tested Daniel Wright for years and can get 70-80% from them on what I consider a hard (but doable) test (and, in fact, that's how I often calibrate myself...if they can't do it, maaaaybe it was too hard). In C division? In my dreams...it's not a slight against participants, either. Science is hard. Science in a timed, stressful environment is doubly so.

The bottom line: I don't remember who it was, but somebody posted how they prepare tests a week or two ago, and East commented that he liked that system. I agree. Prospective event supervisors: prepare your events with the rules in front of you, and be prepared to spend hours crafting thoughtful, interpretive questions/stations/activities that make participants think, maybe challenge or interest them a bit. Expose them to something you've seen that they haven't. Ask about something they have seen from a different vantage point or a higher level. Doing this doesn't magically spread scores, as somebody still has to get 35th place (and they probably aren't bad at the event!). It has a better rate of spreading the high scorers, though.

P.S.-tiebreakers done easy (assuming a pre-selected questions event):
A. Designate some tie-breakers (say, 1st through 4th).
B. Designate 5th through nth as numerical order.
As long as there are free response questions on the test, which there should be, there should not be a scenario where it's impossible to separate two teams, even among really underperforming ones. I would hope that National folks use this or a very similar method.

[efeng]

The maps are called spot maps. The reason why the were centered around the star in the first map was because the people who lived closest to the market were the people who were likely to go, because it was nearby and convenient. The reason why the second map had the cases randomly distributed around the county and the surrounding counties is because there was either a large event which attracted a large population from everywhere (i.e. County Fair, State Fair, etc.), or that the source was evenly distributed among the counties (i.e. supermarkets). Later, it was discovered that it was because of the milk from Supermarket A, and not a large event.

I think knowing what a spot map was boosted my partner and I from maybe 10th to 3rd. That was fairly obscure for such an easy test.

That's not very obscure, it's like the first page of Principles of Epi

Yeah, and there's also John Snow. If you know remotely anything about John Snow's 2nd cholera investigation, you should be able to know spot maps.

[The48thYoshi]

The maps are called spot maps. The reason why the were centered around the star in the first map was because the people who lived closest to the market were the people who were likely to go, because it was nearby and convenient. The reason why the second map had the cases randomly distributed around the county and the surrounding counties is because there was either a large event which attracted a large population from everywhere (i.e. County Fair, State Fair, etc.), or that the source was evenly distributed among the counties (i.e. supermarkets). Later, it was discovered that it was because of the milk from Supermarket A, and not a large event.

I think knowing what a spot map was boosted my partner and I from maybe 10th to 3rd. That was fairly obscure for such an easy test.

That's one of the issues with easier tests. It is harder to differentiate good teams from teams that are good at guessing. Like I said before, most of the test was MC or fill in the blank-esque questions that didn't test a team's real knowledge. I believe there were less than 10 open response questions, and no statistical analysis?

[The48thYoshi]

The maps are called spot maps. The reason why the were centered around the star in the first map was because the people who lived closest to the market were the people who were likely to go, because it was nearby and convenient. The reason why the second map had the cases randomly distributed around the county and the surrounding counties is because there was either a large event which attracted a large population from everywhere (i.e. County Fair, State Fair, etc.), or that the source was evenly distributed among the counties (i.e. supermarkets). Later, it was discovered that it was because of the milk from Supermarket A, and not a large event.

I think knowing what a spot map was boosted my partner and I from maybe 10th to 3rd. That was fairly obscure for such an easy test.

That's one of the issues with easier tests. It is harder to differentiate good teams from teams that are good at guessing. Like I said before, most of the test was MC or fill in the blank-esque questions that didn't test a team's real knowledge. I believe there were less than 10 open response questions, and no statistical analysis?

@efeng- congrats on DD; you must've gotten near full score- getting a single question wrong might have dropped a team 3-4 places

[BigBrown]

Most of my events I enjoyed the challenge of the tests. While I didn't do particularly well I would just like to say that the ecology test is the worst test I have taken all year. I think it would have been better if the event supervisor just printed one of the test exchange. I took invitational tests from October that were much better. It was a pathetic attempt for a national tournament. There were more questions about physics than ecology on the test. Normally I wouldn't be so vocal in my criticism because I like to follow the "if you can't do better yourself than don't say anything", I could make a better test by myself in 4 hours tops. Lastly why is the Enviromental Science event always in stations, given the subject matter the only reason ecology (or green gen) should be in stations is that the event supervisor couldn't print out enough copies, and from what I've heard I wouldn't be that surprised if this was true.

[efeng]

I think knowing what a spot map was boosted my partner and I from maybe 10th to 3rd. That was fairly obscure for such an easy test.

That's one of the issues with easier tests. It is harder to differentiate good teams from teams that are good at guessing. Like I said before, most of the test was MC or fill in the blank-esque questions that didn't test a team's real knowledge. I believe there were less than 10 open response questions, and no statistical analysis?

@efeng- congrats on DD; you must've gotten near full score- getting a single question wrong might have dropped a team 3-4 places

haha, thanks! i would not doubt it if we won by a couple points. i hope chalker answers my questions on raw scores.

[Taliphate]

Chemistry Lab (12) - The test was in short stations like always and required very fast calculator skills in order to finish the 20-40 question packets in the allotted time. The bulk of the questions were in non-metric units such as Fahrenheit, requiring conversions to get to the appropriate unit. The questions themselves, barring the conversions, were very basic, most of which were change in heat and ideal gas law problems. I didn't notice the proctor reuse any old test questions this year, which should probably have happened every year but didn't. Although this many quick questions approach is within the rules, I personally felt it didn't offer me anything interesting or profound, just plain old calculator mashing (Kudos to those teams that can think on their feet and are speedy). 4/10

Hydrogeology (5) - The test was a pretty standard Hydrogeology test, no surprises really. It was a fair bit harder than last year's Hydrogeology test, but could still be completed well within the time limit. The remediation and contaminants were pretty standard and the part 1 questions were mostly things my partner and I had seen before. There wasn't really anything terrible about this test, but there wasn't really anything that was great or impressive either. 6/10

Remote Sensing (2) - This was an amazing test, probably the best test I've taken this year (narrowly beating the Chem Lab test at Golden Gate). The proctor was great, really nice, very encouraging, and super helpful. The test itself was on the hard side (as Nationals tests should be), and hit many of the areas listed in the rules (although there were less A-Train satellite questions than most tests). A good portion of the math involved could be figured out by reasoning and knowledge of physics and content related to the event (such as energy balance and satellite sensor operation), which I personally find better than having a memorized list of formulas. The graphs and charts provided were well used, although I felt the test could've been extended with more questions over them. A vast majority of questions involved lots of thought and it's clear that the proctor spent time crafting this test. I hope my experience in all my events can always be like this. 9.5/10

[Uber]

Biology events at any level should involve less factual recall and more interpretive stuff. Timing-out teams is fine at larger tournaments like this provided it's, still, not a pile of facts copied off the cheat sheet or binder (and this is why I'm concerned about Herp having a binder like Invasive). Now:

just a large amount of easy Pre-AP Biology level questions on the water cycle.

The problem is possibly that:
1. The ES is on the biology committee, and the committee doesn't have that high of expectations for competitors. Look at the rules and training handouts. See point 5.

2. See, events are designed as if participants haven't any special background. An ES is technically not supposed to assume participants have an AP (or equivalent! Even though self-studying is a thing...and the backbone of this program) background and have only started studying from square one. It's why it's mean (albeit fair game...maybe) to put a lot of chemistry on a bio test or excessive mechanics on an Astro test, for example. Those may be prereq skill sets, but events don't have prerequisites.

3. I don't follow this, but the recommendation in my state (which is likely the recommendation elsewhere) is to have a third of the test be easy, medium, and hard each. Consequently...in order to be doable by, I dunno, Montana (sorry Montanites! Montanians? Monts?), there's gotta be easy stuff that some of you have seen at four competitions this season already...according to that system. That takes up time and space and hurts the high-end teams, yes.

Devil's advocate, though:
4. Sometimes, participants feel they perform well in an event due to content mastery when they really perform well due to MC (which is the standard test format) leading them along as opposed to them demonstrating "actual" mastery in free response questions. That's probably not the case here because it sounds like a typical ran-out-of-time test (though this ES does value higher-order thinking in case studies and free response questions), but this is a thing, and I've seen it time and time again.

5. When push comes to shove...there is a point where most teams do crash and burn in C division, and it's kind of hard to gauge where it is at times.
It's easy for me to slap down point 1 above as if SO doesn't expect much from participants, but I believe this comes, in part, anyway, from the experience that participants don't handle brutal tests very well (not to mention them being hard to grade or whatever). They, just, don't perform! I've tested Daniel Wright for years and can get 70-80% from them on what I consider a hard (but doable) test (and, in fact, that's how I often calibrate myself...if they can't do it, maaaaybe it was too hard). In C division? In my dreams...it's not a slight against participants, either. Science is hard. Science in a timed, stressful environment is doubly so.

The bottom line: I don't remember who it was, but somebody posted how they prepare tests a week or two ago, and East commented that he liked that system. I agree. Prospective event supervisors: prepare your events with the rules in front of you, and be prepared to spend hours crafting thoughtful, interpretive questions/stations/activities that make participants think, maybe challenge or interest them a bit. Expose them to something you've seen that they haven't. Ask about something they have seen from a different vantage point or a higher level. Doing this doesn't magically spread scores, as somebody still has to get 35th place (and they probably aren't bad at the event!). It has a better rate of spreading the high scorers, though.

P.S.-tiebreakers done easy (assuming a pre-selected questions event):
A. Designate some tie-breakers (say, 1st through 4th).
B. Designate 5th through nth as numerical order.
As long as there are free response questions on the test, which there should be, there should not be a scenario where it's impossible to separate two teams, even among really underperforming ones. I would hope that National folks use this or a very similar method.

1. Robyn Fischer is not the biology committee chair, Karen Lancour is. The training handouts are supposed to be the basics so that teams can have a starting point to work off of.
Fischer's test last year was well written. Even within the same topics (general ecology, alternative energy, etc.), if you compared the two national tests, you would see how much more critical thinking the 2016 version required. She proved that she could write well-done tests that test both thinking and knowledge (mostly) without testing trivia.

2. Pre-AP Biology here is a freshman-level course that includes ecology. The water cycle questions would have been suitable for an in-class test.

3. Problem is too much of the test was easy, involving only superficial factual recall or random vocabulary. If she wrote critical thinking questions that were on the easy side, it would still help the higher performing teams due to more experience and broader knowledge.

4. Test was 80% MC with 2-word-long short answers, and 1 or 2 sentence-length answers. We finished most of the 3.5 minute stations with 30 seconds or more to spare. There have been many other station tests that were much more time-challenging (MIT especially).

5. MIT ecology was hands down the most brutal test I've ever taken. Golden Gate also had a much more difficult test than nationals, with more critical thinking involved. We won both. The national test came nowhere close. We finished most stations and double-checked with time to spare.

Ecology is an extremely variable event where almost every tournament has a different style and focus. Yes, the style is somewhat different from other tournaments, but the content it tested was overly basic, with some random sections mixed in. The random sections involved guessing what the proctor wanted. Since most of the test was too easy, the series of fill in the blank biodiversity and random vocab was what determined most of the final rankings, and we ended up on the wrong end. When 25% of the teams score within 3% of each other, there's something wrong with the test.