Test Bank for TD Questions

Running Chess Tournaments
21

I think Tim Just and the TDCC can best answer some of these questions, but until then…

Club, Local, Senior, ANTD and NTD have three versions each. The Club and Local tests are really the same tests. A club TD who passes at 70% maintains Club TD status, a score of 80% or higher and they move up to Local TD.
These tests are 40 question multiple choice tests.

Last summer I looked at about 100 or more Club/Local test results looking for possibly badly written questions. It had been noticed that many Club TDs, more than a few with pretty extensive experience, were failing the exams for renewal. I wondered if any of the questions had been nadly worded

I rec’d 3 large scanned sets of results last summer, with the prospects of more but that stopped I suspect as Phil Smith moved on. One version of the Club test, taken by 35 different Club TDs produced the following stats:
(this sample was solely Club TDs looking to recertify and/or upgrade)

Average percentage of right answers across all 40 questions .8844
100% - 7
90-99% - 18
80-89% - 6
70-79% - 6
60-69% - 2
less than 60% - 1 43% was the low
100% of testers passed this version. Appears to be more variability than below, but everyone passed.

Another sample of 37 testers, an equal mixture of Club and Local TDs taking the same version produced:
Average percentage right answer across all 40 questions: .8866
(This one added in Local TDs who were re-certifying after lack of activity)
100% - 3
90-99% - 14
80-89% - 16
70-79% - 4
60-69% - 1
Less than 60%-2 (a 59% and a 47% correct)

83.8% passed, six failed. This collection of tests brought in Local TDs who had failed to direct enough tournaments in the previous 6 years, so had to renew by testing. Their lack of activity seems reflected in the drop in successful tests from 100 to 88%.

The question with 43 & 47% (Same question in both samples) correct seemed badly worded. It was based on the previous question and was related to what was needed to make a correct ruling on the floor. One of those scenarios where any knowledge of two different rules would have been sufficient to produce a correct ruling on the floor. The correct answer for the question required saying you knew both rules and many test takers stopped when they chose the first answer that would have produced a correct question and just didn’t go far enough.

Phil Smith was working on a random test generating system before he left. We’ll have to see if that continued.

Interesting Thread!

Mike

22

The responses to Brian here have said it all better than I ever could.

23

Tim - Anything done with the stats turned in last summer?

Mike

24

If question N depends upon question N-1 being present, that complicates the random selection process a bit, doesn’t it?

25

One possible route to go would be a form of a modified Keller plan, where incorrect answers lead to being asked additional questions on that subject. (That is a lot more work than just writing a large block of questions, though.)

26

Obviously. Any questions linked to each other would have to be left out of a random generator.

27

Or a random generator would have to recognize that they are related and select select both, in proper sequence, or neither.

28

Unnecessary complication. Simpler would be that if there are X number of different sections, each section must be passed with a necessary score (70 or 80%) or at most a couple of sections you can miss the minimum score but the rest you have to meet/exceed the minimum score. Even this, can be seen as unnecessary complication because we sure don’t do this today and really we’re looking for a way to automate the process and provide usable study material.

29

I don’t feel strongly that the certification test questions need to be separated from the study questions. But I don’t see why there couldn’t be two sets if that is deemed important. There are good reasons for both. For study, I am convinced that most people will learn the rulebook better if they are exposed to questions similar to those on the tests. For testing, a question bank would permit the tests to be administered on-line.

By the way, I suspect with an open book test most people are probably passing the tests on the strength of their general familiarity with the book and their ability to find the answers in a reasonable time. That is probably a longer time scale than they would have to find an answer in the rulebook in an actual tournament ruling situation. Fortunately, the situations that arise in tournaments are generally the same basic situations repeatedly. The more exotic rules, which the TD’s are less apt to readily know, probably don’t come up that often.

30

After 35 years TD’ing, I still run into situations that I’ve never experienced before.

Brian, you are a Club TD who expires in 2014. Unless you want to make arrangements to direct a couple tournaments and fulfill the experience component for Local TD you won’t experience taking a test until then. Once you have actually taken a couple, you can do more than suspect. I’m also sure that in the TDCC workshop in Orlando, that arrangements could be made to allow you to experience taking a test in their closed book situation.

MA

31

You can lose the condescension. I have actually directed quite a lot of tournaments over the course of five years, both rated tournaments (as an “aide”) and unrated, as the Chief TD, by the way using a pairing program that I also happened to write myself, which gives me a considerably better understanding of pairing rules than I dare say most TD’s get from running WinTD. In 2010, for example, I TD’ed at six tournaments. One of those was the fifth of the annual series, and I have been a TD at all five. So when I say that most TD activity centers on just a few rules, and that many of the rules hardly ever come up, I am speaking from experience. I note that with all your over-eagerness to put me in my place, you didn’t actually bother to contradict this.

I don’t mean to blow your mind, but some people don’t care that much about certifications. At MACA scholastic tournaments, generally half of the TD’s are not USCF-certified, including people who are main-stays of our scholastic program. (I don’t include myself as a “main-stay”.) The Chief TD is USCF-certified, and generally there is at least one other USCF-certified TD. But there are always two or three others who are not USCF-certified, doing the same type of duties as the TD’s who are certified. I finally filed the form because MACA has been losing some of its USCF-certified TD’s, and we might need some more USCF-certified TD’s. So, you will be seeing me accumulating tournaments in my MSA listing as “TD”, even as “Chief TD”, maybe. But that won’t mean I’m doing anything different than I have already been doing. It will mean only that my name is being put into the tournament submissions to the USCF.

32

I don’t think Mike was being condescending. I thought he was just urging you to get a little experience in taking the test before driving for changes. For that matter, some of the changes you are pushing for may happen anyway (the major sticking point being the generation of enough questions to make a question bank plausible).

The normal testing procedures is a mailed open-book test with multiple weeks available to return it. There is also an option to take the test in a closed book setting under an NTD at a National tournament (with a lower correct percentage required to pass).

APICS uses electronic questions with automatic follow-up questions for those areas that the candidate has missed.

33

What aspect of my suggestion to develop a question bank depends on experience as a TD?

This is a joke. Why bother with such a test? How is the ability to find answers to questions in the rule-book over the course of “multiple weeks” in any way related to the ability to make rulings at a tournament when the heat is on? An open-book test might be alright, but then it should be timed, with not enough time to lookup all the answers, but only some of them. The procedure you are describing tests only that the person (1) has access to the book; and (2) can read.

34

There are people who have failed the local test multiple times, even though it is open book.

I thought that it took a HIGHER score to pass the test closed book (because that is a substitute for having the requisite experience), but it is not something that is requested very often. (I passed the senior exam closed book in 1987, FWIW.)

35

What is the failure rate on the first, second, etc attempts?

36

I don’t have that data, I just know of several instances of players who failed all three tests at either the local or senior level.

37

The FIDE Arbiter exam when offered in conjunction with the Internet based delivery of the seminar is an open-book, open-notes exam which is emailed out and there is 4 hours allotted for the exam (moving to 5 1/2 hours I believe). Has to be received by the grader by the end of the testing period or it doesn’t count.

38

(3) understanding the rules
(4) understanding which rules are applicable
(5) understanding which rules to use when they are in conflict (ridiculously simple example - 6-player four-round event where the only way to equalize colors in round four is for a player to play somebody already played - which rule is more applicable, equalizing colors or avoiding replaying a player?)

A timed test would require a proctor (or an on-line system that doesn’t currently exist and didn’t used to even be possible - something would have to be done to take into account the possibility of internet access lost due to either an ISP problem or a power problem). For some areas, requiring a timed test would have limited TDs to just those willing to travel some distance to take such a test (or would have required a local proctor that a candidate or the USCF would have to pay for before the candidate was able to take a test).

39

When I took the Local TD test, there were three questions on which I was certain that none of the possible answers given were correct. I’m dying to know what the answers to those were, and which parts of the rulebook govern them . . . alas, just before he left, Phil Smith told me that information couldn’t be given out. :frowning: (He did tell me which items I got wrong, though – I missed four, and two of them were among those three.)

40

Like I said, be able to: (2) read, which generally involves not just reading the words aloud, but comprehension. In your example, the rulebook actually states what the priority order is for the pairing goals.

If your (3)-(5) are not based on comprehension of the rulebook, what are they supposed to be based on, and on the basis of what is one answer deemed “correct”?

See point above about reading comprehension.