Or do like most Delegates and just don’t show up. 
Especially if you don’t like political sausage-making (compromises.)
If I offered to make a question bank out of the current questions, for a start, would anybody agree to this, given that this would involve me seeing the questions when I am not – yikes! – even a Local TD? And if I just invented new questions, would anybody accept them – for the same reason?
The question bank shouldn’t be made from questions from the current tests. They should be new questions and vetted here. This way we don’t comprise the integrity of the tests that are being used.
There are two issues.
-
An on-line question bank for anybody to look at and review (similar to the practice questions you get for APICS or ACT).
-
A question bank for the TDCC to use to generate tests and potentially move to on-line testing.
Number 2 would require TDs of at least the level being tested to come up with questions for the tests. It should really be at least a level higher to remove the risk that a lapsed TD would take a test for reinstatement that included some of the TD’s own questions (which narrows down the possible volunteers to create questions for the SrTD tests).
As far as number one goes, Brian actually could be one of those that come up with questions and potential answers. I anticipate that there would be some refinement of the answers, such as making the wrong answers more logical. As an example, when seeing whether or not a test-taker understood multiplication, a multiple choice question asking what 6 times 2 is could have answers of: 8 (wrong - that is 6 plus 2); 12 (right); 3 (wrong - that is 6 divided by 2); and 4 (wrong - that is 6 minus 2). That would be a better test of the understanding of the multiplication operand than giving possible answers of 1, 12, 47 and 864.
The APICS tests generally had wrong answers that could be reached if a formula was used that was not applicable to the situation. They weren’t testing for mathematical capability, but rather for whether or not the test-taker understood what needed to be done.
I don’t see the workshop open to this, but I could be wrong. I can see you in a computer/technical/recordkeeping role, but I would think you need two certified TDs to work with you to sound out the questions. I doubt they would have to be highly certified. Tests have been designed by SrTDs before. TDCC would probably review the final output.
There aren’t many projects of this scope, that are done singlehandedly.
All the best, Joe
IMHO: All new questions would need to be proof read. It would also seem unfair for you to work on this project and not get some TD credit/TD waiver regarding your own certification. If nothing else it would certainly be a special one of a kind TD workshop.
As for you getting copies of the exam right now…I believe that would be up to the Board or the Office.
BTW, read over Jeff’s posts in this thread. It is like he read my mind!?
Agreed.
I think this is a great idea.
Hmm, a “volunteer” being given access to a test bank of questions. Who protects for leaks?
Which compromises them, doesn’t it?
No, because they are questions for the study bank, not the test bank. There is a difference.
Thus those questions can be vetted here since it’s for the study bank.
What is involved in turning a list of current questions (the union of questions on the various versions of current tests) into a test bank? Is there some sort of generic test-generation software that they sell to teachers for this? I seem to remember that such things existed.
It would allow us to segregate data (test questions and correct answers) from algorithm (randomized test-question selection, presentation to candidate, and automated grading). Brian could work on the algorithm side without knowing the questions. Also, the algorithm could be vetted on the study bank and then applied to the test bank.
If we want to be more sophisticated, and ask followup questions based on whether the candidate answers a previous question correctly or wrongly, then the question selection is not independent of the questions themselves, and so those linkages would have to be created by those who know what the questions are.
IMHO I don’t see adaptive testing techniques even remotely necessary here and adds unnecessary complication. If we want to do that well later, then it can be addressed. Start off simple and then move to complex.
Even simple will be resource consumptive to get the questions created and vetted. The first step, IMHO, is to break this up into different knowledge domains / areas of knowledge / testing groups. Then we can see how many of these we have and move onto question creation.
Really we shouldn’t be looking to build a Mercedes when a Kia will do.
Sorry, I don’t see that much difference. In both cases you want to present the candidate with a question he or she has not seen before (at least on the first pass through the question bank) so that he or she has to reason through the choices, but without the pressure of getting it right that is on the test.
If the questions have been posted here and the answers discussed, that sort of defeats the purpose. Not as seriously as with the test questions, perhaps, but you still lose that ‘facing a new question’ atmosphere.
I think it’s unavoidable, unless you have hundreds of questions of the study bank at each level. I am not sure the study bank is all that useful for that reason, but at some point one has to realize that to be completely prepared for the test, the examinee should be focusing on the rulebook and application rather than the few sample questions that are floating around.
If we have sample questions, then we’ll surely get complaints from someone who fails the test that the sample questions were not representative. And to have that argument here would lead to disclosure and thus effective destruction of many existing test-bank questions.
How much of a problem is this? How many people who fail an exam (and maybe a second try) report that they were denied an adequate opportunity to prepare?
Sigh?!
I suggested that the question bank be in QTI format because that seems to give the most options as far as testing software is concerned. Testing systems generally either use QTI directly, or support a method of importing QTI question banks to their native (usually proprietary) format.
A testing system which presents the questions in a QTI question bank is called a “QTI Player”. There are several open source QTI Players available, and as Sevan and I were discussing earlier on this thread, some of the online testing SaaS services also accept QTI input. The popular Moodle Learning Management System, web-based and open-source, includes a QTI Player.
As to question input, there are “QTI Authoring” tools for creating questions in the format. Most of these are proprietary, unfortunately. Examples are, Questionmark, Respondus, and Question Writer. There are however some free solutions, which I am looking at. I do not have a recommendation at this point. Perhaps Sevan can chime in here? While developing a QTI authoring tool that handles the full range of QTI question types is a significant undertaking, our questions are a single format – straightforward multiple choice – and it is not ridiculous to contemplate developing our own simple web-based authoring tool, also.
As to the existing test questions, I am guessing/hoping they are in MS Word format. It could be some desktop-publishing format, though. It might be that they exist only in PDF format, ready to be printed, and it might be necessary to extract the original text from a PDF. The worst case is that the questions exist only in hard-copy and have to be typed in again from the hard-copy. That would not be a disaster, since there are only 240 questions – a few hours work to re-type. Perhaps Nolan or Tim Just can give us some information at this point as to what format the existing questions are in.
Once we have them as text, the questions can be converted to QTI format, probably with some combination of Unix command line tools, perhaps with some scripting and a little manual markup to distinguish questions from answers and to identify the correct answer for each question. Again, the QTI format is rather complex, and it could get burdensome if our existing questions included a lot of different question types with complex scoring rules, adaptive flow, etc. But that is not the case.
Regarding the issue of discussion by the question developers on a public forum, I don’t see that as so much an issue for the study questions. There will be a lot of questions. Someone who decides to study the rules by reading discussion by question developers in a public forum rather than reading the rulebook or just using the final study questions is probably not using his time very efficiently, but he is still learning the rules. Anyway, how people study is up to them. The only real problem arises if study questions ever evolve into being test questions.
As for the actual test questions, having the test be open-book with plenty of time means that people can already Google for the answers during the test. We end up testing their Googling skills not their knowledge and understanding of the rules. “Open book” means that not only the book is open, but the whole Internet is potentially available to answer the questions. But I suppose we already rely on the honor system to keep people from just asking friendly TD’s for the answer, so perhaps the honor system is enough to make “the book” the only source which can be used for the test.
If the honor system does not suffice and we don’t want to facilitate Googling by holding discussions of the test questions where the answers can readily be found via Google, then we need to hold discussion of the test questions in a private forum.
Overall, probably the best thing is not to conduct the discussion in the “Chess Tournaments” forum but somewhere else which won’t be so easy to find later on.
Brian, you are so interested in everything TD—Time to learn what it is all about and become one yourself.
Rob Jones