Has anybody, especially the USCF TD Committee, created a test question bank for TD certification questions? It would be good if such existed and the TD certification tests could be administered on-line, open-book but with a time-limit. Such a question bank would also be good for self-assessment.
For example, I like what Chris Bird has done at chesstournamentservices.com/2010 … questions/ . But, not to look a gift-horse in the mouth, there is only one 20-question quiz on Chris’ site. I finished the test wanting more questions.
QTI v1.2 (en.wikipedia.org/wik/QTI) is the standard format for test question banks. Many major Learning Management Systems, including open source options such as the more recent versions of Moodle, provide a module for administering tests which can draw questions from QTI question banks. And other LMS’s which use proprietary formats for the question banks can covert QTI question banks to those formats.
Chris’ quiz consisted of all multiple-choice questions, which are easy to structure under QTI. A lot of QTI question banks are available, including some in the public domain. How about the USCF putting together a TD certification question bank in QTI format? I would guess that the answer is “We don’t have time”. But I would also suppose that the various incarnations of the TD certification exams are drawn from an informal “question bank”, in the form of a document or file on somebody’s computer. If whoever has that could supply me with that file or files, I would be willing to put the questions into QTI format and provide the question bank back to the USCF.
A great idea but perhaps we should first get the pool of questions together from the TD community which would be more plentiful and rich since it would include from the field questions, vet them out (over in the open section of the forums and not here) and then present an actionable plan.
It’s not like the office has the bandwidth at this time to put this together, nor is there a volunteer infrastructure in place so for the time being they have to be handled grass roots.
I am not suggesting the office do it. I would be willing to do it if the questions used on the TD exams are available in some electronic format. That would be a start, though I think your ideas of letting others add to the list of questions and vetting them on the Chess Tournaments forum are good ideas also.
I don’t think we should use the questions from the TD exams. Those are for the exams themselves. A separate pool of questions can be devised that meets the same purpose and I think that would be a great idea to do.
If you’re talking about his being for training purposes, I fully agree. We’re doing something similar for FIDE Arbiters along with recorded training programs and as we complete vetting those questions out I’d be happy to share them with this type of project.
If we’re talking about for testing purposes for certifications I think it’s quite appropriate to move down that path way to automate the majority of the testing needs and lighten the load on the office. I say majority because if I understand correctly the ANTD and NTD tests are fully essay based. However the majority if not all (I can’t remember anymore) of the Local and Senior tests are multiple choice, what little there is essay based can just be modified to multiple choice. Use an automated system (no need to create it ourselves, there are services that the rest of the world uses to do this that are inexpensive) to lessen the load and move forward. It could even be done that perhaps prior to receive any essay based exam which takes more personal time to grade (and regrade it required) a multiple choice type pre-cursor is provided. Can’t pass it - you don’t get the written version.
I was thinking of this for both self-assessment and certification testing. By the way, with a QTI question bank, the USCF wouldn’t have to do any development to move to on-line certification tests. There are quite a lot of websites for on-line testing where you just upload your questions, including in QTI format. You could probably even do it with one of the leaders in the survey/polling/testing area, surveymonkey.com, though that is targetted more for online surveys. SurveyMonkey supports QTI, among other question formats.
This is one that I’ve used in the past - articulate.com) for building training sessions and testing. It allows formats for both uploading to an LMS or for publishing directly to the web (yours or theirs).
SurveyMonkey is good for surveys but not much else, but I use them as well. They serve a purpose.
There are also LMS systems which are add-ins for Joomla that can be targeted and used.
Back in 2007 when I was working on the Joomla site for uschess.org, I was exploring developing a QTI Joomla module for a prospective client, a startup. There was no QTI module for Joomla at that time. In fact, this was one of the reasons I got interested in the USCF Joomla project. At the time, I didn’t know Joomla, though I had experience with several other CMS’s. Indeed, the Learning Management System and Content Management System worlds, for some odd reason, were entirely separate (with a few marginal exceptions like ACS, one of the early CMS’s, being used for MIT’s .LRN LMS). This was so even though it seems obvious that an LMS is a CMS with some specialized add-ons, such as for grading, assignments, testing, curriculum tracking, etc. A lot of the features that one finds in LMS’s would be helpful in CMS’s and vice-versa.
The Joomla QTI project didn’t pan out, but it is interesting that in the meanwhile someone has built an LMS system on top of Joomla (joomlalms.com). Despite this, it seems the LMS and CMS worlds are still pretty much separate. Opportunity for someone, I reckon.
It has certainly been talked about, but the challenge is writing enough questions. I’ve been told (by people who develop question banks for their own courses that use online testing) that we’d need at least 400 of them, and more would be better.
One of the first things to do would be to categorize the testing areas to create an outline. In general 40-50 questions per testing area is standard depending on how wide the topic area is with rotating through 10 questions per area.
Again there is a difference between the official online testing bank of questions and a study bank of questions. You don’t mix and match.
How many versions of the Local and Senior TD exams are there now and how many questions per exam?
Why? As long as there are many more questions than are going to appear on the tests, I don’t see the problem in selecting the “test” from the same question bank as used for “study” and “self-assessment”. If the test is closed-book, or even just timed, and the test subject cannot reasonably predict which of the “study questions” are going to appear on any particular instance of the test, the subject’s performance on the test will be highly correlated with his knowledge of the answers to the study questions – in other words with his knowledge of the material. This is the point, no?
The testing community is split on the issue, in this case I tend to agree that there may not be much point to having the study questions be totally separate from the test questions, since the goal is to ensure familiarity with the rules and rulebook.
If we had a bank of 500 questions and someone memorized the answers to them all, that person might actually know more about the rulebook (and thus the rules) than many TDs do.
I believe there are three versions of the Local TD, but there may only be two versions of the Senior exam. I think they have either 40 or 50 questions.
I think the last time the tests were revised, it took quite a while to generate enough questions to make up three tests, including vetting the questions and answers to ensure they were correct.
The challenge is writing questions for which it is possible to come up with several wrong answers, ones that may seem reasonable but have a fundamental flaw in them which may not be obvious but points out a misunderstanding of the rules. (The difference between study questions and test questions is often that the study questions include a detailed explanation of why that answer is incorrect, so in some ways writing the study questions is harder than writing the test questions.)
I think Brian’s proposal is a terrible idea. There’s no way that our little TDCC is going to be able to write hundreds of questions like that, and there’s a lot to be said for testing with questions the examinee has not seen before.
And we would have to reuse the questions. The SAT has become much more expensive now that they have to release their questions publicly, which means in their case they cannot reuse any of them, or the test would become known as “a joke”. I would guess the quality of the test as a discriminator of “scholastic aptitude” (whatever that is) has gone down even so, as they can’t reuse the same great question multiple times, but write a new one that isn’t so great.
We would get people studying for a TD test by reading the test bank and ignoring the rulebook. That would likely be the most efficient way to pass but I don’t think it is the behavior we want to promote – for the same reason. Parts of the rulebook and aspects of individual rules that are not tested could be, and would be, ignored. TDs would not learn the skill of reading the rule or rules and applying it (them) reliably to actual situations where its application is clear.
I think anyone with the energy to try to memorize the entire test bank has more than enough time to read the rulebook.
As to the argument that memorizing answers to narrative questions doesn’t impart any knowledge, not everyone would agree with that.
I had a similar debate with my professor when I took my instructional methods course in grad school. A strict belief that rote memorization does not promote learning leads to the conclusion that only essay questions (as opposed to multiple choice or true/false) should be used. (Whether or not memorizing the question bank is rote memorization, like memorizing the times table, is a separate topic.)
I’ve also had that discussion with my wife more than once. She has secondary school teaching credentials and a masters in instructional technology, so she’s been through the issues more than I have.
Of course, understanding the rules or where to find them in the rulebook (and/or passing the test, depending on where you stand on the memorization issue) is just PART of the requirement for becoming a local TD.
I could make the argument that rote memorization of the rulebook doesn’t lead to understanding the rules, but naah!
Anyway, someone can become a club TD without having to pass any test at all, and as another thread pointed out, there really aren’t any mechanisms to prevent a club TD from directing most events. (A club TD wouldn’t be hired by the USCF office as the chief TD of a national event, but there’s nothing to prevent an affiliate from using a club TD for its events, regardless of how big they are or the size of the prize fund.)
But if they have the urge to memorize, they should be focusing it on the rulebook. And then, whether or not they memorized the rulebook, they could not pass the exam unless they could apply the rules to the hypotheticals on the exam.
Now leafy one, I don’t believe this this is a terrible idea. And Brian is not advocating that the TDCC goes and writes these questions themselves. This is where a volunteer infrastructure comes in play. The TDCC can solicit questions from TD’s around the country to develop the pool of questions. The TDCC can establish a sub-committee (I don’t know of any bylaws that would prevent this) of TD’s (I would have to say of NTDs) to sift through the questions, better structure them, and finalize them.
I do agree though that testing with questions that the examinee has not seen before is necessary. Picking from the same pool is asking for issues.
Yes that risk is there as well. However I will bet you that the majority of people taking the TD exams through the senior level have never read the rule book from cover to cover (even though I believe the Club TD application says that you havfe). So if I have the option that I’ve been able to questionize (is that a word?) the rulebook in hopes that the TD is being exposed to the information or they are only cracking open the rule book to pass the test. I’ll take the former over the latter.
If anyone here believes that the TD’s are reading the rulebook, you have a lot to learn about adult learners
This is a clear lack of understanding adult learning and the underlying psychology of adult learning. Your mind is in a different state when reading a book (and a dry one at that) and going through an action/response type scenario of testing taking. So the effect upon the person is quite different.
Memorizing doesn’t impart knowledge. It imparts information access. The use of the information in a scenario is knowledge based. That’s the whole lifecycle - Data, Information, Knowledge (ok we can also add Wisdom at the end but I’m not going there, nor the jokes of the Chief Knowledge Officer or the Chief Wisdom Officer).
And 10+ years of teaching at the undergraduate and graduate level, both online and on-ground, some of which were classes aimed solely for the purpose of professional certification, tells me that rote memorization provides the ability to information dump and not solve problems. You get people with the certification but don’t know boo. The different testing modalities all have their positive and negative attributes, however the question that must be answered is ‘what is the goal’ - for the exams up to and including senior, it’s very little to do with situational analysis and more ‘knowing the rules’. Not much of those exams are talking about application.
What we are doing now is ‘hoping’ they are reading the rule book and then doing an information dump by circling the answers. I personally think there really is little hope there. Creating a study pool, separate from the testing pool, will give us more ‘hope’ that if they are going through the entire pool and memorizing, they are doing more than they would have done otherwise and we’re in the right direction.