We think we’ve figured out why some uploads fail, and it is definitely a problem with the files that certain versions of pairing programs are generating, but I’m not sure what the fix for that will be and I don’t want to get all geeky trying to explain the bits and bytes behind things yet.
I’ll check with the developers and see if they have any workarounds in mind.
I think the login bug will fix half of the problem, now we just need to figure out how to fix the other half.
I think you’d also try contacting those pairing programs and letting them know the issues so they can potentially also work on a solution?
There are clearly some old versions of pairing programs still in use by TDs, which understandably are no longer being updated by the software makers who have newer versions to keep up to date. Folks might not like it but that’s the way of the software world.
I also don’t believe it’s up to US Chess to keep backwards compatibility in this area, although I can’t believe the testing that was done didn’t highlight this particular issue.
Part of the problem is in addition to WinTD and SwissSys there are another half-dozen or so lesser-known pairing programs preparing DBF files and a few programmers who wrote their own code, so we don’t really have a list of developers to contact about this. Writing about it here is probably our best way of keeping people informed about them. Contacting the TDs who are submitting those files (some 500 of them in the last year or so) and asking them if they’re using a 3rd party pairing program or wrote their own code is on my list of things to see if I can do.
I’ll see if I can describe the issue in less techy terms. DBF files have a feature called a memo file which is usually used for large scale free-form text or stuff that just doesn’t easily fit into a character, numeric or logical data field type. The 2C format, which was never officially released so it didn’t go through a step of making sure it met the DBF standards, says to use a character field in the header file (thexport.dbf) for the additional TDs, a field called H_OTHER_TD, Some of the files we are getting are using a memo file instead, which creates a problem because we have no way of uploading a memo file. I’m not sure if these programs are even creating one, but the TDs wouldn’t know to upload it and there’s no place to put in its name.
So we basically have two choices, neither of them great. One is to tell an unknown number of TDs (probably a couple dozen of them) that they need a different version of the pairing program, and that’s assuming the pairing program author even has a version that doesn’t use a memo file for that field.
The other is to to have the Leago programmers kludge their parsing program to convince it not to throw an error because of a missing memo file. That’s probably outside of the scope of their current contract, so they may want to bill us for that time.
I don’t know which approach is more practical, but I’ll probably be having a discussion with the Leago team about it today.
Not having a list of who is creating DBF files is part of the reason I posted a note a few days ago asking programmers to contact me if they want to use the internal Leago APIs,so I have a list of people who’ve asked about it rather than just have programmers find it on their own. And the other issue with these APIs is they were developed for Leago’s internal use, so they may change over time and Leago doesn’t have the resources to be answering questions from dozens of programmers about API issues, at least not without billing US Chess for the time. So what I’ve been telling people who ask about the APIs is that they’re there, how to find the documentation, such as it is, for them, but cautioning them that they could change and there is no support available.
If, at some point, we decide to create external APIs that are not changing and for which there is both proper documentation and some way of getting support on them, that’s likely to be something Leago bills us for as it is out of scope in their current contract. So we might have to have a fee for using those APIS so we have some offsetting revenue to support the cost of deveoping and supporting external APIs.
I see reports that at least OpenOffice Calc will replace any 255 length character fields with a memo file, so I suspect that some dbf utilities would automatically create a memo file for any 255 length character field…and I see that H_OTHER_TD is 255 characters long.
If Leago could just ignore a missing memo file and any associated other TD information, that certainly sounds like a good solution; after all, with no way to upload it, the new system wouldn’t be losing any information over the old.
and a few programmers who wrote their own code
Creating the DBF files was one of the more straightforward parts of that process IME.
And that process was somewhat bungled for the 2C draft format. Mea culpa.
People have told us that they have had to sign in twice, once through the member dashboard and a second time on MUIR.
I think that’s the result of a check Leago put in yesterday to make sure people who are uploading files are really logged in as a TD, because some of the upload error problems were being caused by TDs who, as far as MUIR was concerned, weren’t signed in properly.
That’s a small price to pay for resolving some of the upload file issues, IMHO. But I did want to let people know it might happen so they don’t get concerned about it.
Do you have a list of the pairing programs that are currently known to have problems?
I think if you just posted them here, it would get back to the developers and they could correct the issue.
I’ve said before that it is amazing to me that we are still supporting DBF files. Investing meaningful amounts of time and money into this old tech, when there are actually pretty problematic bugs that need fixed, doesn’t make sense imho. Let the program developers fix the issues. But to do that, they have to know there are issues with their format.
Most of them should have a way to put out patches.
It’s a very simple format and supporting it is better than breaking every single program currently used.
sure, that was not the main point of my comment
Everything in the long run comes down to a cost-benefit issue. When we started requiring SafeSport certification, at an annual cost of about $20 a year, we got a lot of kickback from TDs who didn’t think they needed that training, didn’t think the training was worth doing, and, most likely, didn’t want to spend the $20. Some of those TDs have retired from running US Chess rated events, including, I believe, at least two formerly very active NTD’s.
So, do we want to tell some unknown number of TDs and affiliates, who are using older versions of pairing programs, that they need to spend $100 (or whatever) to get the latest and greatest version of their pairing program? How many of them will decide that running rated chess just isn’t worth it any more?
So from a cost-benefit standpoint, I think it is better to support a format that was old 30 years ago than to risk losing a significant number of active TDs and affiliates.
Should TDs and affiliates upgrade their pairing programs? Of course, and many do on a regular basis. The enhancements, bug fixes, and accommodations for rules changes is IMHO worth the upgrade fee by itself.
As someone who has been in the software business for over 50 years, a continuing revenue stream is necessary to survive. Upgrades have been one of those revenue streams, but they’re not always predictable or sufficient to support further development. That’s why a lot of software products have gone away from the ‘buy our program’ sales model to the ‘rent our program by the month or year’ sales model, which, I would note, is what ChessNut appears to be doing, too.
Again, I think most of the apps in use have a way to put out patches. And it might still be a good idea to document and share which applications are not compliant with the recommended use (even if you do invest in backward compatibility)
Personally I think the sensible thing to do is leave the ability to use the DBF files for some time but create a new more modern system too and then have a date that the DBF files will be phased out permanently.
I mean, we’ve moved to this new system without having an ability to use the old system and TDs have to adapt whether they like it or not.
The plan, subject to budgeting and other considerations, is to release an API that can be used by pairing programs to upload events directly into MUIR. I don’t have a timeline for this, and IMHO it would not be unreasonable once we do that to give the software authors time to implement the API and then give affiliates and TDs some time, like 6 months or a year, to upgrade their programs before we stop accepting the 3 DBF files.
I looked at over 500 uploaded files that used the memo file method, and with one exception they all say they’re ‘SWISSSYS’, but most of the uploads from SwissSys give a version number, like SWISSSYS10, so that self-reported information may not be accurate, and there are a couple of other sketchy things about those files that I don’t want to give specifics on to avoid embarrassing or upsetting anyone.
So they MAY be from SwissSys, but if so I don’t know how old a version of the program, and because of where I reaped these files from there’s no easy way to tie them to an event to say what the affiliate or TD was. (The information about the affiliate and chief TD in the files is part of what’s sketchy.)
Yes, I think you’re correct about the login. When I went to upload my first tournament, I encountered the error. When I looked at the instruction doc from USChess I saw that it said I would see login info in the top right, but I didn’t, so I suspected I wasn’t logged in correctly. The way I got to the upload link was from a link that took my directly to the Portal page, bypassing the main page. Once I went back to the main page and clicked on Portal to get to the Portal page, my tournaments uploaded fine.
Where was the link direct to the portal page, that should probably be changed.
Mike Nolan
I suspect I got it from Facebook. The good news is that now when trying to access the portal page directly it requires a login if not already logged in, and if I am logged in on another tab, it works fine. Just tested.