Any update on this?
Yesterday, I played in the US Championship qualifier at Chess.com and faced a player with a bizarre rating card. His online ratings more than 400 points below OTB struck me as very odd.
USCF ID 12964843
Regular Rating 2283
Quick Rating 2120
Blitz Rating 2218
Online Regular 2268
Online Quick 1707 (p16)
Online Blitz 1685 (p11)
This player had never played online rated before July. His Online Blitz rating was initialized at the Weeramantry Blitz Invitational for players rated over 2200 Regular. Every player in this tournament was an OTB master. Somehow this player earned a 1669 (p8) provisional rating from scoring 2.0 out of 8 against 1 GM, 1 IM, 2 FM both rated over 2400 Regular, and 4 NMs. As a first order approximation, a score of 25% gives a performance rating about 200 points below the average rating of the opponents. I would have expected that to be around 2100. Mind you, six of these master rated opponents also ended up with an Online Blitz rating well under 2000.
This player played one more Online Blitz tournament and pumped his rating up to 1685 (p11).
Then he joined an Online Quick tournament and was assigned a starting rating of 1685 based on 10 games. After playing six games, his new Online Quick rating was 1707 (p16).
What happened? A national invitational blitz tournament full of known OTB rated masters (and most of them juniors) should not end with the majority of those players rated under 2000. No wonder so many juniors have online ratings many hundreds of points lower than their OTB counterparts. I don’t expect a 1-to-1 correlation, but I do not expect a substantial fraction of the rating pool to be severely underrated.
As many others have commented on various online forums, these “official” USCF ratings are a joke!
Michael Aigner
There are a number of issues with how ratings are initialized that bit us with those first on-line quick tournaments—the rating calculations were never designed to be robust to a large number of “first time” players in a rating system who, in fact, were quite experienced, but in other rating systems. One problem was that it too often picked the “wrong” rating as the best fit (an old one when more recent ones were available) and that even if the initial rating was reasonable, the use of N=0 initializations made the calculations of the first tournament ratings far too volatile. The proposed fix isn’t particularly simple (it takes a stale-date weighted average of the available ratings, and gives a higher initial weight to them), but will be able to handle not just the transition from OTB to on-line, but also the future transition from on-line back to OTB, when we will doubtless have quite a few players who have only on-line ratings.
The plan is to implement this retroactively for all on-line ratings as there was also an error in the K factors going back to the introduction of on-line blitz.
The ratings initialization changes Tom is referring to are not simple ones, they will require significant testing on our testbed server. (Finding or generating test cases to check all aspects of the new code will take time.)
The online ratings changes will be retroactive to 2014 when we started rating online events and will include the correction to the K computation that others have noted, the OTB rating initialization changes will have an effective start date that has not yet been set, it will probably not be retroactive at all.
I’m having some issues getting to that server on a reliable basis (it keeps shutting down), and I’m also having some vision issues that limit the amount of time I can stare at a computer screen in a day. (Surgery is scheduled for late January.) Recent staff changes have had a priority on my limited screen time.
I still hope to get the changes made and testing completed before the January rating supplement is generated on Dec 16th, but if there are coding or testing complications that might not happen.
What is wrong about the K factors in onine chess ?? What would you like to see different ?
They’re wrong in the sense that they aren’t what they’re supposed to be (as a function of rating) and are lower than they are for the OTB ratings.
There’s a section of the ratings programming code where it figures out K based (primarily) on a player’s rating. Then that K factor may be tweaked for an event (high rated players in dual-rated events, for example, or the old half-K events).
Due to a bug in the code it uses the wrong K (too small) for all online events, so it affects all online ratings once someone gets to the regular ratings formula (ie, 9 or more games)
In tests, fixing this coding error will raise many, but not all, online ratings.
However, the Ratings Committee also wants to make a change to the way ratings are initialized from other ratings information. This will be fully retroactive back to late 2014 for online ratings but will not be retroactive for OTB ratings. (The date it will be effective will be announced once it has been decided upon.)
I do not know what impact the change to how ratings are initialized will have on online ratings. I’m working to set up a test of this on our testbed server.
I was trying to connect to it last night to copy over our current database, I was getting disconnected every 20-30 minutes. This may be an issue with my home networks (I have 2 of them), but the server in Crossville may also be contribution to these disconnects.
I think I finally got the current database installed on the testbed server late last night, so now I can work on making and testing the changes to the initialization procedures.
This test will need to check that all the possible combinations of ratings information that will make up the new initialization procedure is working, and there are probably dozens of combinations to check. Some of them may not be ones that come up very often, so testing them either requires finding or forcing a lot of possible test cases or just testing what we can find and fixing any problems from unusual combinations of ratings information as they are reported.
The combination of changing how online ratings are initialized and fixing the error in K means that everybody’s online ratings are likely to change, some of them by 100 or more points, once the updated code is activated.
Any update on this?
I do not expect to be able to work on this or other non-essential matters until at least late February due to personal reasons.
Bummer. Given the ongoing pandemic, I am surprised that “official” online ratings incorrect by several hundreds of points is not a bigger issue for US Chess. My guess is that most members are unaware and blindly trust MSA.
Michael Aigner
I don’t know that the ratings are ‘wrong’, per se, because if you do better than expected, your rating still goes up, and if you do worse than expected, your rating still goes down.
Relative ratings are what are meaningful, not absolute numbers. A 1600 player is still, generally speaking, a lot stronger than a 1200 player.
Comparisons between ratings systems are always going to be problematical. If they weren’t, if a rating in system A meant EXACTLY the same thing as a rating in system B, then there’s no reason to have multiple systems at all!
I was thinking the same thing but I knew if I mentioned this I would have gotten ridiculed for it.
When the initialization step gives different results for different players, and those numbers vary by many hundreds of points, then your comparison flies out the window as well. I do not expect to be rated 500 points higher in online blitz than a teenage master who is rated nearly 100 points above me at over-the-board classical chess. Even if I score 50% against him in rated online blitz (unlikely), then my rating takes a hit.
Considering that some organizers now use online ratings to determine eligibility for prize money (bad idea IMHO), I would have expected some people to complain about the incorrect K factors as well, especially concerning rapidly improving players.
Michael Aigner
I think you’re mixing apples and oranges. The new initialization procedures have nothing to do with the K factor problem. I have no idea what impact the revised initialized procedures will have, the K factor issue slows down the rate at which someone’s rating goes up or down, tests on a backup server conducted last year suggested most established players would see a modest increase in their online ratings, probably less than 100 points on average.
I hope to be able to work on ratings issues again by the end of February.
I simply illustrated two issues with the current implementation of online ratings that have been known for many months. As I stated previously, I am surprised the office has not received a wave of complaints. Most people are blissfully unaware.
Michael Aigner
Yes. Even when I started this thread, several people seemed to immediately assume that the ratings estimator must be incorrect and MSA must be correct.
Two things will happen, eventually. One is that the programming error in the computation of K will be fixed. A test of this last year suggested the average established online rating would go up around 100 points, but the relative difference between players will remain about the same, so it won’t have a major impact on expected score.
At that point the Ratings Estimator should be more consistent with actual online ratings computations for established players. (The Ratings Estimator has never been very good at estimating ratings for unrated or provisionally rated players, because it just doesn’t have the information needed.)
The other change is that the initialization process for previously unrated players will be completely rewritten. For online events this will be retroactive to the start of online ratings, for OTB ratings this will not be retroactive, the effective date has not yet been set.
Perhaps the Ratings Committee has some ideas as to how this will change ratings, but I do not.
It is worth pointing out that this does not directly deal with systemic divergences between ratings systems, which have multiple causes including inactivity in one ratings system but not another. Because a blending of other available ratings information will be used to initialize someone’s rating when they begin playing events rated in each ratings system, it might have some minor effect on inter-system issues.
Does anyone have a link to the Ratings Estimator? Both the link I bookmarked and the link at the bottom of the homepage are broken. I realize the numbers are probably flawed due to the K factor issue.
Michael Aigner
Testing is in progress for the ratings initialization changes given in the September 2, 2020, proposal.
The coding error that has affected all online events since 2014 will also be corrected.
The effective date of these programming changes (which will result in a retroactive change affecting all online events) is not yet known. The effective date of the revised initialization changes for OTB events has not yet been determined, either.