I don't know that I like the idea of prominently displayed RR indicators. I like them being squirrelled away on the profile, since I don't want to create RR "classism".

(also, you forgot negative RR :) )

I wonder if there should be some sort of weighting for turn length, on the assumption that missing shorter turns is more likely and perhaps should be considered less serious an issue?

That would allow combining the decay with that too - NMRing once in a 5 minute game is a bit more likely and less of a concern than NMRing in a 3 day game.

Why are we getting rid of the online status?

Also, a subtle yet important point: NMR can be a legitimate diplomatic tactic.

abgemacht, to have things clear, I'm not arguing the current system is perfect. Yeah, I agree, these percentages doesn't always seem really meaningful. Nor do I object to making recent achievements more important than those long ago. However, as I wrote earlier, I don't think it would be a good idea to exclude past CDs entirely as if they have never happened (with "entirely" and "never" emphasized).
Those who have ever had a CD should be able to improve their reliability rating to something close to a perfect score, but not to perfection itself (with "close to" emphasized).

A minor recommendation: floor the reliability rating. E.g. 99.9\%, which is clearly less than 100\%, ought to be rounded down to 99\%, not to 100\%.

Something else, 100\% reliability is more impressive if it's achieved by someone who has completed many (e.g. 100) games, than by someone who's new and has only finished very few (e.g. 1) game(s). Ideally neither new nor old players should be favoured.
Perhaps live games should be treated (weighted) differently than non-live games.

Nescio, I've already agreed with you. 100% should be reserved for someone who has never CD'd.

abge, is there a 100 club that I can join then?

Abge is correct, and I've witnessed it first had. An NMR can be used as a diplomatic tactic.

abgemacht, yes, I know, I don't disagree with you either; I just wanted to make it clear I'm not opposed to improving the current system, provided 100\% is meaningful.

Espi: http://webdiplomacy.net/board.php?gameID=166032

It seems that the core cause of many CD's is newer players not realizing that CD's are frowned upon and will affect their ability to enter future games. A while back I read a suggestion of posting a message right at the top of every game page, such as "LEAVING A GAME OR NOT ENTERING MOVES WILL HURT YOUR RELIABILITY RATING AND NEGATIVELY AFFECT YOUR ABILITY TO JOIN FUTURE GAMES!" I think a message like this could greatly reduce the number of CD's and NMR's.

I like the suggestions of weighting more recent CD's more heavily. I also like @abgemacht's suggestion to use percentile's instead of percentages, since it may be harder to apply a relative perspective to a straight percentage number.

This is how I got my CDs^^

There was a boy at school who had a 100% attendance record. He was a complete tit.

We should have a 100 club.

Why don't we approach this as an empirical question? We have *a lot* of data from all games on the site. So we can ask how well you can predict whether a player is going to NMR or CD on a particular game, knowing the player's history, and we can ask what function (exponential decay or some other weighing function) does the best job. I would argue that the best possible estimate for reliability rating is that function.

I have some experience with this type of modeling in my own work, so I'm happy to do it if someone can provide the data. It's essentially a fancy regression analysis.

Bas: That would be lovely. I think ATC has offered to do data dumps before, send him a PM I think. :-) I hope you can find a function that isn't too expensive to run - if we have to run a query over the player's entire turn set every time that would be ugly. :-)

I PM'd him.

I will start by looking at exponentially weighted averages, since those are easy to compute online. I'll probably try arbitrary weights next, but my hope is that the solution with arbitrary weights will look very similar to an exponential weighting. Or maybe there's something else I can't think about at the top of my head. But I'll keep computational cost in mind.

Awesome. I just took a look at the code, the current thing is entirely done inside an SQL UPDATE statement. It does look at all NMRs (and CDs) by looking them up, but it just counts them. There is also a per user phaseCount - which is simply a stored number, the query doesn't go looking at all of the user's turns ever. It might be able to (I haven't verified that), but I suspect that time based queries are going to get DB expensive, which might be a problem.

Currently CDs vs Games one part of the rating, and NMRs vs Phase Count is another part of the rating.

If anyone wants to look at the code:

https://github.com/kestasjk/webDiplomacy/blob/master/gamemaster/gamemaster.php

const RELIABILITY_QUERY = ...

Is the SQL statement in question. It's run for all users in a game when a turn processes. :-)

I love me a good old fashioned database, but running statistical queries are *not* their strength. :-/

Do resignations count as CDs?

A resign is just a country that ended a game in CD without being eliminated.

CD == didn't check the game for two phases (only counting phases where that player had orders)
NMR == allowed the game to hit the deadline without any orders saved (doesn't count games with Wait-For-Orders set)

For game results:

Defeated == all centres and units taken/destroyed (regardless of whether the power was in CD)
Resigned == the game ended in a draw or win with this player in CD

thorfi: I can't imagine a reasonable RR system that couldn't be easily implemented on top of the webdip DB. Even though that query isn't a fast query, it's certainly fast enough - it can compute RR for *all* 73,000 players in just over a second (and we only run RR for all players if the ratings need to be recalculated due to a bug or change in formula).

Sure, the DB probably isn't the best place for doing complex analysis to design a rating system, but it makes sense for that to be done offline anyway.

A bigger hurdle is that we want whatever system replaces this one to be easy to understand. So any complex rating system would have to be *much* better than a naive system in order to pay the understandability cost.

It's clear that the current system needs to be replaced though - it's not easily understandable, and without a time cutoff or decay it's not very accurate.

This is WebDipistan. No past sins can EVER be forgiven, because the Overlords are Infallible.

Fuck off, ssoren.

I still love you! ;)

I wouldn't be surprised if the data showed significant correlation between a player's probability to NMR now and his record infinitely far into the past. Some people care more about the game/are online more often/are more reliable people in general. In that case, the RR would have a non-decaying component. However, I without looking at the data this is all pure speculation.

I am all about forgiveness. A vote for ATC is a vote for weblibtardia.

That F'ing +1 whore^^^

ATC: Yeah, *that* query is fast. :-) It's only doing a couple of COUNT()s and fishing a small amount of stuff out of the user table.

I can imagine many queries that would very much not be fast.

And if you need to do it outside of SQL, especially if you need to fetch more than just a few rows per user and then try to process it in the CGI, that's going to start getting very slow.

But yes, if whatever algorithm turns out to be good can be done as a single SQL update statement, it will likely be fine. I'm happy to help with that if bas can let us know what's good. I've definitely built some horribly large-but-performs-fine queries in my time. :-)

@basvanospheusden It occurs to me that being able to present some kind of estimate error margin on the RR value would be cool. :-)

^ (Although that would probably just add to the confusion.)

Whatever we come up with, it is very likely that it can be easily implemented in the database since game data moves forwards one game/phase at a time (and very occasionally backwards in the case of a cancel).

I'd also argue that if there WERE any rating systems that were too complex to build on top of our small database, then they wouldn't fit the simplicity requirement.

I'd appreciate help with the implementation. I have no expertise with queries in SQL and PHP, in fact I don't even know what those words mean ;)

If we treat RR like an estimation problem, we'll definitely have error bars, or even a complete posterior distribution. We might not want to represent those on people's profile, but we could take them into account somehow.