Reliability Ratings

[RoganJosh]

As I understand it, the discussion was about making RR a better indicator of someones true reliability.

Yes, but the premises of the discussion was that a .66 RR for someone who went in CD 13 times in ~40 games, all within one year, is unfair. I think Swordsman was arguing that this is actually a fair rating. Hence, there is no problem.

[RoganJosh]

That said, the suggestions you quoted are all reasonable. But notice how they focus on the actual problem: newbies joining too many games and then CD'ing in them.

[Nephthys]

Just gonna quickly jump back in here, I'm not saying that my rating is unfair (although I do find it an inaccurate representation of my reliability) I was just saying that I feel as though my (naive) views when I first joined the site are costing me now. Most higher stakes (100+ bet) games are reliability locked and not only these games significantly more interesting and fun and I find that in the case of 95% of these I am unable to join.

What I ideally would like to see (while understanding that new changes take time, of which is limited) is;

1) A way that protects newer users from CDing, I personally really like the lock of games until X number of phases has been played (also a possible deterrent for multi's as it requires a little bit of extra effort).

2) A more clear algorithm for RR. I have no clue what it takes for me to rise by a certain percentage. Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)

[Claesar]

...

2) A more clear algorithm for RR. I have no clue what it takes for me to rise by a certain percentage. Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)

I think I've seen the formula before, but I can't remember it. Jmo, Ghug or ATC should be able to provide it, once they read this thread.

[ChippeRock]

In my opinion, after multiple CDs, it make sense that you would have to establish yourself over many games before your rating would change. Even one CD can be a terrible experience for your fellow players. The passage of time does not establish that a player is reliable.

I had 40 some CDs... And after playing 100+ games with a near perfect record I was only able to raise it 10 percentage points. I feel like you should be able to advance higher in RR quicker than this.

[Foxcastle]

Perhaps there should be some sort of rehabilitation program for players who are serious about fixing their RR. Something like: after you've played consistently without an NMR/CD for a period of time (a month? more? X number of games?), you can apply to the mods, and for every X consecutive turns you go without NMR'ing after applying, they'll erase one NRM/CD from your record. It doesn't require any coding, and it's some effort for the mods, but it's not that hard to check, especially if you make the players compile the documentation of their performance.

[Claesar]

Actually, I'm not sure we can erase NMRs.. If we can't, it would require a lot of coding. Perhaps admins can, though.

[swordsman3003]

I had 40 some CDs... And after playing 100+ games with a near perfect record I was only able to raise it 10 percentage points. I feel like you should be able to advance higher in RR quicker than this.

You went into CD in almost 1/3 of all the games you have played...40 CDs is a terrifying number. That represents maybe 200+ bad experiences you caused for your fellow players, which is equivalent to a player's-entire-webDip-career's worth of damage (I've been a member of this website for years and have barely played over 300 games).

You dug yourself deep, deep in a hole my friend, I presume because you didn't value your RR for a long time.

But your RR is more than just your CDs; it also accounts for your NMRs. You have NMR'd as recently as games that are still going on right now as I write this post. I think a reliability rating of 72% is more than fair...

To get my RR to 99%, I reached a point where I actually had <1% of my games be CDs (3/300+).

[PRINCE WILLIAM]

Certainly, it is not the same a CD in a live game and the CD in a phased game of a day ie. In a live game an internet malfunction or an unexpected real-life need, from a call to the call of nature, can result in losing two turns. My two CDs were because of calls from clients. It shouldn't weight the same on the RR.
Can someone explain the mechanics of RR? My NMR to the whole is less than 1% and yet my RR is 97%. I had tried many calculation forms to find how it is computed but I failed to solve it.

[jmo1121109]

The actual query controlling RR is:

Code: Select all

UPDATE wD_Users u 
			SET u.cdCount = (SELECT COUNT(1) FROM wD_CivilDisorders c WHERE c.userID = u.id AND c.forcedByMod=0),
				u.nmrCount = (SELECT COUNT(1) FROM wD_NMRs n WHERE n.userID = u.id),
				u.gameCount = (
					SELECT COUNT(1) 
					FROM wD_Members m
					WHERE m.userID = u.id) + (
					SELECT COUNT(1) 
					FROM wD_CivilDisorders c LEFT JOIN wD_Members m ON c.gameID = m.gameID AND c.userID = m.userID AND c.countryID = m.countryID
					WHERE m.id IS NULL AND c.userID = u.id),
				u.cdTakenCount = (
					SELECT COUNT(1)
					FROM wD_Members ct
					INNER JOIN wD_CivilDisorders c ON c.gameID = ct.gameID AND c.countryID = ct.countryID AND NOT c.userID = ct.userID
					WHERE ct.userID = u.id AND c.turn = (
						SELECT MAX(sc.turn) 
						FROM wD_CivilDisorders sc 
						WHERE sc.gameID = c.gameID AND sc.countryID = c.countryID
					)
				),
				u.reliabilityRating = (POW( (
					(100 * ( 1.0 - ((cast(u.cdCount as signed) + u.deletedCDs) / (u.gameCount+1)) ))
				    +  (100 * (1.0 -   ((u.nmrCount)/(u.phaseCount+1))))
			    )/2 , 3)/10000)"

I'll try and find time this weekend to post the English friendly breakdown but NMR's are weighted to about 1/3 of a CD. CD's and NMR's are each counted. CD's forced by mods are ignored. NMR's and CD's for cancelled games are still counted.

[jmo1121109]

Surely at the very least this can be made public (as far as I know it isn't, may be wrong though)

As site code is public for anyone curious at https://github.com/kestasjk/webDiplomacy

By searching "NMR" there you can see all the code relating to NMR's. Not super easy if you aren't technical, but it is public. Code contributions are also taken. So if anyone with a little SQL skill wants to take a shot at writing a better formula I'd be happy to review it and get it live.

[Mercy]

Mentioning vDiplomacy is really becoming a meme of mine. Here we go again.

If you play at vDiplomacy and hover the mouse above your own reliability rating on your profile, you see the text:

''RR = ((noNMR + noCD)/2)^3"

This means:

"RR = 100 * ( ([# of phases played - # of NMRs]/[# of phases played] + [# of games played - # of CDs]/[# of games played])/2 )^3"

NMR's are weighted to about 1/3 of a CD.

Really? That is a difference between webdip and vdip then.

[jollycream]

As someone who joined today, I noticed when I had joined my first game that I straight away could join games with a reliability rating of 100%. It feels wrong to say the least, as it allows any newbie to join 5-15 games straight away without having proven any reliability whatsoever.

One way to fix it would perhaps be to have a minimum value on the divisor in the calculation of reliability? That way, newbies cannot come in and ruin too many new games without having proven themselves reliable first.

[Squigs44]

Yeah the code jmo posted isn't too hard to translate, and it basically looks like what mercy described vdip uses. The only thing that seems different is the +1 in the divisors. If anyone wants to know how many more games they need to boost their RR to a certain percentage, I'll be happy to calculate it for you.

[Squigs44]

For example on Nephthys:
*Disclaimer - I don't know what his NMR and phases are so this is just an estimate. It actually looks like his NMR count probably isn't too high compared to his CD's.

To raise your RR to 70% would take about 8 clean games
75% would take about 21 clean games
80% would take about 40 clean games
85% would take about 72 clean games
90% would take about 133 clean games

The way the formula is set up, the cubing power makes it pretty unforgiving with that many CD's to get into the higher (85%+) RRs.

[jmo1121109]

I'm an admin over on vDip too, Oli made a variety of changes and experiments pretty frequently with new ideas there. He's added a few (like limited the number of games people can join as new members), letting mods give forgiveness points, etc. Eventually some of those will be brought over but a few are too new to see how they work yet. And I'd like to only have to make one round of changes. That said, if anyone comes up with a creative formula change I'd be happy to look into it.

[bo_sox48]

Feel free to run these changes by the rest of the gang before you make any promises, vDip admin.

Correct me if I'm wrong, but haven't we done some sort of RR forgiveness in the past?

[SuperSteve]

I don't care about the "fairness" of ratings. I'm for a strict system that rewards reliability and punishes unreliability.

Still, it would be nice if the system weighed recent results more heavily than results in the distant past. Not for the benefit of the unreliable players but in the interest of accuracy.

[CptMike]

A few words about the 1v1 case :

I have been playing a lot of /live/ 1v1 games.

First point : it is really frustrating to see players just dropping the game, often without even leaving a message to inform you. I don't create them any more without requiring an RR of 85-90 %. But this does not prevent apparently fresh players to come (with 100 % RR) and behave that way...

Second point : when you create a live 1v1 or when you register to this, it does not start immediately. And a player "forgets" about this. I don't think it is done purposedly (even if it is very frustrating). I have experienced this a dozen of times and I have done so myself several times. I remember a game with Nealkochhar where we *both* forgot the games.

I'd suggest that the first behaviour is "punished" strongly a way or the other. You could afford to the remaining player (players is standard games) to triple the sentence of the drop of RR due to this ?

The second case on the other side is an accident. Maybe it should be weighted another way even if it is frustrating as well to the other player !

A 'resign' button may offer to the "bad player" a "clear way" to leave a game that is lost. But then I suggest 'resign' should be confirmed (to avoid accidents ...)

[ssorenn]

RR should not be given to any new player till x amount of game have been finished by the player. Then the aggregate of his moves from the X amount of games is complied giving him a RR score.
This stops newbs from joining games with RR requirement. Too many games, especially gunboat are ruined by new player who don’t know the ethics of finishing what they start. I think that was swordsmans point. Or part of it at least.

I will say for the record that i committed these transgressions when i first joined. This was the first real competitive site i ever joined online. But as a new player, i thought”it’s just a stupid game, who cares”. Only later to figure out it really matters to a lot of people.

So, having to earn a score may be better, and some type of testimonial when you join, explaining that these games take a long amounts time and people take them seriously.

What the X number is , up for debate.

instead of creating the algo to fit X in, why not just create another peramiter in the game creation field that allows for some to determine that X for themselves

ex. RR85%
X#games on site >20