Proposal for a new ranking model

[rommager]

Hello all,

I have been researching existing games for about the past week, and have been coming up with some new metrics for creating a new player ranking method.  I have identified several key areas in which players can be comparatively rated, and now I would like to share what I have found.

First off, let me explain how the ranking works right now.  It's a simple system - Basically, the more games you take first place in, the higher your ranking.  If you tie with other players on your number of wins, then your ratio is used as the tie-breaker.  If you tie both categories, which is extremely rare, then the sort order goes to the player with the highest score, etc, etc.

Using the current system, players who play most frequently are most rewarded.  While this is an important factor (no player should rank high with just one or two freak games), it does not reward players who play the game with the most skill.  You can theoretically win 26 games with miserable colonies rankings and still take a top 10 place (as of this writing).

Currently, there are 6 key areas on which the new proposed ranking model is built:
1) Average Player Score
2) Average Colony Score
3) Number of Games played
4) Average Game Ranking (whether you place 1st, 2nd, 3rd, or 4th)
5) Average Percent of Colony Score (similar to ranking - higher ranked players have a higher percentage of the total colony score)
6) Average Number of AI players you play with per game, also counting players who abandon as ai (higher percentiles means fewer ai players)

Each one of these factors are broken down into a percentile, so each player is ranked relative to all other players.  For instance, if you have an average score percentile of 95.6, then that means your average score is higher than the average score of 95.6% of all other ranked players.

The only percentile which is modified is the number of games played.  Due to the extreme high number of players who have played only 1 or 2 games, the minimum number of games played to rank in this percentile is 3.  Players who play only 1 or 2 games will be in the 0.0% percentile of number of games played, which limits the ability of players to rank high with just 1 or 2 games.

The final ranking is determined by averaging together your six percentiles in a weighted average to come up with the composite ranking percentile.

The proposed weights of these six factors are as follows:
average score = 3x weight
average colony score = 3x weight
number games played = 6x weight
average ranking = 2x weight
average percent of colony = 1x weight
average ai players ranking = 1x weight

Now, a quick note about the number of ai players: remember that you are always ranked relative to all other players.  We all have problems with players bailing from games, and the numbers show that those players in the low percentiles are players who consistently play with 2 ai players.  Playing consistently with too many ai players can drag your ranking down, but the weight of this is also only 1/16 of your overall ranking.  Please keep this in mind before complaining about other players abandoning and dragging rankings down.  Nearly everyone has played games where other players have quit before the end of the game, so long as you don't consistently play with 2 ai players, there is almost no penalty.

As far as which games are counted, any tournament game finished by at least one human player is counted in ranking, and only the final round 12 results are counted.  I have been contemplating disqualifying any tournament game that did not finish with at least 2 human players, but have not worked this into my calculations yet.

Anyway, this is your opportunity to voice your opinions.  Do you feel like there should be other metrics involved?  What are your thoughts on the weights of the player metrics?  Do you have more detailed questions?  Let me know!

Thanks!

[rommager]

I do have another thought on this.  The final ranking will also probably take abandonment ratio into the ranking as well.  That way, abandoning a game will be able to negatively affect player ranking.  Should abandonment negatively affect player ranking?  There is one note here - if all players abandon a game, then the game would not be ranked at all.  That way if a player drops out, there would be no penalty for all players quitting and restarting a new game, if desired.

[Paladinian]

Thoughts:

- Does your Number of Games played refer to the number of games Finished (so disregarding Abandoned) or the total joined (Finished + Abandoned)?

- Why is the Number of Games three times more powerful then the Game Ranking; namely, why is the number of games so much more important to determine one's rank then whether one *won* those games?

- Why do we care about Colony Score when determining rank?  Player Score, sure, but Colony Score doesn't really reflect on the player in particular.  It would make higher ranked players loathe to play with beginners, as they would bring their rank down due to poorer overall performance.  Now the Percent of Colony makes sense, as it still reflects overall prowess compared to one's competition.  But if we just use that, correspondingly experienced players would *only* want to play lower ranked, so that they dominate that much more.  I personally would leave Colony Score factors out of the equation.

- Very much approve of factoring in AI player quantity in determining the strength of the win.  

Edited for typos...

[C64 nostalgia]

I disagree with your proposed weights. First, I believe who wins a game is very important. Yes, it encourages a win-at-all-costs motive. Winning the game, most of the time, is the primary goal in my games. About 20-odd years ago when I first started playing, this is how I played with my brothers--this is my nostalgia. So, therefore I think the colony-related weights are too high. Additionally, I think the games played weight is way too high. I think it is significant how often you win. But as long as you have "enough games," I think the total games played in much less important. Furthermore, an area missing is something measuring 1st place wins and/or the ratio of 1st place wins over total games. Also, an area weighting overall rank by 1st place needs to be added.

So, my weighting would look more like this (compromising from your weights and taking out average player score; First place ratio is better than average player score. Plus, average player score is more about the colony than the player. To have a high player score usually means the colony did well. Two colony-based weights already exist.):

first place wins =1x
first place over total games =2x
overall ranking calculated by the above = 1x
average colony score = 1x weight
number games played = 0.5x weight
average ranking = 2x weight
average percent of colony = 1x weight
average AI players ranking = 2x weight

As far as AI's are concerned: Playing with an AI makes winning and high colony scores substantially easier. You know almost exactly what an AI will do, especially if you use them often (learning their behavior by watching them play). Any games starting with AI's should not be ranked. If players care about the love of the game, they shouldn't care if their games are ranked. So, if a player just wants to play immediately, they play with AI's. Without the game being ranked, their playing is about the love of the game. If a player cares about rank, they should be willing to wait for 4 human players. Consider it the cost for the privilege of ranking. Last without, at least, 2 human players at the conclusion of a game, that game should not be ranked. I understand people bail or what-not, but to keep playing when it's only you -- love of the game, not rank worthy.

Thanks for your consideration.

Your abandonment affecting ranking is okay with your all-abandon stipulation.


edit: added abandonment sentence

[rommager]

C64, I respect your thoughts a lot.  You are definitely a serious player, and your stats are top notch in all six categories.  I will create a percentile for the number of games won, and will play with your suggested weights.  The only thing I don't get right off is "overall ranking calculated by the above = 1x"

...
first place wins =1x
first place over total games =2x
overall ranking calculated by the above = 1x
...

Do you mean that we should create another metric for this?  Since we are talking about percentiles, it would basically be the same thing as counting the above metrics again, which would make their overall weights go from 1x/2x to something more like 1.5x/3x.  Did you mean to place more emphasis on the first two metrics?

Paladinian, I also respect your input as well.  Again, the serious players will have the most pull in these decisions.

Here's what we have so far:

• Number of wins is more important than number of games played
• Average AI players ranking is a good thing
• Colony Score should not be important
  Should we consider factoring colony score for players with a higher percentage of the colony score, or just dump that metric altogether?

Questions:

• Using percentiles is a more accurate way to be ranked relative to other players.  Are we in agreement about using percentiles in the key areas for ranking purposes?
• Are we in agreement that the rankings should also take player skill (scoring, percent of colony, etc) into account, and not just the number of wins?
• I think that if we keep number of wins that the number of games played will be a much less important stat.  In this case, do you think we should just remove the number of games played?

To do:

• Create a percentile for number of wins
• Create a percentile for wins ratio
• Create a percentile for abandons

Just to clarify, my original proposed weights were merely a starting point, so don't take them as final.  Thanks for your input guys!

[C64 nostalgia]

I must admit thinking about percentages of many percentages is confusing me. So, let me clarify another way.

Starting here:
Reset the rankings completely
Make games starting with AI's unranked
Make games concluding with less than 2 human players unranked

These metrics are most important to me:
Win/loss ratio (with some added value for high ratios with many total games)
Wins in games with good players
Individual game ranking (something like -- no value for fourth, most value for first; second and third in-between, respectively)

Additionally, some colony metrics should be included to temper the win-at-all-costs motive. Colony metrics would carry much less weight than the individual metrics.

Thinking some more... abandon rates should not affect rankings unless a history of consistent abuse emerges.

I hope this helps... my goal was nice and simple (for me, at least ).


[edited for formatting]

[Intergalactic Mole]

Correct me if I'm wrong, but in your proposed ranking system, technically you can both play each other until one of you reaches rank #1.

[piete]

Hi! Too tired to give a thorough read atm, just a few comments on C64's last post (that has supposedly already thought over the previous ones):

Make games concluding with less than 2 human players unranked

These metrics are most important to me:
Win/loss ratio (with some added value for high ratios with many total games)
Wins in games with good players
Individual game ranking (something like -- no value for fourth, most value for first; second and third in-between, respectively)

Additionally, some colony metrics should be included to temper the win-at-all-costs motive. Colony metrics would carry much less weight than the individual metrics.

I'm against not ranking a game finished with less than 2 human players, a bad sport could ruin the winner's game if this was a rule.

I used to play a league with my friends with a scoring 3-2-1-0 (or 2-1-0-(-1)), and a penalty for a low colony (under 60000), it was either -1 point to everybody or one point less than in a "good" finish. This would encourage to try one's best even if victory was out of reach. Maybe even more points to the winner, though. And leaving the game (either purposedly or accidentally) means 4th place...

[rommager]

Correct me if I'm wrong, but in your proposed ranking system, technically you can both play each other until one of you reaches rank #1.

Actually, one can do that today.  You and I can decide to play each other several games a day, as quickly as possible with no regard to scores, and take our turns winning to get ranks 1 and 2.  Going with percentiles and weighing the factors it ensures that players should strive to play skillfully (get the most points).

My original thought was going by sheer number of wins is not necessarily a benchmark of skillful playing.  It just means that you play the game a lot.  The fact that we're talking about ranking players on individual attributes and piling them together in a weighted average means that someone could not necessarily rank #1 by merely winning games or playing a high number of games.  It just means that your scores will not be pulled down as much by playing the highest number of games.

Take a look at the percentiles for number of games played (in the data I have):

1   0.0   (below floor)
2   0.0   (below floor)
3   26.0
4   42.6
5   54.6
6   63.1
7   69.8
8   73.2
9   75.9
10   78.7
...
47   99.0
50   99.2
59   99.4
60   99.6
63   99.8
65   100.0

You can see the more games you go up (or the higher number of samples you have), the more close together the percentiles get, which makes this score less and less important as a deciding factor for ranking, but it is very significant for those who have more samples in their data.

That is why the number of games in my initial proposal has such a high weight.  It is really just a limiting factor to ensure that the number of games to sample data means your average score is more consistent.

I originally didn't factor number of wins because that is a cut down version the more robust average game ranking.  Basically, if you took 1st place 10 more times than I did, that's like getting 10 more 1's averaged in to your average game ranking.  This is the same as percentage of the colony score.  If you are in 1st place, you naturally have the highest percentage.

[Paladinian]

I've thought more about the proposal and, while I understand the desire to rank by percentile, intuitively it isn't the most striaghtforward system for people to understand.  

I think the current system isn't bad... it is understandable enough that #of wins = highest rank, with other metrics for tie-breaking.  I do concur that it is disappointing that coming in second is the same as coming in last.

Personally, if I were designing the thing, this is what I would do... what are your thoughts?

 - Players are ranked by victory points: most points wins.  Victory points are earned by scoring higher in a game then another human player in the game.  So, in a 4 human game, winning gets you 3 points, second gets you 2, and third gets you 1.  In a 2 human, 2 AI game, the winner only gains 1 point.

 - If one wants to further increase the competitiveness at the loss of simplicity, one could add a few tweaks to the above:  Winning might always grant an additional +1 on top of the rest so as to increase the benefit of true victory.  You might also gain an additional point for every player you defeat who was ranked higher then you.  (So the #2 player beating the #1 player gets more benefit then the #1 player beating the #2).  

 - I would never take points away... the complaints to such events would overwhelm these boards without a doubt... though it would be tempting to put some manner of penalty on Abandons.  Again, the potential for complaint is too high, so I would favour leaving such situations out.

The relatively straightforward Victory Point method simplifies greatly any talk of "weighted averages", I think, while still giving people *some* progress from playing a good game but not quite winning.

The one issue is that you do get a situation where one who plays many games marginally can out-rank one who plays now and then but consistently wins.  I can certainly see how people would dislike that aspect.

Whad'ya think, rommager?

[Intergalactic Mole]

Actually, one can do that today.  You and I can decide to play each other several games a day, as quickly as possible with no regard to scores, and take our turns winning to get ranks 1 and 2.  Going with percentiles and weighing the factors it ensures that players should strive to play skillfully (get the most points).

Unless you penalize players for playing the same people over and over, the ranks have absolutely no credibility (including the current system).  And so changing the current system without fixing that problem is pointless.  I hardly consider some dude who had a private tournament between him and his buddy to be the best MULE player. LOL!  And to be honest, I don't feel like sifting through everyones games to find out who is legit and who isn't.  I can just assume that everyone at the top of the list simply doesn't deserve it.     I'm willing to bet that at least 2 or 3 of those players played with each other in a heck ton of games.

[rommager]

Unless you penalize players for playing the same people over and over, the ranks have absolutely no credibility (including the current system).  And so changing the current system without fixing that problem is pointless.  I hardly consider some dude who had a private tournament between him and his buddy to be the best MULE player. LOL!  And to be honest, I don't feel like sifting through everyones games to find out who is legit and who isn't.  I can just assume that everyone at the top of the list simply doesn't deserve it.     I'm willing to bet that at least 2 or 3 of those players played with each other in a heck ton of games.

Well, you also have to look at the fact that most of us play whoever is available at the time.  I end up playing several of the same guys over again.  It's not like I plan it that way, it just happens.  The sad fact is that this is still a small community of faithful players to a game that is relatively obscure.

I'm sure that there are buddies out there that play mainly each other over and over, but IMHO that's still better than playing the ai repeatedly to boost scores.

Paladinian,  I actually kind of like the idea of points, although it would actually be difficult to calculate a point for beating a higher ranked player on the historic games.  We simply don't have historical ranking data.  Still, it's going to be difficult to make everyone happy with a new ranking system.

I think in the end, if we're going to change the ranking system, then someone will just have to make a decision and roll with it, then perhaps present the candidate systems for a majority vote.

[Intergalactic Mole]

Well, you also have to look at the fact that most of us play whoever is available at the time.  I end up playing several of the same guys over again.  It's not like I plan it that way, it just happens.  The sad fact is that this is still a small community of faithful players to a game that is relatively obscure.

Good point. I guess what I'm trying to say is that probably no matter how the ranking system is revised, I will always consider it just bragging rights amongst a small group of players and not anything really factual or credible.

[Mega Byte]

What I think would help bring more people to the site, and motivate them to play is a daily/monthly/annual/all-time scoring system.  I don't really care what model is used, at least this way, you've got some chance to be "relevant".  I think High Score for game (Individual) and TEAM should also be listed this way.  Give lots of opportunities to stand out, even if you just started playing a day ago.  People like to be seen and recognized.  It's part of why to play.  If the only way you can ever get to number 1 is winning the most games, then anyone who didn't start playing in December 2009 is pretty much screwed at this point.

[piete]

What I think would help bring more people to the site, and motivate them to play is a daily/monthly/annual/all-time scoring system. 

If the only way you can ever get to number 1 is winning the most games, then anyone who didn't start playing in December 2009 is pretty much screwed at this point.

I second to weekly/monthly/quarterly/annual/all-time ranking (daily is maybe ott, not that many players must play more than 1-2 games per day)