Principles for a Computer-based Ranking System

In the last post, I implied that a well-designed, simple, and transparent computer program would be superior to the existing human-expert system (polls), when selecting teams to compete in a playoff.  Let’s consider what I mean by each of the three characteristics.  By “well-designed”, I mean a system that evaluates and ranks teams based on available data, and which mimics how human evaluators would rate the teams if the evaluators were truly unbiased (i.e., blind to the hype and history surrounding a team, and how others have recently rated it).  By “simple”, I refer to a system that does not require complicated calculations (like derivatives or logarithms), or a large set of data points for each team, or ambiguous or potentially misleading figures (like attendance, turnover ratio, or total yards gained).  Instead, a simple system would use a small set of essential data points from each game, and stick to basic math operations, such as adding, multiplying, and dividing.  It is important to note that a simple system would still require a large number of calculations because of the number of teams and games involved (but to a computer which can do repetitive calculations with 100% accuracy, that is simple).  Finally, a “transparent” computer program would have the source code available to the public for scrutiny to be sure it met the principles of the proposed ranking system, and it would be documented in such a way that other programmers could write their own programs that meet the same principles, and get exactly the same results.

Then what principles make sense to rate teams for the purpose of selecting a few to participate in a national championship?  There are several strategies to choose from, such as calculating which teams are the strongest at the end of the season (sometimes referred to as a power rating), or determining which teams have the most potential to win their next game.  Another possibility would be teams that are most likely to have strong audience support.  It could be argued that all three of those strategies are used to select at-large college basketball teams for March Madness, and that in general those strategies have been effective.  However, it’s my opinion that in college football, the teams that most deserve to participate in a playoff are the ones who have achieved the most over the course of the season.  Why?  As mentioned before, the season is short relative to the number of teams involved, so it’s difficult to compare teams on a head-to-head basis.  More importantly, football is a game in which a team’s performance, compared to its potential, can vary greatly from one game to the next.  Also, opponent strengths can vary significantly from week to week.  For example, one week a contender may face a team with a strong passing game, and the next it may face a team that specializes in the run, followed by a third team that finds success in a strong defense and a well-disciplined kicking game.  With one week to prepare, it can be difficult to make adjustments.  Therefore, it seems the fairest way to assess a team is to measure if it has been balanced and consistent over the length of the season.  By extension, then, an unfair way to measure a team is to expect it to be perfect for the entire season.

How should one measure achievement?  The most straightforward answer is: by winning games.  Winning a game is always an achievement, no matter how poor the opponent, or how close the score.  On the other hand, there is no achievement in losing a game, even if an underdog team comes close to knocking off a strong opponent.  But to distinguish great teams from good ones, one needs to measure more than the number of games won; one needs to assign a value to each win/achievement.  What criteria can be used to assign a value?  Certainly, a win against a strong team (one that few have defeated) should be worth more than a win against a weak team (such as one that has been defeated by many).  Secondly, a win by a large margin is better than a small margin, against opponents of essentially equal strength.  Finally, wins in the later part of the season, when all teams have matured, are more important than wins in the early part of the season.  In summary, then, the ideal playoff-worthy team will win a lot of games against strong teams by a large margin, with the best wins coming late in the season.

The three listed criteria are described in general terms so far; specific details are necessary to assign values.  For now, it is worth exploring what criteria should not be used to assign value.  The attendance figures, television appearance, home field advantage, or competency of the officiating crew do not seem valid tools for measuring how good the achievement is.  Likewise, injuries or suspensions of key players, weather conditions, and “lucky breaks” should not be considered in calculating the value of a win.  While it is possible every one of those criteria will affect the outcome of a game, it would be nearly impossible to quantify the effects.  In the end, a truly great team, nearly all the time, will find a way to win despite any disadvantages or mishaps that occur along the way.  It is also worth noting here that in-game statistical criteria such as number of take-aways, 3^rd-down conversion rate, time of possession, or even yardage gained should not be used to help assign value to a win, since it is certainly possible to win in all or nearly all of the statistical categories, and still lose the game.  Coaches design their game plans around how to score more points than the opponent, not how to beat the opponent in statistical categories.  While it may be interesting after the game to discuss how amazing it was that Team A outgained the opponent 500 yards to 195 yards, or that Team B had a +3 take-away margin, the bottom line is, who won?  And, how close was the final score?

Since the last post, there have been five weeks of games.  Some teams ranked high at the beginning of the season are undefeated and continue to rank high, but others have dropped out after losing once or twice.  Still others have risen dramatically, and then have dropped with an unexpected loss.  I'll discuss some specifics in the next post.