What about computer-based evaluation systems?

Instead of a human-expert evaluation system, let’s consider if a computer-based evaluation system could be employed in a fair way to pick the top 4 teams to participate in a post-season playoff.  The BCS committee has been using computer-based systems as part of the selection process since the inception of the BCS process, but they are not held in high esteem by the committee.  Out of six computer rankings, the BCS system averages the results from the middle four for each team, then averages that result with two human-expert polls (the USA Today Coaches Poll and the Harris Interactive Poll) to determine the overall ranking.  Thus, each computer-based system has about 6% influence in the final results, while each human-expert system has about 33% influence.  The full BCS selection procedures can be found at this website: BCS Selection Procedures

Currently, there are 6 different computer-based methodologies in use (Anderson & Hester, Richard Billingsley, Colley Matrix, Kenneth Massey, Jeff Sagarin and Peter Wolfe).  All of them employ some method of calculating schedule strength, but by decree of the BCS committee, none of them employ margin of victory directly or indirectly in their calculations.  This website:  BCS Know How - Computer Rankings  summarizes each one of them in turn.  There are many other computer ranking systems developed, with their results available online.  Each ranking system emphasizes the available statistics differently, based on the philosophy of the developer.  Some are interested in a power ranking (how good is the team right now?); others are interested in potential (how will the team do in its next game?).  Still others are interested in results (how much has the team achieved so far this year?)  Some think that the location of games (home vs. away) is critical; some even think that the number of people in the stands is important.

One could argue that we should not rely much on computer-based ranking systems because computers are not emotional and therefore can’t fairly evaluate the “intangibles”.  But that is exactly the advantage that a computer-based system has, if designed adequately – it can provide an unbiased assessment of the results on the field, because it can repeatedly and accurately evaluate the data without being distracted by emotion.  One could also argue that we should not rely on computer-based ranking systems because computer programs have the programmer’s bias built in, and inevitably have bugs in them, which will lead to bad results.  While this is a possibility, a well-designed, simple, and transparent computer program would certainly be less likely to provide undesirable or controversial results than depending on a human-expert system which makes no attempt to correct or prevent a known source of bias.

Next: how to design a computer-based ranking system that mimics “blind” evaluators.