This is a problem that has already occurred in another arena: competitive games! Here, too, the goal is to assign each player a global “rank” on the basis of a series of 1 vs. 1 comparisons. The difficulty, of course, is that the comparisons are not transitive (I take “subjective” to mean “provided by a human being” in your question). Kasparov beats Fischer beats (don’t know another chess player!) Bob beats Kasparov, potentially.
This renders useless algorithms that rely on transitivity (i.e. a > b and b > c => a > c) as you end up with (likely) a highly cyclic graph.
Several rating systems have been devised to tackle this problem.
The most well-known system is probably the Elo algorithm/score for competitive chess players. Its descendants (for instance, the Glicko rating system) are more sophisticated and take into account statistical properties of the win/loss record—in other words, how reliable is a rating? This is similar to your idea of weighting more heavily records with more “games” played. Glicko also forms the basis for the TrueSkill system used on Xbox Live for multiplayer video games.