Last Player Picked

Rudy recently did a study to see how the RazzBall Point Shares system stacked up against the existing competition. In setting up my own fantasy valuation system, the Price Guide, I was of course curious to see how it would do as well.

So, to compare the valuation systems, I decided to do a Retro-Draft for the 2008 season. That is, a draft for the 2008 season, right now, after the season. In this draft, each of the systems gets a team that will use that system’s rankings as their exclusive draft sheet. When the draft is over, I can figure up the league standings and instantly tell which team won.

The systems I considered were:

Last Player Picked’s Price Guide
ESPN’s Player Rater
BaseballMonster’s rankings
RotoTimes’s Player Rater
RazzBall’s Point Shares

All of the teams have the benefit of 20/20 hindsight. Everyone knows the exact stats each player ended up with, and so the only difference should be their ranking methodology.

The league I’m drafting for is a standard rotisserie mixed league: 12 teams, 14 hitters, and 9 pitchers. A hitter has to play 20 games at a position to qualify. I consider traditional rotisserie to be neutral ground that shouldn’t favor any system.

With 12 openings and 5 competitors, one problem is figuring out how to handle the draft order. The ideal situation might be simulating the draft thousands of times, each time picking the teams at random. Eventually it would begin to become apparent what each system’s average finish would be.

But I’m doing this by hand, so thousands of drafts are out of the question. Instead, I decided to just do two drafts, using the following teams in each league:

League I
ESPN A
Last Player Picked A
BaseballMonster A
ESPN B
Last Player Picked B
BaseballMonster B
ESPN C
Last Player Picked C
BaseballMonster C
ESPN D
Last Player Picked D
BaseballMonster D

League II
RotoTimes A
RazzBall A
Last Player Picked A
RotoTimes B
RazzBall B
Last Player Picked B
RotoTimes C
RazzBall C
Last Player Picked C
RotoTimes D
RazzBall D
Last Player Picked D

I wanted each system to have enough teams represented that I would get a good feel for how they finished (without worrying about flukes). So each system gets four teams in the league, with each of their teams drafting independently of the others. Since I’m most concerned with my own ranking system, I have the Price Guide competing in both leagues.

I spread out the draft order to try and prevent any advantages from picking at a certain point in the draft. It is sometimes said that picking first is an advantage, so I’ve made it so that every system has a team drafting near the top, a couple in the middle, and one near the bottom. If picking first is really an advantage, we would expect the “A” teams to end up on top, and the “D” teams to end up at the bottom.

While drafting, I strictly followed each system’s rankings, picking the player that they ranked as the best available. I held to this even when I knew that their choices were not optimal.

A common situation where this came up involved the last team to fill a position: A smart player in that situation would recognize that they can wait as long as they want to fill that spot, and they will still be able to get the same player. They would be better off skipping over that player on their draft sheet (since no one else could draft him) to focus on drafting at other positions.

The teams in my Retro-Draft never did that. They blindly picked the best available player on their draft sheet with no thought of strategy whatsoever.

I did enforce positional eligibility, and each team could only draft a player if they had a position open for him. In situations involving a player who was eligible at multiple positions, I would shift their position if it allowed the team to draft the best available player. For example, if the rankings had Chipper Jones as the best available hitter and a team already had Aubrey Huff at 3B, then I shifted Huff to 1B and drafted Jones at 3B.

I also did all of the drafting by hand, so there could have been some minor mistakes. On the whole, though, I’m pretty confident that things ended up accurately.

How did everything work out? Tomorrow, I’ll break down the first draft.

Related posts:

• Evaluating the Player Evaluators, Part II (Baseball Monster and ESPN)
• Evaluating the Player Evaluators, Part III (Razzball and RotoTimes)
• Evaluating the Player Evaluators, Part IV (Showdown: Price Guide vs. Point Shares)
• Another Look at the RotoTimes Player Rater
• More Thoughts on ADP