Last Player Picked

I hadn’t planned on running another scenario, but Rudy left a comment on the Price Guide vs. RotoTimes vs. Razzball post:

Interesting test. Simulated draft tests are difficult to do w/o introducing bias. [...] I would argue that creating 4 800 IP teams is an artificial construct. You’re never going to see a league where 4 teams punt pitcher counting stats. I’d be interested to see how the test would go if you credited each team a realistic 5 starters and aim for a 1200 IP avg per team.

Rudy’s point is a good one, and one that I acknowledged in both Part II and Part III. Since both of us are interested in what would happen in a more typical scenario, let’s do one more trial.

This test will be a showdown between 5 Price Guide teams and 5 Point Shares teams. It will be a Retro-Draft like last time, but I’m going to change it up to give Point Shares every possible advantage.

Start One Catcher Per Team
The first time around, I drafted for a league that starts two catchers. Since Rudy mentioned that Point Shares assume one-catcher per team, let’s go with that this time. In fact, let’s use the complete league defaults from an ESPN league: 10-teams, 1 catcher, 5 OF, no CI, no MI. This is the Point Shares’s home turf.

Update: Rudy mentions that, despite what was shown on the Razzball site, the Point Shares are built based on a league with 1 CI and 1 MI. You can check the comments to see the results when I add CI/MI (and a few other modifications).

Draft More Starters
To get the Price Guide teams to draft more starters this time, I requested dollar values pretending that the league required 6 SP and 3 RP (instead of 9 P like last time). Basically, I’m forcing the Price Guide to ignore what it has already demonstrated to be the optimal strategy and instead take the typical approach (i.e. more starter focused). With this constraint in place, the Price Guide builds rankings so that each team should average 1309 IP.

So how did things turn out?

LPP C 63.5
LPP E 62
LPP B 58.5
RB C 58
LPP A 57.5
LPP D 55.5
RB B 53
RB A 52.5
RB D 48
RB E 41.5

Razzball manages to sneak one team into the upper half of the standings, but otherwise the Price Guide ends up with the clear edge.

This time, the LPP teams drafted SP like they were supposed to, averaging 1,505 IP per team.

On the other hand, it was the Razzball teams who were loading up on RP! They averaged only 3 SP per team and 1,065 IP. (Unfortunately, the draft-lots-of-relievers strategy didn’t work as well for them as it had for the Price Guide earlier.)

If anyone is interested in the details, I’ve got the full draft results. I’d also be interested if anyone has any critiques for the method I’m using to compare the systems.

Last time, I took a look at how the Price Guide stacked up against a couple of other valuation systems, one from ESPN and one from Baseball Monster. Now I want to see how it compares to two others: RotoTimes and Razzball.

The format is the same as before: A 2008 Retro-Draft that gives all of the systems the benefit of perfect hindsight. The starting positions and categories are standard for 5×5 rotisserie. 12 teams will be drafted, with 4 teams representing each of the 3 fantasy raters. I put the teams in this order to try to remove any bias from selecting early or late:

RotoTimes A
Razzball A
Last Player Picked A
RotoTimes B
Razzball B
Last Player Picked B
RotoTimes C
Razzball C
Last Player Picked C
RotoTimes D
Razzball D
Last Player Picked D

Each team can only draft players at positions they qualify at (20 game requirement). I did all of the drafting by hand, so there could have been mistakes (although I doubt it would affect the outcome if there were).

So how did the draft turn out? Well, the first round was pretty indicative of each team’s “strategy”:

RT A: Roy Halladay
RB A: CC Sabathia
LPP A: Albert Pujols
RT B: Tim Lincecum
RB B: Cliff Lee
LPP B: Hanley Ramirez
RT C: Francisco Rodriguez
RB C: Johan Santana
LPP C: David Wright
RT D: Jose Reyes
RB D: Mariano Rivera
LPP D: Dustin Pedroia

What do we see? All four of the Razzball teams grabbed pitchers in the first round. The RotoTimes teams drafted three pitchers. LPP’s choices, however, look more like the conventional first round picks (with a bit of an emphasis on infielders).

Those basic trends continued throughout. The Razzball teams were the first to fill up their pitching. RotoTimes also favored pitchers but focused more on closers. They were also most likely to grab guys for SB, especially outfielders. LPP got the best catchers (starting with LPP A grabbing Mauer near the end of the 2nd round) and middle infielders, and filled in their pitching staffs with the leftovers.

Here are the final standings, which ended up much more clearly-cut than I expected:

LPP C 78
LPP B 75
LPP A 73
LPP D 72
RB B 69.5
RB D 66.5
RB C 63
RB A 62
RT B 58
RT A 58
RT D 54
RT C 51

My thoughts, once again:

1. Like last time, the Price Guide takes Gold, Silver, and Bronze (and 4th, whatever that would be). Since the methodology seems to me like the most logical way to value players, it was reassuring to see that the theory holds true in practice.

2. I actually expected RotoTimes to do better, since their site allows you to customize the number of hitters and pitchers. (Both RotoTimes and LPP’s picks were based on their rankings for 14 hitters and 9 pitchers.) Since Razzball’s Point Shares aren’t customized for the number of hitters or pitchers, they were starting with a possible disadvantage.

3. Like last time, I didn’t specify a minimum for IP or AB. With no minimum and with the other teams drafting SP like crazy, the Price Guide decided that 60 good innings from Geoff Geary or Jim Johnson were better than 180 innings from an average starter. The LPP teams ended up last in W, S, and K; and first in ERA and WHIP (in addition to being very good offensively). They also ended up with about 650 IP each.

I realize this isn’t realistic with a lot of leagues that have a minimum IP, so out of curiosity I went back and substituted some of those middle relievers on LPP teams with SP that went undrafted (Hiroki Kuroda, Tim Wakefield, Paul Maholm, A.J. Burnett, etc.) to get each of them above 800 IP. That hurt the LPP teams in ERA and WHIP, and it wasn’t enough to catch any of the teams in the other pitching categories. The results on the final standings were minor:

LPP C 73.5
LPP B 73
LPP D 70.5
RB B 70.5
LPP A 68
RB C 68
RB A 68
RB D 67.5
RT B 58
RT A 58
RT D 54
RT C 51

One Razzball team moves into a tie for third, and an LPP team slides into a three-way tie for fifth. Everything tightened between LPP and Razzball, but RotoTimes was completely unaffected.

5. I’m pretty confident that the Price Guide is the best, but can we tell how the others stack up? It’s hard to say for sure without directly comparing Razzball with ESPN or RotoTimes with Baseball Monster. I would guess that RotoTimes is actually the least accurate, judging by their uniformly poor performance this time. Competing against them may have benefited Razzball some, but I don’t think it’s clear that it did.

Someone else is welcome to do their own evaluation to see how they rank.

In the end, though, Last Player Picked’s values look to be more accurate for indicating league standings than any of the others. Since they are customizable for any fantasy league, I’d wager that they perform just as strongly for any other league configuration.

Yesterday, I explained that I was going to try to compare the various player valuation systems by having them pick teams in a mock Retro-Draft–a draft for the 2008 season after the season. Doing that will let us see which one does the best job of ranking players, because we can look at the final standings as soon as the draft is over.

The first league involved my own Price Guide, ESPN’s Player Rater, and BaseballMonster.com’s rankings. You can read all about the league setup in the introduction, but this was the draft order:

ESPN A
Last Player Picked A
Baseball Monster A
ESPN B
Last Player Picked B
Baseball Monster B
ESPN C
Last Player Picked C
Baseball Monster C
ESPN D
Last Player Picked D
Baseball Monster D

The ESPN A-Team started things off with Pujols as the first overall pick, although all of the systems had him ranked #1. After that, things diverged quickly: BaseballMonster and ESPN had three pitchers in the top 10 (Halladay, Sabathia, and Lincecum), while the Price Guide only had one (Halladay at #6).

All of the Price Guide teams had filled both C spots by Round 13. Catchers were the last two picks for all of the other teams.

I don’t find draft write-ups to be particularly interesting, so allow me to skip the details and just show how the final standings shaped up:

LPP D 79
LPP C 77
LPP A 75
LPP B 68
ESPN D 63
BM C 63
BM B 62
ESPN B 60
BM D 60
ESPN A 59
ESPN C 58
BM A 56

Some thoughts:

1. I’m quite pleased to see Last Player Picked come away with the top four spots. The key difference that I noticed while drafting was the Price Guide’s proper adjustment for replacement level for C and middle infield. I mentioned it above, but LPP teams started grabbing catchers in the 2nd round; everyone else waited until their last two picks. In this draft and in the draft I’ll show tomorrow, an LPP team took Pedroia and Utley at the turn of the 1st and 2nd rounds.

2. On a related note, it’s pretty clear why LPP B ended up at the bottom (relative to the other LPP teams, that is). Their first five picks went like this:

David Wright – 3B
Joe Mauer – C
Brian McCann – C
Joakim Soria – P
Geovany Soto – C

So three catchers in the first five picks, which means one goes in the Utility spot.

Now, if this were a person drafting, they would probably recognize a couple of things:

a) The positional replacement levels don’t really apply to Utility–at that position you just want the best stats available.
b) Catchers only get the benefit of the catcher replacement level when they are put in a catcher slot. When they are put at Util, they are just like any other player.

I realized these things as I was drafting, but my rule was to completely auto-pick with each system’s rankings.

So a real person drafting would have (wisely) skipped over Soto and grabbed one of the next names on the list, like Jermaine Dye or Vladimir Guerrero. (That was a second issue with this team: Not only did they fill their Utility spot early with a lesser player, but they managed to not draft any OF until round 16.) If LPP B had picked Guerrero instead of Soto, they would probably have ended up with about 74 points (and a commanding lead over the 63 point teams).

3. It’s hard to tell which of ESPN and BaseballMonster did better. BaseballMonster went for more pitchers and corner infielders, ESPN tended to get the outfielders. The end result was about the same, and it’s hard to pick a 2nd place winner. (Tomorrow’s results ended up more clearly cut.)

4. In this scenario there didn’t seem to be any advantage to picking first or any disadvantage to picking last. I don’t see any pattern to how the A, B, C, and D teams finished.

5. With the other two systems placing a premium on SP, the Price Guide teams ended up missing out on a lot of the big names starters. As a result, they had the chance to draft better hitters (doing very well across the board in the offensive categories) and better relievers (dominating ERA and WHIP and doing very well in S).

The problem comes if this were a league with an IP minimums. Each of the LPP teams only drafted about 600 IP, which might not be enough depending on what the minimum is. Out of curiosity, I tried replacing the last RPs drafted by LPP teams with SPs that went undrafted (guys like Jered Weaver, Randy Wolf, and Greg Maddux) to bump them up to about 900 IP. That resulted in much tighter standings, with the Price Guide still holding on to the top spots.Dealing with IP minimums looks like it is one shortcoming of the Price Guide’s current system. In this case, though, it turned out to be a non-issue.

6. I mentioned above that the Price Guide teams all benefited from taking catchers early, which made me wonder if I was being unfair by setting the league requirement at two catchers. It kind of ended up looking that way, but it certainly wasn’t my intent. While it was my goal to pick a pretty standard league setup, in retrospect it is clear that it hurt all of the other systems.

Here’s the bottom line, though: The Price Guide’s values will adjust for any league configuration. It handles two catchers. It handles any stat category. It handles any size league. There’s probably some league configuration out there that ends up making it a close competition. But there are a whole bunch of configurations for which the Price Guide is going to do just as well as it did here.

***

With the Price Guide holding up very well against its first two competitors, it’s time to see how it does against some others. Tomorrow, I’ll run through the draft results of the second league, with teams from the Price Guide, RazzBall Point Shares, and RotoTimes.

Rudy recently did a study to see how the RazzBall Point Shares system stacked up against the existing competition. In setting up my own fantasy valuation system, the Price Guide, I was of course curious to see how it would do as well.

So, to compare the valuation systems, I decided to do a Retro-Draft for the 2008 season. That is, a draft for the 2008 season, right now, after the season. In this draft, each of the systems gets a team that will use that system’s rankings as their exclusive draft sheet. When the draft is over, I can figure up the league standings and instantly tell which team won.

The systems I considered were:

Last Player Picked’s Price Guide
ESPN’s Player Rater
BaseballMonster’s rankings
RotoTimes’s Player Rater
RazzBall’s Point Shares

All of the teams have the benefit of 20/20 hindsight. Everyone knows the exact stats each player ended up with, and so the only difference should be their ranking methodology.

The league I’m drafting for is a standard rotisserie mixed league: 12 teams, 14 hitters, and 9 pitchers. A hitter has to play 20 games at a position to qualify. I consider traditional rotisserie to be neutral ground that shouldn’t favor any system.

With 12 openings and 5 competitors, one problem is figuring out how to handle the draft order. The ideal situation might be simulating the draft thousands of times, each time picking the teams at random. Eventually it would begin to become apparent what each system’s average finish would be.

But I’m doing this by hand, so thousands of drafts are out of the question. Instead, I decided to just do two drafts, using the following teams in each league:

League I
ESPN A
Last Player Picked A
BaseballMonster A
ESPN B
Last Player Picked B
BaseballMonster B
ESPN C
Last Player Picked C
BaseballMonster C
ESPN D
Last Player Picked D
BaseballMonster D

League II
RotoTimes A
RazzBall A
Last Player Picked A
RotoTimes B
RazzBall B
Last Player Picked B
RotoTimes C
RazzBall C
Last Player Picked C
RotoTimes D
RazzBall D
Last Player Picked D

I wanted each system to have enough teams represented that I would get a good feel for how they finished (without worrying about flukes). So each system gets four teams in the league, with each of their teams drafting independently of the others. Since I’m most concerned with my own ranking system, I have the Price Guide competing in both leagues.

I spread out the draft order to try and prevent any advantages from picking at a certain point in the draft. It is sometimes said that picking first is an advantage, so I’ve made it so that every system has a team drafting near the top, a couple in the middle, and one near the bottom. If picking first is really an advantage, we would expect the “A” teams to end up on top, and the “D” teams to end up at the bottom.

While drafting, I strictly followed each system’s rankings, picking the player that they ranked as the best available. I held to this even when I knew that their choices were not optimal.

A common situation where this came up involved the last team to fill a position: A smart player in that situation would recognize that they can wait as long as they want to fill that spot, and they will still be able to get the same player. They would be better off skipping over that player on their draft sheet (since no one else could draft him) to focus on drafting at other positions.

The teams in my Retro-Draft never did that. They blindly picked the best available player on their draft sheet with no thought of strategy whatsoever.

I did enforce positional eligibility, and each team could only draft a player if they had a position open for him. In situations involving a player who was eligible at multiple positions, I would shift their position if it allowed the team to draft the best available player. For example, if the rankings had Chipper Jones as the best available hitter and a team already had Aubrey Huff at 3B, then I shifted Huff to 1B and drafted Jones at 3B.

I also did all of the drafting by hand, so there could have been some minor mistakes. On the whole, though, I’m pretty confident that things ended up accurately.

How did everything work out? Tomorrow, I’ll break down the first draft.