SubscribeYesterday, I explained that I was going to try to compare the various player valuation systems by having them pick teams in a mock Retro-Draft–a draft for the 2008 season after the season. Doing that will let us see which one does the best job of ranking players, because we can look at the final standings as soon as the draft is over.
The first league involved my own Price Guide, ESPN’s Player Rater, and BaseballMonster.com’s rankings. You can read all about the league setup in the introduction, but this was the draft order:
ESPN A
Last Player Picked A
Baseball Monster A
ESPN B
Last Player Picked B
Baseball Monster B
ESPN C
Last Player Picked C
Baseball Monster C
ESPN D
Last Player Picked D
Baseball Monster D
The ESPN A-Team started things off with Pujols as the first overall pick, although all of the systems had him ranked #1. After that, things diverged quickly: BaseballMonster and ESPN had three pitchers in the top 10 (Halladay, Sabathia, and Lincecum), while the Price Guide only had one (Halladay at #6).
All of the Price Guide teams had filled both C spots by Round 13. Catchers were the last two picks for all of the other teams.
I don’t find draft write-ups to be particularly interesting, so allow me to skip the details and just show how the final standings shaped up:
LPP D 79
LPP C 77
LPP A 75
LPP B 68
ESPN D 63
BM C 63
BM B 62
ESPN B 60
BM D 60
ESPN A 59
ESPN C 58
BM A 56
Some thoughts:
1. I’m quite pleased to see Last Player Picked come away with the top four spots. The key difference that I noticed while drafting was the Price Guide’s proper adjustment for replacement level for C and middle infield. I mentioned it above, but LPP teams started grabbing catchers in the 2nd round; everyone else waited until their last two picks. In this draft and in the draft I’ll show tomorrow, an LPP team took Pedroia and Utley at the turn of the 1st and 2nd rounds.
2. On a related note, it’s pretty clear why LPP B ended up at the bottom (relative to the other LPP teams, that is). Their first five picks went like this:
David Wright – 3B
Joe Mauer – C
Brian McCann – C
Joakim Soria – P
Geovany Soto – C
So three catchers in the first five picks, which means one goes in the Utility spot.
Now, if this were a person drafting, they would probably recognize a couple of things:
a) The positional replacement levels don’t really apply to Utility–at that position you just want the best stats available.
b) Catchers only get the benefit of the catcher replacement level when they are put in a catcher slot. When they are put at Util, they are just like any other player.
I realized these things as I was drafting, but my rule was to completely auto-pick with each system’s rankings.
So a real person drafting would have (wisely) skipped over Soto and grabbed one of the next names on the list, like Jermaine Dye or Vladimir Guerrero. (That was a second issue with this team: Not only did they fill their Utility spot early with a lesser player, but they managed to not draft any OF until round 16.) If LPP B had picked Guerrero instead of Soto, they would probably have ended up with about 74 points (and a commanding lead over the 63 point teams).
3. It’s hard to tell which of ESPN and BaseballMonster did better. BaseballMonster went for more pitchers and corner infielders, ESPN tended to get the outfielders. The end result was about the same, and it’s hard to pick a 2nd place winner. (Tomorrow’s results ended up more clearly cut.)
4. In this scenario there didn’t seem to be any advantage to picking first or any disadvantage to picking last. I don’t see any pattern to how the A, B, C, and D teams finished.
5. With the other two systems placing a premium on SP, the Price Guide teams ended up missing out on a lot of the big names starters. As a result, they had the chance to draft better hitters (doing very well across the board in the offensive categories) and better relievers (dominating ERA and WHIP and doing very well in S).
The problem comes if this were a league with an IP minimums. Each of the LPP teams only drafted about 600 IP, which might not be enough depending on what the minimum is. Out of curiosity, I tried replacing the last RPs drafted by LPP teams with SPs that went undrafted (guys like Jered Weaver, Randy Wolf, and Greg Maddux) to bump them up to about 900 IP. That resulted in much tighter standings, with the Price Guide still holding on to the top spots.Dealing with IP minimums looks like it is one shortcoming of the Price Guide’s current system. In this case, though, it turned out to be a non-issue.
6. I mentioned above that the Price Guide teams all benefited from taking catchers early, which made me wonder if I was being unfair by setting the league requirement at two catchers. It kind of ended up looking that way, but it certainly wasn’t my intent. While it was my goal to pick a pretty standard league setup, in retrospect it is clear that it hurt all of the other systems.
Here’s the bottom line, though: The Price Guide’s values will adjust for any league configuration. It handles two catchers. It handles any stat category. It handles any size league. There’s probably some league configuration out there that ends up making it a close competition. But there are a whole bunch of configurations for which the Price Guide is going to do just as well as it did here.
***
With the Price Guide holding up very well against its first two competitors, it’s time to see how it does against some others. Tomorrow, I’ll run through the draft results of the second league, with teams from the Price Guide, RazzBall Point Shares, and RotoTimes.
Related posts:
Great site Mays, first off. And real nice to see someone really try to come up with an accurate online valuation calculator, since valuing players correctly just isn’t discussed enough
Second, it’s really no surprise that your Price Guide team cleaned up. I’ve always known the various rating systems you’re “competing” against have been terrible and don’t adjust for position. I’m not sure why they have never been changed when they are so obviously wrong! Any system that properly adjusts for position is going to easily sweep these player raters.
I’ve probably had the most experience looking at the RotoTimes rater and I’m positive it’s terrible, with no positional adjustment whatsoever. I’m curious how the Point Shares system stacks up.
Yeah, I’m also looking forward to how RotoTimes and especially the Point Shares systems do. One thing I’ve noticed is that those two systems operate off of an average, whereas the Price Guide uses a replacement level baseline. I’m sure you’ll get into this in Part III, but does changing the baseline change the results? I’m thinking it might make a difference at a position like shortstop, where there are a lot of similar players at/near replacement level, but where the average is skewed by the Hanley, Reyes, Rollins trio.
I would also add that in a real draft, while you need a baseline to account for differences in position, what we’re really after is marginal value compared to the remaining players at each position. it’s all about marginal value, not necessarily points above replacement or average. I think your simulation here is great for what it is aiming to do, however.
Oops. Sorry. That penultimate sentence was not meant to add emphasis. I just rewrote what I was trying to say and then forgot to take the original out…
@Mike: I appreciate the kind words, and I hope you find the site to be useful.
I don’t want to give away tomorrow’s results, but I can assure you that RotoTimes did not do well.
@Nick: Using the average player or the replacement player as a baseline should give the same results, because either way you’re subtracting a constant from all of the players at a position. You could use 10 million as your baseline, and, since you adjust every player by that same amount, they are still the same in relation to each other.
The only reason I use replacement level is that it allows you to figure dollar values, based on a $1 replacement level player.
Are you sure though? Consider this hypothetical 4 team NL only league
http://www.lastplayerpicked.com/priceguide/index.php?t=4&l=NL&m=260&b=1&ds=09M&dis=250&AVG=Y&R=Y&RBI=Y&HR=Y&SB=Y&W=Y&S=Y&ERA=Y&WHIP=Y&K=Y&C=1&1B=1&2B=1&3B=1&SS=1&OF=3&LF=0&CF=0&RF=0&CI=0&MI=0&Util=0&mg=20&SP=5&RP=2&P=0&ms=5&mr=5
The SS replacement level is -3.23 (Tejada) and the 2B replacement level is very close at -3.63 (Kelly Johnson). However, if you look at positional averages, the SS avg is .765 while 2B is substantially lower at -1.3. That’s a difference of more than 2 using the positional average, as opposed to only .4 using replacement as a baseline.
So wouldn’t the baseline make a difference in an extreme case like this, or did I mess up somewhere?
Thanks for all your work on the projection system. I have been trying to figure out how to do this on my own for a long time, and thanks to your articles on creating the price guide, I’ve been able to put this together on my own.
I’ve not gotten as advanced as this is, but my goal has always been to get a sense of how my own player projections might alter their value–and now I have a way to do that.
This is fascinating stuff, and I’m glad you’ve taken the time to do this.
Have you given any thought to a draft spreadsheet? Last year a guy made one that worked very well at NineBoJacksons.com. The site is now gone, but he produced a couple spreadsheets that I found very helpful.
John
Just a quick comment on this as I have a longer comment on the previous analysis. On ESPN’s behalf, their default roster setup is 1 Catcher. But they don’t factor in position scarcity so it’s not a major issue here. But 2 catcher would exacerbate that lack of a factor…
@John: I have considered a spreadsheet, but it’s pretty low on my priority list. I’ve thought there might be a place for something offline to complement the online Price Guide.