The Diamondbacks' Outfield Defense is Very Good, but Can Their Outstanding UZR Numbers be Trusted?

Written by Amit Lal | 09 September 2011

There is no question that the Diamondbacks' outfield defense has very good range, and has been a big part of the team's success this year. According to the Ultimate Zone Ratings (UZR) at FanGraphs, the most valuable defensive player in the National League this year has been Gerardo Parra, with Chris Young #2 and Justin Upton #6. In terms of just range (excluding arms and errors), the Diamondbacks have 3 of the Top 5, with Upton the NL leader by far. At the team level, the Diamondbacks' UZR of +52.6 Runs gives them a 12.8 Run advantage over the #2 Reds (+39.8), and a whopping 39.2 runs over the #3 team, the Padres at +13.2. Let's take a closer look at the Diamondbacks' outfield defense and how the various defensive metrics rate them.

At the most basic level, the job of a team's defense is to turn batted balls into outs. The stat Defensive Efficiency Rating, first developed by Bill James, measured exactly that - what percentage of batted balls are turned into outs. In this stat, the Diamondbacks rank right in the middle of the pack, #13 out of 30 teams at a DER of .712 (Or inversely, a BABIP of .288).  

But one problem with evaluating a team defense by looking at DER or BABIP is that some types of batted balls are more likely to turn into outs than others. For example, fly balls are much more likely to turn into outs than ground balls, and infield flies are almost a certain out.

So a pitching staff that gets more flyballs will have a higher team DER and a lower BABIP (although they will also give up more homeruns and extrabase hits, but that's not the point of this exercise). The Diamondbacks' pitching staff in 2011 gets a lot of fly balls - their FB% rate of 37.7% is the highest in the NL and #4 in the Majors. So this would suggest that the D'Backs should have an even better DER, but they don't. So the question remains, if the team's DER is not exceptional, especially when the fly ball nature of the staff is considered, why does UZR like them so much?

Unfortunately, the raw data used for FanGraphs' UZR calculations is not available (as far as I can tell). So it's not clear why Upton's RngR (runs saved due to range) is the best in the NL.  However, FanGraphs does post some of the raw numbers from The Hardball Times' Revised Zone Ratings (RZR). RZR is a simpler, but in many ways more intuitive, defensive metric. RZR divides the field into discrete zones, and just counts how many balls are hit into each zone, and how many are turned into outs. A fielder's "Zone" is defined as those areas where the average fielder at that player's position turns at least 50% of balls into outs.  Here are the 32 qualifying NL outfielders, ranked by RZR:

Upton and Young are both in the upper third in turning "Balls in Zone" into Outs, but Parra is near the bottom. Where all three outfielders stand out, however, is in "Out of Zone" (OOZ) catches. In fact, Upton, Young, and Parra rank #1, 2, and 3 in the NL in making OOZ plays. Keep in mind that OOZ is a counting stat, not a rate stat, so all three are helped by being behind a fly-ball heavy pitching staff. Upton and Young have also played the most innings, although Parra and Upton rank #1 and #2 in OOZ plays per inning.

What are the differences between RZR and UZR? Well, there are a lot. First, UZR does not use discrete "Zones", but a more continuous distribution method. Second, and probably most important, UZR considers the time the ball takes to reach the zone (equivalent to a measure of how hard the ball was hit). Then, UZR gives a player positive or negative points based on how often a play is made on that type of batted ball compared to that of an "average" fielder at that position.  Finally, UZR makes many adjustments to the data, on things like park effects, the handedness of the pitcher, the base/out situation, and the batted-ball nature of the pitching staff. While all of these are important effects, it does make the end result of UZR much harder to follow.

In addition to UZR, there are at least three other defensive metrics worth examining - John Dewan's Plus-Minus (also called Defensive Runs Saved, or DRS), Baseball Reference's Total Zone, and Baseball Prospectus' Fielding Runs Above Average. DRS is the closest in concept to UZR, as both break down batted balls by type, location, and velocity, and then estimate the probability of the play being made. The two systems even use the same database. Yet the two systems often produce very different results, which is one of the most disconcerting aspects of fielding stats. An excellent discussion of the differences between the two systems is here. The basic differences are:

• UZR uses multi-year samples to estimate out probabilities, while DRS uses data over one year.
• DRS uses smaller buckets to divide plays.
• UZR uses a blanket park effect for the outfield and another for the infield, while DRS applies park factors to more buckets.
• They have slightly different techniques in assigning run values to each play made or not made.

While these don't seem like major differences, the discrepancies between UZR and DRS can occasionally be large. The creators of these stats acknowledge that sample sizes are a problem for all defensive stats, and suggest using multiple years of data to get a valid sample, or even regressing the UZR numbers halfway to toward zero. BR.com's Total Zone uses play-by-play data (i.e., fly ball to CF, or ground ball to 2B), rather than the direct observation of UZR and DRS. Baseball Prospectus' latest version of Fielding Runs is described here, and it uses neither direct observation or play-by-play data. Rather than evaluate the pros and cons of each method, let's just see how all four rate the Diamondbacks' outfielders.

All four systems score Parra highly, but he is the only one of the three with a consistent rating. What's most troubling for me is the large discrepancy between UZR and DRS, since both are based on the same dataset. Clearly, the adjustments and weights given to the raw data can make a huge difference. FRAA likely rates Young low because the outfield just doesn't make as many plays as would be expected by a fly-ball heavy staff. It is not at all clear which metric is the most correct, since they all differ so much. To me, the methodology of UZR seems the most sensible, but their results are also more likely to fail the "eye test", especially when compared to DRS (See for example, the great UZR scores this year for players like Nick Swisher or Pablo Sandoval, or for Pat Burrell last year).  Also, the UZR variations of players who change teams (i.e., Carl Crawford from Rays to Red Sox or Kelly Johnson from Braves to D"Backs) makes me question if UZR is really measuring the player's defense.

One other UZR issue that has come up recently is the possible correlation between Outfield UZR and a team's fly ball rate. Over the last two seasons, teams with a high fly ball rate have been scoring very high in UZR. 

Of course, teams that have a lot of fly ball pitchers would want to have a good defensive outfield. But this link between fly ball rate and Outfield UZR does look suspicious. And a look at some of the individual outfielders on these teams adds to the worry. The Yankees, for example, have a +11.6 from Nick Swisher this year. In 2010, the Giants scored a +6.1 from Pat Burrell, +1.5 from Aubrey Huff, and even a +1.0 from Jose Guillen. It almost seems like the adjustments used in UZR are not properly accounting for the effects of a strong fly ball staff.

In previous years, I think Tango's Fan Scouting Report was a great way to study a player's defense. Unfortunately, I think people's evaluations are being influenced more and more by the ubiquity of the UZR data, and the FSR is not as independent as it used to be.

Conclusions
Even with the recent improvements in defensive metrics, it is still very difficult to evaluate a player's defense. While UZR and DRS both seem like a logical way to measure defense, the answers spit out by these two methods differ more than one would expect. And the correlation between outfield UZR and the pitching staff's fly ball rate suggests that there could be a problem in the way UZR adjusts for the style of the pitching staff. 

Calculations like RZR and OOZ are more straightforward and easy to understand, since they are basically just counting stats, and do not make the extra step in trying to decide if an "average" fielder would have made the play. They at least show that the Diamondbacks' outfielders make a lot of "Out of Zone" plays. There is still a question as to why the team turns so few batted balls into outs. The team ranks #13 in DER, which is not a very good number for a team with a fly ball pitching staff and seemingly several good defensive players.

So are Parra, Young, and Upton the three most valuable defensive outfielders in the National League? I'm not sure if I would go that far. All of the metrics do agree on Parra, and most observers feel that he has played well in LF. In addition to good range, Parra has a strong and accurate arm too. Young's defensive metrics are all over the place - my visual evaluation rates him as above average for range (but not throwing), but I wish the metrics were more consistent. As for Upton, there is no doubt that he gets to a LOT of fly balls, and he also makes a few obvious errors. Overall, I think Upton is still a pretty big positive in the outfield. As a group, I think the trio of Parra, Young, and Upton are one of the best in baseball. I just wish I could trust the UZR numbers a little more.