MLB Ground Truth Testing

How does MLB check the accuracy of pitch tracking at ballparks?

MLB Ground Truth Testing

MLB works with the Washington State University Sports Science Lab (SSL) to independently measure pitch-tracking accuracy in every MLB ballpark. Specifically, WSU performs ground-truth tests at every MLB park. A ground-truth test is designed to precisely identify the position of a baseball during a trajectory event such as plate crossing or pitch release. The ground-truth reference is the standard by which we evaluate the accuracy and precision of in-stadia tracking systems.

Figure 1. Broadcast representation of pitch tracking data using a live strike zone.

Overview

Ground-truth testing refers generally to the independent on-field evaluation of the in-stadia ball-tracking systems. The goal of this testing is to validate accuracy and assess tracking health using a precise and independent method. Ground-truth testing is done after a tracking system is installed and calibrated at a stadium while the field approximates game-ready conditions. It can also be performed periodically as a health check for stadium tracking or when field changes or tracking issues present an opportunity to benefit from an independent assessment.

MLB contracts the WSU Sports Science Laboratory to perform ground-truth testing at every MLB stadium. WSU specializes in bat and ball collision dynamics to help regulating agencies understand equipment performance through modeling and experiment. The lab is certified with USA Softball, the NCAA, USA Baseball, and contracts with bat and ball manufacturers to verify compliance and evaluate prototype products.

Ground Truth Testing Method

The ground truth method relies on the frame-synchronized filming of a ball passing over the plate by two high-speed cameras. WSU uses Phantom cameras which operate at 2500 frames per second (fps). These 1920×1200 cameras are focused down to a field of view (FOV) of 5 x 3 ft on the space surrounding the strike zone as in Figure 2. This yields a spatial resolution of ~0.03 inches/pixel within the camera’s field depth. Exposure times of 50 µs allow still images from video with a motion blur of less than 80 thousandths of an inch (< 0.080”) as shown in Figure 3.

Figure 2. WSU’s high-speed camera focused on the strike zone.

Figure 3 shows a composite image of a 95-mph fastball filmed at 2500 fps with a 50-µs exposure time. This composite illustrates the minimal motion blur made possible by a 50-µs exposure time. At 2500 fps a pitch moves about 0.67” in between frames this can be seen in Figure 4, a video taken at 2500 fps. High resolution, high frame rate, and small FOV allow the ball position to be measured with high precision within the focal plane of the cameras.

Figure 3. Composite image of a 95-mph baseball filmed at 2500 fps.

In order to translate the high-precision within the cameras’ focal planes to the real world, an image calibration board is used, as shown in Figure 5. This is used to map the 2D data at the focal plane of each camera into the three dimensional XYZ coordinates of the real world. The calibration board consists of two panels tilted out of plane from each other. Each plane has precisely spaced and sized high-contrast dots. By using the dots as a standard reference, motion capture software is used to map them into 3D-space. A precision error is reported by comparing the calculated locations of the dots to the measured locations of the dots.

Figure 5. WSU’s Image calibration board of high-contrast dots with precise spacing and sizes.

WSU places two markers on the plate to identify the y direction (toward pitcher) and the plate tip as in Figure 5. If the plate is not level, these markers are shimmed to make level with respect to the upward (z) vector. The marker on the tip of the plate is directly used in motion capture software to assign the global origin of the coordinate system. The calibration reference images are shown in Figure 3.

Figure 6. Calibration panel reference images from each camera for the same scene.

Once the initial calibration is complete, a pneumatic cannon is used to launch pitches over the plate at 95 mph through the calibrated volume as in Figure 7. These pitches are filmed by each camera until around 100 pitches have been captured.

After collecting the high-speed videos from a ground-truth test, ball position is meticulously tracked in each frame by a technician in the lab. The technician uses software tools to optimize the images, and identify the center point of the ball as shown in Figure 8. Seven frames are tracked from each camera while the ball is crossing over the leading edge of the plate.

Figure 8. Still frames tracked in the lab showing pixel boundary identification in a synchronized pair of frames, one from each camera.

Since we would be very lucky to have a frame timed exactly as the ball crosses the front face of the zone, the ball position at y=17 inches is interpolated from trajectory data before and after the ball crosses into the zone. Here, the side (x) and height (z) vectors are regressed to the y direction (toward pitcher) and a linear equation is used to interpolate the side and height position when directly over the leading edge, at y = 17 inches. Pitch location at the front face of the strike zone (y=17” plane) is the central data product of each pitch during a ground truth test.

Pitch Release

Using a similar method to the strike-zone ground truth, we can also precisely track ball position at pitch release. In the case of a pitch-release ground truth test, a larger calibration board is elevated to calibrate the volume surrounding a pitcher’s release point as in Figure 9.

Figure 9. Elevated calibration board for a pitch-release ground truth test.

In a pitch-release ground-truth test, the ball is tracked as it passes through the calibrated space and the release point can be identified down to the frame-rate resolution of the cameras used. Figure 10 shows the frame-to-frame ball movement near the pitch-release point. Here, the ball position in subsequent frames is represented by red dots.

Figure 10. Red dots identify the ball center in subsequent frames near the release point of the pitch.

This method allows the pitch release point to be resolved in three dimensions down to the inter-frame distance traveled by the ball near release. The pitch-release event is shown in Figure 11 at 2500 frames per second.

Summary

This post briefly explains MLB’s ground-truth measurement method and shares some example videos. This program allows us to measure with high precision where the ball is when it crosses the plate or when it’s released. We compare this data to a stadium’s ball tracking system to assess accuracy and precision. This ground-truth testing exercise has been performed annually across all 30 MLB parks. Additional tests are scheduled to validate new tracking system installations and to conduct in-season health checks. The ground truth testing program is important to MLB as a broad quality control tool for ball tracking but also as a foundational assessment of the many metrics derived from pitch positional data.


MLB Ground Truth Testing was originally published in MLB Technology Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

MLB Ground Truth Testing