How good is an ensemble at capturing truth?: using bounding boxes for forecast evaluation

Ensemble prediction systems aim to account for uncertainties of initial conditions and model error. Ensemble forecasting is sometimes viewed as a method of obtaining (objective) probabilistic forecasts. How is one to judge the quality of an ensemble at forecasting a system? The probability that the bounding box of an ensemble captures some target (such as truth in a perfect model scenario) provides new statistics for quantifying the quality of an ensemble prediction system: information that can provide insight all the way from ensemble system design to user decision support. These simple measures clarify basic questions, such as the minimum size of an ensemble. To illustrate their utility, bounding boxes are used in the imperfect model context to quantify the differences between ensemble forecasting with a stochastic model ensemble prediction system and a deterministic model prediction system. Examining forecasts via their bounding box statistics provides an illustration of how adding stochastic terms to an imperfect model may improve forecasts even when the underlying system is deterministic.