Object Based Verification of a Multi-Model Convection ...

Object Based Verification of a Multi-Model Convection ...

Object Based Cluster-Analysis and Verification of a Convection-Allowing Ensemble during the 2009 NOAA Hazardous Weather Testbed Spring Experiment Aaron Johnson and Xuguang Wang School of Meteorology and Center for Analysis and Prediction of Storms Fanyou Kong (CAPS), Ming Xue (SOM&CAPS), Kevin Thomas (CAPS), Keith Brewster (CAPS), Yunheng Wang (CAPS), Jidong Gao (NSSL) Warn-on-Forecast and High Impact Weather Workshop 9 February 2012 1 Motivation Convection allowing forecasts result in realistic looking convective systems for storm mode forecasts. (Coniglio et al. 2010). Object based verification is more consistent with subjective evaluations of high resolution precipitation forecasts than traditional metrics (e.g., Davis et al. 2006a; Johnson et al. 2011a). Physically descriptive diagnosis of errors Deterministic, not just probabilistic, verification is needed for model development and ensemble design. A different perspective on deterministic verification than traditional metrics (e.g., Kong et al. 2009) Object based cluster-analysis can show impact of perturbation sources on forecast diversity (Yussouf et al. 2004)

Model dynamics have a dominant impact on the 2009 CAPS ensemble clustering (Johnson et al. 2011b) Object-based verification can help us understand the differences between ARW and NMM Optimal grid spacing to balance computational cost and forecast quality is still an open question (e.g., Schwartz et al. 2009) Is it worth going from 4 km to 1 km grid spacing? 2 Outline Overview of ensemble and object-based methodology Object-based cluster analysis Forecast object realism evaluated with sample-climate average of object attributes Forecasts vs. observations ARW vs. NMM 1 km vs. 4 km Forecast accuracy evaluated with object-based MMI and a newly proposed OTS Forecasts vs. observations ARW vs. NMM 1 km vs. 4 km Summary and Conclusions 3 IC

LBC R MP PBL SW Rad. LSM CN NAMa NAMa CN em em N1 N1 CN nmm nmm N1 N1 CN etaKF etaKF N1 N1 CN CN etaBMJ etaBMJ N1 N1 CN + em em N1 N1 CN + nmm nmm N1

N1 CN CN + etaKF etaKFN1 N1 CN CN ++ etaBMJ etaBMJ N1 N1 CN NAMa NAMa CN nmm nmm N1 N1 CN etaKF etaKF N1 N1 CN CN etaBMJ etaBMJ N1 N1 CN + em em N1 N1 CN + nmm nmm N1 N1 CN CN ++ etaBMJ etaBMJ N1 N1 CN

NAMa NAMa NAMf NAMf em N1 nmm nmm N1 N1 etaKF etaKF N1 N1 etaBMJ etaBMJ N1 N1 em N1 nmm nmm N1 N1 etaKF etaKF N1 N1 etaBMJ etaBMJ N1 N1 NAMf NAMf nmm nmm N1 N1 etaKF etaKF N1 N1

etaBMJ etaBMJ N1 N1 em N1 nmm nmm N1 N1 etaBMJ etaBMJ N1 N1 NAMf NAMf Y N Y Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y N

Thompson Thompson Ferrier Thompson Thompson WSM6 WSM6 WSM6 Ferrier Thompson Ferrier Ferrier Ferrier WSM6 WSM6 WSM6 Thompson Ferrier Lin Lin MYJ MYJ YSU MYJ YSU MYJ MYJ YSU MYJ YSU MYJ

MYJ YSU YSU MYJ MYJ YSU YSU TKE TKE Goddard Goddard Goddard Dudhia Dudhia Goddard Dudhia Dudhia Dudhia Goddard GFDL GFDL Dudhia Dudhia Dudhia GFDL GFDL Dudhia 2-layer 2-layer NOAH NOAH

NOAH RUC NOAH NOAH NOAH NOAH NOAH RUC NOAH NOAH NOAH NOAH RUC RUC RUC RUC NOAH NOAH Member ARWCN ARWCN ARWC0 ARWC0 ARWN1 ARWN1 ARWN2 ARWN2 ARWN3 ARWN3 ARWN4 ARWN4

ARWP1 ARWP1 ARWP2 ARWP2 ARWP3 ARWP3 ARWP4 ARWP4 NMMCN NMMCN NMMC0 NMMC0 NMMN2 NMMN3 NMMN4 NMMP1 NMMP1 NMMP2 NMMP2 NMMP4 NMMP4 ARPSCN ARPSCN ARPSC0 ARPSC0 Initial background field from 00 UTC Planetary NCEP NAM

analysis. 10 members are from WRF-ARW, members 20 initialized 00 UTC, integrated 308hours over nearPerturbations to Microphysics, Boundary Assimilation of radar reflectivity and velocity using members CONUS domain on 26 days from 29 April

through 5 June 2009, on Layer, Shortwave Radiation andperturbations Land from WRF-NMM, and 2analysis members from ARPS. Coarser (~35 km) resolution IC/LBC ARPS 3DVAR and cloud for 17 Surface members from NCEP SREF forecasts 4obtained km grid without

CP.schemes. Model physics 4 Example of Objects by MODE 5 Object based scores Similarity of two objects is quantified by Total Interest, I (0 < I < 1) Function of area ratio, aspect ratio difference, orientation angle difference, centroid distance Mean value of attributes is also used to evaluate overall realism of objects Median of Maximum Interest (MMI; Davis et al. 2009) Compute the maximum possible Total Interest (I) for an object, when compared with all other objects at that time. Take the median of such maximum interests from all forecast and observed objects Object based Threat Score (OTS; Johnson et al. 2011a) Weight the area of each object by its Total Interest when compared to the corresponding object in the opposing field. Sum over all pairs of corresponding objects and divide by total area of all objects: 1 P p p p OTSij

w *(ai a j ) Ai Aj p 1 6 7 Need for object-based approach ED= 1305 mm 1-OTS = 0.486 ED= 1595 mm 1-OTS = 0.381 OTS is subjectively more reasonable as a distance measure Advantage of object-based clustering NED HCA is strongly sensitive to locations and amplitude OTS HCA can form clusters based on storm modes Similar to how we interpret them subjectively Consistent with severe storm forecasting applications (Johnson et al. 2011ab, MWR) 8 Object based clustering of 3-h forecasts How sensitive are forecasts to different source of perturbations?

Quantify similarity with OTS Members in same cluster are systematically more similar than those in different clusters 3-hour forecasts are most sensitive to DA, model dynamics, and microphysics scheme (Johnson et al. 2011ab, MWR) 9 Object based clustering of 24-h forecasts Little impact of DA Model dynamics still dominant Secondary clustering by PBL schemes rather than microphysics 10 Average attributes: forecast vs. observed Too many objects forecast after 1h lead time. Average forecast object is smaller, more circular and farther east than average observed object.

11 Average attributes: ARW vs. NMM Objects from NMM model are on average more numerous, larger, more circular, more zonally oriented and, beginning at 18 UTC, farther south than ARW. Number of objects, mean aspect ratio and mean angle are most similar to Observations for ARW. Mean area is most similar to Observations for NMM. 12 Average attributes: 1 km vs. 4 km 1 km forecasts fewer, larger, less circular, and farther west on average than 4km.

1 km is generally closer to obs for number of objects, area and E-W location than 4km. 13 Accuracy: All members OTS maximum at 12-h lead time caused by better forecasts of large precipitation systems at 12 UTC. Diurnal cycle similar to traditional ETS. MMI maximum at 24-h lead time caused by realistic meso-scale placement of small precipitation systems at 00 UTC. Control member is generally more accurate than perturbed members. NO DA members were worst especially at early lead times. 14 NMM worse than ARW and ARPS. Accuracy: ARW vs. NMM ARW group has significantly higher OTS and higher frequency of containing the best OTS than the NMM

group except short lead times Similar result for MMI but less pronounced Diagnostics found NMM configurations best at maintaining initial storms and ARW configurations best at forecasting future storms 15 Accuracy: 1 km vs. 4 km No significant difference in OTS between 1 km and 4 km members. 1 km member has significantly lower MMI at 12, 24-h lead times. Lower MMI at 12-h lead time caused by missed observed objects, small objects in particular. Consistent with worse under-forecasting at 12 UTC seen earlier 16 Summary and Conclusions Clustering analysis: Cluster analysis shows large impact of model dynamics on forecast clustering, even after bias adjustment, and additional impact of microphysics at 3-h lead time, PBL scheme at 24-h lead time Verification:

Mean attributes On average, forecast objects are too numerous, small, circular and east compared to observation. ARW vs. NMM: ARW better for number of objects, mean aspect ratio and mean angle. NMM better for mean area. 1km vs. 4km: After 1-h lead time, 1km better for number of objects, area and E-W location than 4km. Accuracy After 1-h lead time, ARW members are more accurate than NMM members. For short lead time, NMM configurations seem to evolve assimilated storms better. Generally similar accuracy at 1 and 4 km grid spacing. 17 4 instead of 16 km smoothing radius Still less objects, but not as few as obs Still larger area (better) Now, similar aspect ratio (less rounded than obs QPE) More similar location 18 19

Recently Viewed Presentations

  • Human health, public health and health&#x27;s promotion ...

    Human health, public health and health's promotion ...

    Human health, public health and health's promotion. Considerations. Human health Health promotion Public health Marcio Ulises Estrada Paneque. MD. PhD. Conceptual frame in HP action. To include an ample rank of strategies, such as communication, education, organizational change, communitarian development...
  • Pushin&#x27; Geo - Kennesaw State University

    Pushin' Geo - Kennesaw State University

    As well as colors, normals, and other vertex data. Pushin' Geo to the GPU. Jeff Chastine
  • Lecture 8: Schema Refinement and Normal Forms; Physical

    Lecture 8: Schema Refinement and Normal Forms; Physical

    Physical Database Design Database development involves three steps ER design Schema refinement (normalization) and view definition This defines the conceptual and external schemas Physical Design Choose indexes More schema refinement Consider denormalizing Vertical and horizontal decomposition Tuning the database and...
  • Also Speaking Today Robert Stewart Director, Asbury Park

    Also Speaking Today Robert Stewart Director, Asbury Park

    Preservation of Collections Decisions that can Affect your Library's Future Preservation Advisory Committee New Jersey Library Association History and Preservation Section
  • Title

    Title

    A fine-grained sedimentary rock formed from the lithification of clay, silt, or mud. Shale has a laminated structure, which splits easily. For the purpose of this specification, mudstone and claystone are also considered to be shale.
  • Of Plymouth Plantation - SCHOOLinSITES

    Of Plymouth Plantation - SCHOOLinSITES

    Of Plymouth Plantation ... Maybe out of modesty or to seem more trustworthy to his readers Biblical Allusions ~God's providence Classical Allusions ~Roman Stoic philosopher, Seneca Other imaginative comparisons: ~Plight of the Pilgrims/Paul's shipwrecked company. ...
  • ИСТРАЖИВАЊЕ У МАТЕМАТИЦИ ПОМОЋУ РАЧУНАРА - како рачунар ...

    ИСТРАЖИВАЊЕ У МАТЕМАТИЦИ ПОМОЋУ РАЧУНАРА - како рачунар ...

    Title: ИСТРАЖИВАЊЕ У МАТЕМАТИЦИ ПОМОЋУ РАЧУНАРА - како рачунар помаже у математичким истраживањима -
  • Graduation Requirements for 2017-2018 9th Graders Benefits of

    Graduation Requirements for 2017-2018 9th Graders Benefits of

    For specific colleges or programs (Engineering or Business), in order to start taking courses in their major as a freshman, students must be admitted under the assured admissions. What Colleges Are Saying . cont …