Progress towards hybrid 4DVar with the FV3 dynamical

Progress towards hybrid 4DVar with the FV3 dynamical

Progress towards hybrid 4DVar with the FV3 dynamical core Daniel Holdaway, Yannick Trmolet, Anna Shlyaeva, Mark Miesch, Stephen Herbener, Guillaume Vernieres, Rahul Mahajan and Jong Kim Joint Center for Satellite Data Assimilation (JCSDA) NASA Global Modeling and Assimilation Office (GMAO) Introduction 4D-Var and hybrid 4D-Var have been the algorithms of choice for the past 20 years. With this in mind NASAs Global Modeling and Assimilation Office (GMAO) has developed the tangent linear and adjoint of three different version of FV(3). Hybrid 4D-Var with FV3 has so far proved illusive and currently NASA run hybrid 4DEnVar systems based on EnKF with 32 ensemble members. The lessons learnt from GMAOs progress towards 4D-Var with FV3 could be quite relevant to other centers considering 4D-Var with next generation models. Recent attempts to decrease the performance gap between hybrid 4DEnVar and hybrid 4D-Var have not been successful (Bowler et. al. 2017) so developing efficient 4D-Var remains an appealing goal. A number of factors motivate a further attempt at developing a 4D-Var for FV3 models at the current time. A new era of dynamical cores GMAO has struggled to implement 4D-Var is in part because of being ahead in terms of modelling, having been using the FV3 dynamical core for nearly 10 years now. DA with FV3 has been hard to work with due to its relatively short timestep (~60 seconds vs ~900 seconds GFS/IFS/MetOfficePF). Number of cores needed to meet NGGPS operational requirements Changes were made in the TLM and adjoint to give efficiency gains but they came with a loss of accuracy. Fortunately NGGPS forced lots of improvements to FV3 and these have been incorporated into a new adjoint version by NASA GMAO.

Single precision. Tuning and clean up. Intel 18 optimization for intel chip sets. OpenMP. Previous GEOS config ~10,500 IFS ~2500 A new era of dynamical cores 200 mb KE spectrum Day-10 (mean) In the NGGPS Dynamical Core Bake-off there was significant focus on the effective resolution of the models. By design large time step models have lower effective resolution, e.g. GFS, IFS. FV3 and MPAS capture the -5/3 spectrum, GFS does not. Higher effective resolution Though the challenges are large theres big potential payoff from using a smaller timestep and resolving smaller scales and so other centers are moving to FV3 like non-hydrostatic dynamical cores.

NASA GMAO: FV3GEOS (NGGPS version) NOAA NWS: FV3GFS NCAR: MPAS Naval Research Lab: Neptune Met Office: LFRic DWD: ICON Is 4DVar with these kinds of models a reality? FV3 adjoint system A new FV3 TLM & adjoint has been developed: Based on NGGPS version of the FV3 core, now used at GMAO and NCEP in Jan 2019. Capable of being run in single precision. Non-hydrostatic dynamics can be exercised (unlikely to be useful outside of sensitivity studies for now). Generated using Tapenade automatic differentiation to cope with community nature of the code. But with custom trajectory saving subroutines to avoid excessive recomputation (if enough memory present), saving and to allow compile time (not adjoint generation time) precision choice (github.com/danholdaway/TapenadeCustomPushPop). Linear advection/remapping with extra damping. Internal reference state can also be linear for speed up in 4D-Var. Can run on Intel and GCC compilers for community use. FV3 adjoint system: TLM correlations Analysis increment correlation between nonlinear and tangent linear models. v tv q dp qi

ql o3 10 10 20 20 20 20 20 20 20 20 30 30 30 30 30 30 30

30 40 40 40 40 40 40 Model Level 10 Model Level 10 Model Level 10 Model Level 10 Model Level 10 Model Level 10

Model Level Model Level u 40 40 50 50 50 50 50 50 50 50 60 60 60 60 60 60

60 60 70 70 70 70 70 70 70 70 0 0.5 1 0 Mean correlations: 0.5 1 0 0.5

1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 FV3 adjoint system: FSOI % recovery 8 Observation Impact Time Series | 01 October 2016 to 31 October 2016 | Every 6 hours (e4dect: Percent captured = 77.21% | Relative difference = 0.228 | Correlation = 0.823)

(e4dect: Percent captured = 86.02% | Relative difference = 0.140 | Correlation = 0.865) Model Error (Jkg1) 6 With new adjoint we recover an extra 10% or so of the observation impact. 4 Forecast from xb 2 Forecast from xa Nonlinear impact Control imapct New adjoint impact 0 2 4 02Oct16 07Oct16 12Oct16 17Oct16 Date 22Oct16 27Oct16 GEOS adjoint timing in GSI Adjoint running within GSI 4D-Var system.

70 iterations C180 (50km) resolution Sufficient number of cores to turn off forward calculation. 2400 processors (total = 23.54 mins) 4374 processors (total = 17.78 mins) Boundary layer ADM Moist ADM Dynamics ADM Starting to approach a reasonable running time at around 17-18 minutes. Not fast enough but doesnt include: Dynamics FWD Boundary layer TLM Moist TLM Much manual refinement of Tapenade produced code. OpenMP Lower resolution inner loops as part of multiple outer loops. Intel 18 optimization. Dynamics TLM 0 100 200 300 400 Seconds (70 iteration configuration)

500 600 700 Fast, flexible, highly scalable DA To run 4D-Var we will likely need to run the DA with a similar number of cores as the model (5400). Given the time needed by the adjoint this requires highly efficient and scalable DA. GSI is run with far fewer cores than the forecast model at both NCEP and GMAO. As weve tried to run 4D-Var weve run into the issues that motivated this choice. Grid transforms, the recursive filter and variable transforms dont scale well. Cube to lat-lon and lat-lon to cube and their adjoints are an example of a current issue. This will be alleviated with a move to ESMF-7 but clearly avoiding these grid transforms would be beneficial. When increasing number of procs the grid transforms (red) go from ~25% to ~85%. The FV3 part scales well. C2LL FV3 PARTS MOIST PHYS PARTS 600 PROCS Not enough memory to turn of forward calculations, 3.5Tb of RAM.

LL2C 4374 PROCS JEDI The Joint Effort for Data assimilation Integration (JEDI) is a new data assimilation system in development by the Joint Center for Satellite Data Assimilation (JCSDA) and in cooperation with NASA, NOAA, Met Office, ECMWF, Meteo France, NRL and NCAR JEDI is based on grid agnostic data assimilation components OOPS, UFO and IODA with a model interfacing to perform DA on the native model grid. We are currently working towards integrating the FV3 models, GEOS and FV3GFS into the JEDI framework. Currently we are at the point of performing data assimilation with sample observation data sets. As well as hybrid 4D-Var we plan to implement hybrid 4dEnVar as well as numerous other flavors of data assimilation suitable for a wide variety of computing platforms. 3DEnVar software demonstration Pure ensemble B matrix using BUMP. 10 member ensemble from FV3GFS. Sample ROAB observations 2 outer loops 20/10 inner loops 2 hour assimilation window C48 (200km) resolution. Tested with up to 384 cores. 4D-Var RAOB software demonstration 6h window 4D-Var with FV3 dynamical core TLM/ADM but no physics. 2H 1H 3H 1H

4H 5H 6H Upcoming work JEDI DA Develop native grid static B matrix for FV3-JEDI using BUMP (code sprint in August) Complete interfacing to full FV3-GFS and GEOS models. Complete interfacing to UFO (conventional, radiances and GNSS-RO). Implement full TLM/ADM physics from GEOS. FV3JEDI-TLM & Adjoint Refine some of the Tapenade generated code to reduce in-loop checkpointing. Implement custom checkpointing for the physics linearized using Tapenade. Finish building and test a blended parallel approach Blended parallel approach FV3 has a halo of three grid points around each patch 5400 in operations These are potential target resolutions for inner loops of an incremental 4D-Var. m 0k 0

1 Equivalent to range of ECMWF inner loops. 50km nalysis) a t n e r r u (c 25km 12km (Forecast) Using all the available cores is going to be key. However, seven halo points for every one compute grid point is not going to scale well. It will be helpful for dynamical cores to utilize OpenMP as well as MPI domain decomposition. Some balance then needs to be struck between the number cores directed to each. Fortunately FV3 uses MPI and has OpenMP embedded. Tapenade supports OpenMP but we havent tested it yet. Summary Developing 4D-Var for a modern dynamical core has proven to be a significant challenge, due to the complex code structure, small time step and large memory requirements.

After almost a decade of use in the GEOS forecast model efficiency gains to FV3, paired with new software and hardware technologies, is making the possibility of FV3-based hybrid 4D-Var a reality. A key new technology is the JEDI data assimilation system, which offers the ability to perform DA on the native grid, avoiding costly grid transforms. JEDI is being designed from the bottom up ensuring that itll be extremely efficient and scale well. Recent advances in automatic differentiation and custom built wrappers facilitate short development lead times, making dynamical core adjoint development less of a burden (physics is another matter). 4D-Var for the next generation of models is possible but challenging, and highly interdisciplinary. Advanced dynamical core developers should be aware of DA needs and the adjoint should be developed in tandem with the nonlinear model. FV3 in JEDI FMS + FV3 GEOS/FV3GFS JEDI LAYER (model agnostic) TLM & ADJOINT OOPS

innovations GEOMETRY Interpolation INCREMENT & STATE (FIELDS) Sol ver alg /line ins eb a tru ra r ctio ns MODEL LAYER CLASSES UFO Lo ca tio ns COVARIANCE MATRIX B & LOCALIZATION IODA

Recently Viewed Presentations

  • Snack Wise - Drexel University

    Snack Wise - Drexel University

    What is a Snack? A small amount of food eaten in between meals. Meant to provide and energy and nutrients to hold you over until the next meal. Can be part of a healthy eating style if you choose the...
  • The Structure of Drama - Carroll County Public Schools

    The Structure of Drama - Carroll County Public Schools

    Theme vs. Moral. Moral is a lesson or a principle contained within a play. Although some plays make moral statements, many plays have no particular moral. Review. Vocabulary Terms - know definitions. ... How does modern drama differ from traditional...
  • Ecology Quiz Review Empty slide to keep flashcards

    Ecology Quiz Review Empty slide to keep flashcards

    Ecology Quiz Review. Empty slide to keep flashcards in order. What is the study of living things and how they interact with their environment? ... Tropical Rainforest. Tropical Rainforest: Found along the _____ Temperature, rainfall (250cm/yr), humidity. Very high _____...
  • Sample Slides - Woodland Park Zoo

    Sample Slides - Woodland Park Zoo

    The roles of amphibians in ecosystems. Frog and toad larvae feed on organic debris and algae contributing to the breakdown of organic matter and controlling algae growth
  • Radical radicles: Flowers, fruits, & seeds Flower development

    Radical radicles: Flowers, fruits, & seeds Flower development

    Beans, peas, lentils, carob, peanut, tamarind, alfalfa Conifers & Cones Cones develop from female flower Seeds borne under bracts Have single wing Some cones are "serotinous" and open only after fire Forgotten seed cache Environmental triggers Temperature, moisture, sunlight Scarification:...
  • The Pillsbury Doughboy By Alleen Pace Nilsen And

    The Pillsbury Doughboy By Alleen Pace Nilsen And

    The Steel Workers' union accused automobile executives of trying to steal their benefits. 10. If you pull a tendon in your heel, it can take several weeks to heal (i.e. to regain your health). * Polysemes are spelled the same...
  • 323 Notes on Phonemic Theory in Terms of

    323 Notes on Phonemic Theory in Terms of

    323 Notes on Phonemic Theory in Terms of Set Theory 1. Notes on Phonemic Theory Here I will discuss phonemic theory in terms of set theory. A phoneme is a set of allophones which are not contrastive and share a...
  • ASF in Eastern Europe: achievements and perspectives

    ASF in Eastern Europe: achievements and perspectives

    ASF in Eastern Europe: achievements and perspectives. Ministerial Meeting, Tallinn, 26/02/16. Dr. B. Van-Goethem, Director . Crisis Management in Food, Animals and Plants. DG SANTE. ASF in the EU. Eradicated from. France (1974) Mainland Italy (1993) Belgium (1985) Netherlands (1986)...