![Page 1: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/1.jpg)
Earth System Model Evaluation Tool (ESMValTool)Axel Lauer1, Yoko Tsushima2, Mattia Righi1, Veronika Eyring1, Alexander Löw3, Ulrika Willén4, and Mark Ringer2
1 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany2 Met Office, Exeter, United Kingdom3 Department of Geography, University of Munich (LMU), Germany4 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden
27 May 2015
![Page 2: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/2.jpg)
Motivation – Development of anEarth System Model Evaluation Tool
• Facilitate the evaluation of complex Earth System Models, e.g.
• quick look at standard diagnostic plots & output diagnostic variables
• easy comparison of new simulations (e.g. sensitivity runs or runs with new model versions) with existing runs and with observations
• Raise the standard for model evaluation
• include additional diagnostics of ongoing evaluation activities so that we do not have to start from scratch each time
• implement more observations, account for observational uncertainties
• Quick assessment of where we stand with a new set of model simulations via standard namelists that reproduce specific papers, reports, etc.
• Traceability and reproducibility
• Easy expendability
• synergies with ongoing projects to expand the tool (e.g. NCAR CVDP)
• useful for model groups & those analyzing models
• useful for model development
![Page 3: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/3.jpg)
Technical ImprovementsSchematic Overview of the ESMValTool Structure
![Page 4: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/4.jpg)
Development of an Earth System Model Evaluation Tool Within EMBRACE: DLR, SMHI & EMBRACE partners in collaboration with NCAR, PCMDI, GFDL
• Open Source: Python Script that calls NCL (NCAR Command Language) and other languages (e.g. R, Python)
• Input: CF compliant netCDF model output (CMIP standards)
• Observations: can be easily added
• Extensible: easy to (a) read models (b) process output [diagnostic] with observations and (c) use standard plot types (e.g. lat-lon map)
Current developments include for example• Essential Climate Variables, e.g.
- Sea-Ice- Temperatures & water vapor- Radiation- CO2- Ozone
• Tropical variability (incl. monsoon, ENSO, MJO)• Southern ocean• Continental dry biases and soil-hydrology-climate interactions (e.g. standardized precipitation index) • Atmospheric CO2 and NO2 budget• Aerosol, clouds• Stratospheric ozone
Goal: Standard namelists to reproduce certain reports or papers (e.g., IPCC AR5 Chapter 9, Massonnet et al., 2012; Anav et al., 2012; Wenzel et al. 2014; Eyring et al., 2013)
Release as open-source
software planned for later
this year.
![Page 5: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/5.jpg)
ESMValTool: A community diagnostic and performance metrics tool for routine benchmarking and process evaluation of Earth System Models in CMIP
Similar to Figure 9.24 of AR5
Similar to Figure 9.7 of AR5
Similar to Figure 9.5 of AR5 Similar to Figure 9.24 of AR5
CMIP5 MMM
CMIP5 MMM - OBS
Monsoon Precipitation Intensity and Domain
Similar to Figure 9.7 of AR5
Link to projections
Running along-
side the ESGF
![Page 6: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/6.jpg)
Considering Observational Uncertainty in the ESMValTool
Observational uncertainty needs to be considered in the assessment of model performance and should be an integral part of the evaluation tools
The ESMValTool is quite flexible and can consider observational uncertainty in many ways, e.g.:- using more than one observational dataset to evaluate the models
- showing the differences between the observational datasets in addition to the difference between the model and the reference dataset
- easily usage of an observed uncertainty ensemble spanning the observed uncertainty space, consistent with statistics
Example: HadCRUT4 surface temperatures, 5 (out of 100) realizations
![Page 7: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/7.jpg)
WCRP Grand Challenges: (1) Clouds, circulation and climate sensitivity, (2) Changes in cryosphere, (3) Climate extremes, (4) Regional climate information, (5) Regional sea-level rise, and (6) Water availability, plus an additional theme on “biospheric forcings and feedbacks”
Meehl et al., EOS, 2014
Goal
ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and CMIP6 simulations running alongside the ESGF
![Page 8: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/8.jpg)
Work as part of ESA CCI CMUG
• Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process.
• However, ESA data is currently not used in the context of routine model evaluation.
• Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities.
• CMUG will therefore establish a standardized model benchmarking approach for the CCI in close collaboration with the CMIP and WGNE/WGCM Climate Model Metrics Panel, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects – coordinated by DLR.
• The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years.
It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for obs4MIPs as well as the corresponding technical notes for obs4MIPs are created and submitted by the individual ESA CCI teams.
![Page 9: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/9.jpg)
WP 5.1 – Subtasks
1. Contribute to the community-wide development of an Earth System Model Evaluation Tool (ESMValTool) by developing code and integrating diagnostics and performance metrics into the tool
2. Ensure integration of ESA CCI datasets as well as other ESA datasets (in particular GlobAlbedo or other Glob-products) in the ESMValTool for routine model evaluation and benchmarking.
3. Provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4MIPs (http://obs4mips.llnl.gov/).
4. Provide recommendations of standardized benchmarking to the CMIP and WGNE/WGCM climate model metrics panels.
5. Visualize the results through a website to enhance visibility of the ESA CCI data. The ESMValTool will be extended with capabilities to visualize the resulting plots on a website.
6. Support the CCI Climate Research Groups in the use of the ESMValTool by providing a user guide with the goal that the CCI CRGs can use the tool in their own evaluation work.
![Page 10: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/10.jpg)
Goal: Performance Metrics calculated with ESA CCI Data
• Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Figure 9.8 of IPCC AR5 (Flato et al., 2013).
• A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference data set and if available an alternate observational data set for comparison.
![Page 11: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/11.jpg)
Baseline Proposal and Batch Options 1 and 2
Baseline Proposala) Atmospheric ECVs (aerosols) [DLR]
b) Terrestrial ECVs (soil moisture, fire, land cover) [MPI-M]
c) Ocean ECVs (SST, sea-ice) [DLR, MPI-M]
Batch Options 1 – O5.1d) Atmosphere: Ozone, GHG (CO2) [DLR], Clouds [Met Office]
e) Ocean: ocean colour [MPI-M]
f) Atmosphere, land and ocean [SMHI]
Batch Options 1 – O5.2g) Evaluation of a selected set of CMIP5 [DLR, MPI, Met Office] and CCI4MIP simulations [MPI-
M]
h) Using the results from the ESMValTool, contribute to a report that includes an assessment of:
i. how well state-of-the-art Earth system models simulate climatological mean, variability and trends of selected ECVs,
ii. how the new observational time series of the ESA CCI change the model evaluation and benchmarking assessment of the models in comparison to observations provided by obs4MIPs,
iii. whether observational uncertainty is still a limiting factor in benchmarking models.
Batch Options 2 – O5.1i) Atmosphere: GHG (CH4) [DLR]
![Page 12: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/12.jpg)
DLR contributions
What has been done so far
1. Maintenance and documentation (Eyring/Lauer/Righi) (WP5.1)
a) A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool.
b) A user guide is in preparation that will support the ESA CCI teams in the use of the toolc) Work on ESMValTool documentation paper startedd) Technical improvements are on-goinge) First release planned for fall 2015
2. Development and integration of ESA CCI data into the ESMValTool (WP3.6, WP3.7, WP5.1, O3.2, O5.1, O5.2)
f) Atmospheric ECVs: aerosols, Righi/Lauer), ozone and GHG/CO2 (Wenzel), clouds (Lauer/Righi)
g) Ocean: sea ice (Senftleben)
• Work with new ESA’s new 17 year ATSR climate data record (AOD) startedWill be continued in year 2, performance metrics plots will be provided for all CMIP5
models, AOD will be added (also WP3.7)• Cloud products (CLT, LWP) from non-ESA sources (also WP3.7)
• O3, CO2, sea ice (also for WP3.6): work with ESA CCI data will be started in year 2
![Page 13: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/13.jpg)
Technical ESMValTool workshop3-4 March 2015, Munich, Germany
• Present status and future perspectives of the ESMValTool
• Specific questions addressed
• What technical improvements can be made that would facilitate broad community development and usage?
• How to simplify ESMValTool integration with existing data repositories and the ESGF infrastructure?
• What are necessary steps towards a roadmap for ESMValTool enhancement for CMIP6 and beyond?
• How to prepare the software release and what is an appropriate license?
→ Technical improvements of the ESMValTool are supported also by ESA CMUG
![Page 14: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/14.jpg)
Technical ESMValTool workshop3-4 March 2015, Munich, Germany
• Timeline for submission of first ESMValTool documentation paper
• The workshop was of particular importance for the technical development of diagnostics of the ESA CCI datasets and their integration into the ESMValTool, and for technical requirements to run the tool on new CMIP data alongside the ESGF.
• Defining the further mid-term roadmap for ESMValTool development on the international level.
• A first release of the ESMValTool as open-source software is planned this year as a deliverable of EMBRACE. This release will form an important basis for the work on the ESMValTool as part of ESA CMUG and will allow to include the ESA CCI data in an efficient manner. From ESA CMUG, we are planning to include some selected diagnostics using the Aerosol CCI 17 year ATSR climate data record already in this first release.
→ Technical improvements of the ESMValTool are supported also by ESA CMUG
![Page 15: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/15.jpg)
Assessment of CMIP5 models with new ESA’s 17 year ATSR climate data record
As in Fig. 9.29 of IPCC Chapter 9.AOD integrated over the oceans for CMIP5 models,
MODIS and ESA-CCI
![Page 16: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/16.jpg)
Clouds20
-yr
aver
age
LWP
(19
86-2
005)
CM
IP5
in
com
par
iso
n w
ith
UW
isc
sate
llit
e cl
imat
olo
gy
![Page 17: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/17.jpg)
LMU contributions (Alexander Löw)
Technical work
• LMU has implemented a flexible python interface into ESMValTool that allows for integration of new diagnostics.
• LMU has developed a prototype for flexible automated testing of diagnostics within ESMValTool. This will maximize the quality of diagnostics across all ECVs.
• LMU has started to implement a soil moisture diagnostic to integrate ESA CCI soil moisture data (Loew et al., 2013) – to be finalized for first CMUG release of ESMValTool.
Administration
• LMU has so far not signed a sub-contract; draft of sub-contract was submitted by Met Office and is currently with LMU administration. Until signature and hiring of staff only minor progress can be expected.
![Page 18: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/18.jpg)
SMHI contributions (Ulrika Willén)
First tests using ESA-CCI SST with the ESMValTool
Example plot
![Page 19: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/19.jpg)
© Crown copyright Met Office
Metrics of Clouds and RadiationYoko Tsushima
ESA CMUG integration meeting May 2015
![Page 20: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/20.jpg)
© Crown copyright Met Office
Auto-assess:
• Aims: Primary aim is to produce a comprehensive set of metrics & diagnostics to inform model development.
• Current status:
• Scientific development currently frozen whilst code is re-written in Python using iris library.
• Aim to produce a first external release to UM user community in 2015.
• Currently assumes native UM file format (pp).
![Page 21: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/21.jpg)
Current & planned assessment areas
• Global tropospheric circulation• Conservation• Clouds & radiation• Hydrological cycle• Asian monsoon systems• MJO• ENSO• Processes over Africa• Tropical cyclones• Mid-latitude storm tracks and blocking• Stratosphere• Land surface• Aerosols• Ocean• Sea Ice
© Crown copyright Met Office
![Page 22: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/22.jpg)
© Crown copyright Met Office
Auto-assess metric plot
![Page 23: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/23.jpg)
© Crown copyright Met Office
RMSE metric based on cloud clustering of daily ISCCP simulator data
The error can be broken down into different cloud regimes, contributions of frequency of occurrence component and in-regime cloud radiative effects
Code provides as one of CFMIP diagnostics
Cloud Regime Error Metric for present day (CREMpd)(Williams and Webb 2009, Tsushima et al.,2013)
Tsushima et al 2013
![Page 24: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/24.jpg)
© Crown copyright Met Office
CF
MIP
1 -> C
MIP
5 Change
in in-cloud Net C
RE
error in regim
e Net C
RE
RM
SE
Improvements are seen in mainly in the radiative properties of optically thicker low and high topped cloud regimes rather than frequency of occurrence.
Change between CFMIP-1 and CMIP5 of in-cloud Net Cloud Radiative Effect (Net CRE) Error in Net CRE RMSE within daily ISCCP simulator cloud regimes in the Tropics
Tsushima et al 2013
Shallow Mid-level Thin Low level Convective Deep Stratocu Cumulus Topped Cirrus Transition Anvil Conv
![Page 25: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/25.jpg)
Cloud Feedback Model Intercomparison Project (CFMIP) CFMIP diagnostics code repository
Klein et al., 2012
Nam et al., 2012Qu et al., 2012 Sherwood et al., 2014
Tsushima et al.,2013, Williams and Webb, 2009Cloud Regime Error Metric for present day (CREMpd)
CFMIP encourages climate modelling and diagnostic groups to both use the existing metrics andprovide additional relevant codes.
![Page 26: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/26.jpg)
Summary and OutlookAn ESM Evaluation Tool is available that facilitates the complex evaluation of ESMs and
their simulations (e.g. submitted to international Model Intercomparison Projects such as CMIP, C4MIP, CCMI)
The tool is developed under a subversion controlled repository
‒ Allows multiple developers from different institutions to join the development team
‒ Broad community-wide and international participation in the development is envisaged
‒ Collaboration with the WGNE/WGCM metrics panel
Current extensions
‒ Atmospheric dynamics, biogeochemical processes, cryosphere and ocean
‒ Need for a wide range of observational data to be used with the tool
‒ Observations should ideally be provided with uncertainty and a technical documentation (e.g. similar to those from obs4MIPs) and in CF compliant netCDF
‒ Improving statistical comparison and visualization
Regular releases to the wider international community
‒ Further develop and share the diagnostic tool and routinely run it on CMIP DECK output and according observations (obs4MIPs) alongside the Earth System Grid Federation (ESGF)
‒ Release of ESMVal tool to the public → will contribute to metrics panel code repository
‒ Work towards a full community-tool developed by multiple institutions / countries
![Page 27: Earth System Model Evaluation Tool (ESMValTool) Axel Lauer 1, Yoko Tsushima 2, Mattia Righi 1, Veronika Eyring 1, Alexander Löw 3, Ulrika Willén 4, and](https://reader035.vdokument.com/reader035/viewer/2022081515/56649dc95503460f94abfbac/html5/thumbnails/27.jpg)
Thank you!