The Weather, Climate, Hydrological and Farming pilot aims to demonstrate pre-exascale scalability of state-of-the-art meteorological and climatological models developed by ECMWF and MPI-M research centres. In particular, we are targeting numerical weather prediction ensemble at 5km on a global-scale and Grand-Ensemble climatological simulations.

The pilot will exploit the ACROSS computing infrastructure and the FDB, a domain-specific object store developed at ECMWF, to support low-latency exploitation of the huge dataset generated by the meteorological and climatological simulations executed in ACROSS and of MARS, the world’s largest meteorological archive. The MARS archive currently holds over 300 PiB of primary data and is growing at a rate of 200 TiB per day.

To enable HPDA on multi-petabyte meteorological and climatological archives, we will build an environment for in-situ data processing, supporting user-defined data processing and analysis procedures. We will demonstrate the effectiveness of this approach by integrating ML feature detection, product generations and further application-specific post-processing such as large scale hydrological simulations performed by Deltares and regional downscaling of NWP and subsequent farming advisory services developed by Neuropublic.

Objectives:

1. Improve the existing operational system for global numerical weather prediction, post-processing and data delivery by exploiting hardware-acceleration and data streaming/object store techniques to demonstrate exascale scalability.
2. Enable low-latency exploitation of climate simulations by integrating data delivery trough domain-specific object store
3. Develop and demonstrate an environment for user-defined in-situ data processing. The system will enable HPDA on multi-petabyte meteorological and climatological archives and data streams to enable data analytic workflows that improve insight to data.