Wave Forecast Verification
One of the most important activities of the ETWCH, in support of Met-Ocean Information and Maritime Safety Services, in particular, continues to be the Operational Wave Forecast Verification Project. A routine inter-comparison of wave model forecast verification data was first established in 1995 to provide a mechanism for benchmarking and assuring the quality of wave forecast model products that contribute to applications, such as safety of life at sea, ship routing, and, in general, the Global Maritime Distress and Safety System GMDSS. A Technical Report was produced on this activity in early 2006[Bidlot, J.R. and Holt, M.W., 2006: Verification of Operational Global and Regional Wave Forecasting Systems against Measurements from Moored buoys, JCOMM Technical Report No. 30].
The project now includes 17 centres (as of 2012), many running global wave forecast systems, with different wave models, different wind forcing, and different model configurations, and the goal is to continue to add new participants, including regional participants (where appropriate), and to expand the scope of the intercomparison as feasible.
It is recognized that centres engaged in wave forecasting benefit from this activity in the same way as weather centres benefit from the exchange of forecast verification scores.
Guide to Wave Analysis and Forecasting (WMO-No.702, http://www.jcomm.info/WaveGuide)
Status and future works
- Recent validation results: see the latest Summary Report
- Participating Centers (as of May 2012)
Overview and justification
A routine inter-comparison of wave model forecast verification data was first established in 1995 to provide a mechanism for benchmarking and assuring the quality of wave forecast model products that contribute to applications, such as safety of life at sea, ship routing, and, in general, the Global Maritime Distress and Safety System GMDSS.
This original inter-comparison was developed around the exchange of model forecast data at an agreed list of moored buoy sites at which instrumented observations of significant wave height, wave period and wind speed are available over the WMO GTS. Five centres (the ECMWF, Met Office (United Kingdom), FNMOC (USA), NCEP (USA) and the Canadian Meteorological Centre) routinely running global wave forecast models contributed to the original exchange. The initial results were presented during the WAVES97 meeting (Bidlot et al. 1998 )
The exchange was subsequently expanded with the addition of data from the Météo-France system in 2001. A paper discussing results from the exchange was published in 2002 [Bidlot, J.R, Holmes, D.J., Wittmann, P.A., Lalbeharry, R., Chen, H.S., 2002: Intercomparison of the Performance of Operational Ocean Wave Forecasting Systems with Buoy Data.Weather and Forecasting 17, pp. 287-310]. The Expert Team on Wind Waves and Storm Surges, during its first meeting (ETWS-I, Halifax, Canada, June 2003) noted the value of the exchange, and endorsed the further expansion of the scheme to include other wave forecast systems.
The exchange has expanded from the six centres participating in 2001 (ECMWF, Met Office (United Kingdom), FNMOC (USA), NCEP (USA), Canadian Meteorological Centre and Météo-France) to include Deutsche Wetter Dienst (DWD), the Bureau of Meteorology (BOM), Service Hydrographique et Océanographique de la Marine (SHOM) and Japan Meteorological Agency (JMA). All ten centres actively contribute data on a routine basis.
The mechanism for the data exchange is similar to the one set up for the original exchange. On a monthly basis, each centre provides a file of model data collocated with the buoy locations in an agreed format to the ECMWF, where the data are collated for subsequent access. The collated datasets are then processed to provide statistics for each centre at each buoy. Observation data are also collated at the ECMWF, and are quality controlled, with wind speeds adjusted to 10m height.A report on the intercomparison was submitted following the SCG-I meeting, and was published in 2006 [Bidlot, J.R. and Holt, M.W., 2006: Verification of Operational Global and Regional Wave Forecasting Systems against Measurements from Moored buoys, JCOMM Technical Report No. 30 ]. This Report provides a description of the project activity, and includes a full technical specification of the data exchange process.
The project was again discussed during the 10th International Workshop on Wave Hindcasting and Forecasting, in Hawaii in November 2007. A small summary of the project was produced as part of the workshop proceeding.
Review of validation data
The Wave Forecast Validation Project has originally focused on validation with buoy data only. However, other routine observations of waves are made and are available routinely. In evaluating the present project, the ETWS deemed it desirable to expand the project to include other observation sources. The initial focus is on providing data base entries for the data, so that the data can be archived. These data can then be used for both validation, and for the development of appropriate validation techniques.
The following alternate data sources, and corresponding requirements for wave model result archiving have been identified:
- Spectral wave observations from in-situ platforms, requiring wave model spectra to be archived at selected locations and times
- Spectral wave observations from remotely sensed sources, for instance SAR data, requiring wave model spectra to be archived at selected locations and times.
- Observations of wave heights and wind speeds from space-borne altimeters, requiring collocated wave model wave heights and wind speeds along altimeter tracks in both space and time.
It has also been observed that wave data over the ocean is notoriously sparse. This has two important implications for wave model validation.
- There is systematically insufficient data to provide spatial wave field analyses without a major impact of the first-guess wave fields used in such analyses. It is therefore misleading to validate wave models with analysed fields, since the features of such fields are dominated by the underlying wave model, and not by the observations.
- Even without the availability of objective analyses, much can be learned from a comparison of wave fields from different wave models. It is therefore useful to add the archiving of selected wave fields to project, to facilitate spatial wave field intercomparisons between wave models.
Finally, it should be noted that the equations used in wave models represent a strongly forced and damped system. Unlike models for predicting atmospheric and oceanic circulation, waves do not pose an initial value problem, and initial conditions and data assimilation are not critical for wave models. In fact, the basic behavior and quality of a wave model can be assessed solely by hindcasts studies, and is most clearly assessed by validating wave model results obtained without any form of wave data assimilation. Hence, it would be beneficial for centers that utilize data assimilation in their forecasts system to also provide hindcast model results obtained without data assimilation. To facilitate analysis of such model data and/or alternate model runs at centers, the original archive data format needs to be expanded to identify alternate model runs at participating centers.
Data dissemination policies
Some of the additional data to be considered in this project are proprietary. Also, some model data are not considered to be freely available to the general public by the centers involved. To assure that data dissemination policies do not hamper the project, raw data from the archives will be available to team members only. Derived products such as summary graphics will be made available through this web site subject to consent of members of the group.
Requirements for data formats
Data formats are central to any intercomparison work involving several centres, hence specification and implementation of appropriate data formats is key to the success of the project. The formats employed need to have the following characteristics:
- Backward compatibility: the centres involved in the exchange are operational centres, and the flexibility for updating the operational systems is often less than that for research systems. Timescales for all centres to conform to new requirements can therefore be long. In consequence, backwards compatibility of data formats is essential to avoid disruption to the exchange during the introduction of changes.
- Simplicity: format specifications should be designed to be as simple as possible whilst ensuring that the content is adequate for the purposes of the exchange. Simple formats are generally easier to implement, and reduce the risk of discrepancies in interpretation of the format specification between the participating centres.
- Flexibility: formats should be designed to be extendable allowing the future expansion of the exchange. This may include the potential to add additional parameters, additional meta-data, and increased data volumes.
- Precisely defined and strictly adhered to: to ensure that the work involved in setting up routine, automated processing of the data the formats need to be defined precisely, and participants need to ensure that their data conforms to the definition. Even small deviations from the definition can generate additional work within the subsequent processing of the data.
Specification of the data formats for the existing exchange and the proposed extensions are given in Annex 1 of the JCOMM Technical Report No.30 .
Outputs from all participating partners (fully operational forecasting centres) are compared to buoy and platform data as broadcasted to the meteorological community via the Global Telecommunication System (GTS). The different data sets from participants in the JCOMM Wave Forecast Verification are subsequently merged and made available to all participating partners for further evaluation (see Catalogue of wave data). Some examples, in graphical and tabular forms, can be found in the Monthly Reports on intercomparison of operational wave forecasting system processed at ECMWF:
- BoM: Bureau of Meteorology (Australia)
- CNR-AM: Consiglio Nazionale delle Ricerche (Italy)
- DMI: Danmarks Meteorologiske Institut (Denmark)
- DWD: Deutscher Wetterdienst (Meteorological Service of Germany)
- ECMWF: European Centre for Medium-range Weather Forecasts
- FNMOC: Fleet Numerical Meteorology and Oceanography Center (USA)
- JMA: Japan Meteorological Agency
- KMA: Korea Meteorological Administration
- METNO: Norweign Meteorological Institute
- MSC:Meteorological Service of Canada
- NCEP: National Centers for Environmental Prediction (USA)
- NIWA: National Institute of Water & Atmospheric (New Zealand)
- Puerto del Estado (Spain)
- SHN-SM: Department of Meteorology of the Naval Hydrographic Service (Argentina)
- SHOM: Service Hydrographique et Océanographique de la Marine (France)
- UK MetOffice