GEF-DRP: Testing of Indicators

Looptijd:
1 September 2006 – 1 March 2007

Betrokken medewerkers:
dr. J. Dogterom, drs. J.P.E.van Leeuwen

This study is a follow up of the GEF-Danube Regional Project Indicators fase I and II. In this study, the testing of a selection of the proposed core list of process, stress reduction and status indicators has been reported. The results show, that the GEF-DRP and ICPDR have generated in the past 10 years a huge library with data and information that in principle allow a retrospective evaluation of the impact of most of the activities of the project and the Danube Commission, but not for all.

Stress reduction indicators
With regard to stress reduction indicators, it can be concluded that sufficient and good historical data on investments (category 2 indicators, see annex II, report April 2004) are available in the DABLAS database and that sufficient and good historical data are available for calculations of loads of pollutants (category 3 indicators) in the TNMN database. They are available in a form that allow trend assessments over a period of 7 (DABLAS database) or 10 years (TNMN database). The situation for other categories is different. Information on category 1: implementation and enforcement of regional and national legislation and regulations, and on category 4: implementation of stakeholder involvement and public awareness raising programmes is not directly available. For category 1 stress reduction indicators, sometimes specific reports are available (industrial and agricultural policies; introduction and use of phosphorous detergents), but for others no information is readily available, e.g. the implementation of the EU nitrate directive. This is probably the case for the implementation of all EU directives. In principle, this information can be collected by the distribution of questionnaires to government officials and/or NGOs (see process indicators).

Status indicators
The TNMN database allows the assessment of practically all status indicators, with the exception of the quality of suspended solids and sediments. The indicators are reported in the TNMN Yearbooks and in special reports, like the Briefing Note to the GEF Council: Trends in nutrient loads from the Danube River and trophic status of the Black Sea, September 2006 (the figures in Chapter 3).

Process indicators
For process indicators, hardly any information has been systematically collected from the beginning of the GEF-DRP until now.

This study tried to collect information on a limited number of process indicators by sending web based questionnaires to GO’s and NGO’s. It has been difficult to obtain a sufficient amount of responses for a representative assessment of the process of implementation of the GEF-DRP and the JAP of ICPDR. The amount of responses by GO’s was disappointing (5 out of 78) and an other way to collect responses has to be applied. Maybe, it will be more successful, if the ICPDR distributes questionnaires through official letters to the governments of the member countries. The number of responses from NGOs was also disappointing (17 out of 180). Because of the low number of responses, statistical analysis was not done. The results presented should be regarded as indicative. The methodology was tested and the results are encouraging. It is expected that if enough responses are obtained, meaningful statistical analysis can be done. The questionnaires themselves may be improved by limiting the number of questions, although the average time used by the NGOs to fill in the questionnaires was about 20 minutes.

Conclusions

  • GEF-DRP and ICPDR have sufficient data and information to allow the evaluation of the GEF-DRP and JAP with regard to all status indicators (one exception: sediment quality) and of 2 out of 4 categories stress reduction indicators (investments, reduction of loads).
  • The information to evaluate 2 categories stress reduction indicators (category 1: implementation and enforcement of regional and national legislation and regulations, and category 4: implementation of stakeholder involvement and public awareness raising programmes) was not systematically available. For some category 1 stress reduction indicators, specific reports are available (industrial and agricultural policies; introduction and use of phosphorous free detergents), but for others no information is readily available. For category 4 indicators (stakeholder involvement and public awareness) no information was available.
  • For process indicators, no information was available.
  • A methodology to collect in retrospect the information on process indicators and category 4 stress reduction indicators was tested (web based questionnaires), but suffered from a lack of response by stakeholders (GO’s and NGO’s).
  • The number of responses from NGO’s allowed to present an indicative result on the performance of the GEF-DRP since 1996: all process indicators tested showed a positive trend with regard to the development of the capacities of NGO’s, increase of public awareness, access to information and public participation. Figure 24 is an illustration of the assessment by the NGOs and shows that the contribution of the GEF-DRP to this process since 1996 is regarded as improving and positive.
  • The indicative results of the NGO responses to the web questionnaire show, that in principle the methodology of collecting and processing information through web based questionnaires can work; it is crucial though to add a mechanism or an incentive to be sure that enough questionnaires are returned to allow statistical analysis.

Recommendations

  • In order to apply proper monitoring and evaluation procedures for GEF projects, it is necessary to produce consistent and systematized logical frameworks, including definition of the process indicators to be monitored during the implementation of the project .
  • At the start of a project, a core list of indicators (process, status and stress reduction) have to be agreed between the GEF management and the project staff .
  • Data and information has to be collected systematically on the selected core list of indicators from the beginning of the project to allow monitoring during the project implementation period and final evaluation.
  • GEF projects should from the beginning build up databases (good examples: DABLAS and the TNMN database) to allow monitoring and evaluation.
  • For process indicators and the categories 1 and 4 stress reduction indicators, regular and structured consultations of stakeholders should be organized (eg once every 3 years). These consultations can be done through questionnaires, like the ones tested in this study.
  • A mechanism has to be developed in order to receive enough responses to the questionnaires to allow statistical analysis. Possibilities are: (1) to give an incentive to the respondents, (2) make returning of questionnaires conditional to the receipt of grants (for NGOs only), (3) distribute questionnaires during meetings or conferences and not through the web
     
NHL.nlKennis en BedrijfLectoratenGEF-DRP: Testing of Indicators