SPECIAL REPORT No 15/98 on the assessment of Structural Fund intervention for the 1989-1993 and 1994-1999 periods together with the Commission's replies (submitted pursuant to Article 188c(4)(2) of the EC Treaty) (98/C 347/01)

1. INTRODUCTION

1.1. Under the 1988 reform of the Structural Funds (SFs), the Commission and the Member States were required to evaluate European structural policies. In 1993 this requirement was strengthened by the amendments to the Regulations that were introduced in the context of the revision of the SF reform (1994-1999) and the standard clauses of the new Community Support Frameworks (CSFs) and Single Programming Documents (SPDs).

1.2. Evaluation, an essential tool for the sound management of the SFs, is defined in the main provisions of the Regulations (1) as the evaluation of effectiveness on the following three levels:

(a) the overall impact of the structural measures on the objectives set out in Article 130A of the Treaty;

(b) the overall impact of the measures laid down in the CSFs;

(c) the specific impact of each operational intervention (programmes and other forms of intervention).

1.3. There are three main phases to this evaluation:

(a) prior appraisals (2), which enable a correct identification of the socio-economic problems that need to be resolved and the objectives to be achieved, on the one hand, and, on the other hand, of the strategy adopted and the means available for this purpose;

(b) ongoing assessments, which are carried out during implementation of a programme and make it possible to check whether the objectives are still appropriate and the desired results on the way to being achieved;

(c) ex post evaluation, which is carried out after the programme has been completed. Its aim is to analyse whether the results and impact obtained are to be considered positive, bearing in mind that the long-term effects of structural measures can only be appreciated several years after their completion.

1.4. The 1993 versions of the Regulations reiterate the responsibilities of both the Member States and the Commission with regard to assessment and stress that it is to be carried out within the partnership system (3). In this context, the Commission has taken the initiative concerning the supervision and general coordination of the implementation of evaluation and, in accordance with Article 16 of Regulation (EEC) No 2052/88, submits every three years an evaluation report on the progress achieved towards economic and social cohesion and, in particular, the contribution made by the Funds and national co-financing.

2. THE SCOPE OF THE AUDIT

2.1. The general aim of the Court's audit (4) was to check the progress achieved in evaluating the Community's structural interventions, in terms of systems and evaluation procedures and at a methodological level. The audit looked at the methods used to evaluate structural interventions both by the Commission and under the partnership system. It also examined the legality and regularity of the utilisation of the appropriations earmarked for evaluation in the budget of the European Communities and the quality of the financial management of these appropriations.

2.2. The audit also involved an appraisal of the techniques used for evaluating the socio-economic impact of structural measures. Although it did not aim to evaluate the programmes directly with the intention of producing new scientific results, it nevertheless made use of the results of work by experts so as to obtain a better understanding of the relationship between the economic policy instruments used and the results obtained.

2.3. Specific audits were carried out on the ex post evaluations of the CSFs and Operational Programmes (OPs) for Objectives Nos 1 and 2 for the 1989-1993 programming period and on the ongoing assessments of the CSFs, SPDs and OPs for the 1994-1999 programming period. All the ex post evaluations for the 1989-1993 Objective No 1 CSF were taken into account, as well as approximately 20 % of the ex post evaluation reports for the 1989-1993 Objective No 2 CSF and 15 % of the ongoing assessment reports for the 1994-1999 CSFs, selected on the basis of their geographical distribution and the nature of the interventions (CSFs, SPDs or OPs). In addition, the Court analysed 15 out of the 25 evaluation reports that dealt mainly with macroeconomic aspects. The audit was carried out at the Commission and in eight Member States (Belgium, Greece, Spain, France, Ireland, Italy, the Netherlands and Portugal), where the auditors consulted the assessors and national authorities concerned.

3. SUMMARY OF THE MAIN OBSERVATIONS

3.1. Evaluation developed and spread, at the Commission's initiative, mainly during the 1994-1999 programming period, in particular in the context of the ERDF, leading to its recognition in the field of management and programming. As from the beginning of this programming period, evaluation has been systematically applied to the first two phases (prior appraisal and ongoing assessment) and will be applied to the third at the end of the period (ex post evaluation) (see paragraphs 4.2-4.8).

3.2. Although the Commission has the figures regarding its own direct expenditure on evaluation, it does not always have access to all the information for expenditure on evaluations carried out under the partnership system (see paragraphs 4.9-4.11).

3.3. The Commission made a considerable contribution towards spreading the idea of evaluation, even though it did not introduce an adequate framework for evaluation indicators and methods in its programme on 'Methods for Evaluating Activities of a Structural Nature` (MEANS). In view of the complex nature of the methodological aspects of evaluation, there is no single methodology that can be applied to all specific situations. It is, however, important that the Commission should be in a position to be able to supply methodological frameworks that provide appropriate guidelines for the implementation of evaluations suited to specific situations (see paragraphs 4.12-4.13).

3.4. The quality of the indicators is closely linked to the reliability of the information gained from the monitoring of existing interventions and statistical bases. The Member States are responsible for these latter and they are often inadequate. The Member States should improve their quality (see paragraphs 4.14-4.21).

3.5. Many evaluations are completed late, which may proportionately delay the adjustment of programmes. Furthermore, the decision-making system could take evaluation results concerning the performance or effectiveness of intervention schemes better into account for future programmes (see paragraphs 4.22-4.27).

3.6. With regard to the objectivity of evaluations, the Commission makes no on-the-spot checks of the consistency and reliability of indicators or the systems that produce them (see paragraphs 4.29-4.30).

3.7. There are still significant shortcomings in the Commission's procedures for awarding evaluation contracts to consultants. The choice of consultants is not always clearly justified. Contract specifications are very general in terms of methodological techniques, indicators and the need to achieve results. This leads to delays and disturbances in the submission of evaluation reports, qualitatively variable analyses, and, sometimes, vague recommendations (see paragraphs 4.31-4.36).

3.8. The implementation of the ex post evaluation methodology for Objectives Nos 1 and 2 (1989-93) came up against statistical shortcomings, a lack of objectives, an absence of quantified forward indicators, an ineffective monitoring of impact and was also not sufficiently stringent. Nevertheless, these evaluations revealed that the SFs did indeed contribute Community added value, both in terms of the achievements, results and effects of the CSFs, OPs and measures and from the point of view of programming, organisation, monitoring, coordination, partnership and additionality (see paragraphs 4.15-4.19, 5.1, 5.4, 5.10-5.16).

3.9. With regard to the ongoing assessment of the 1994-1999 SF interventions carried out in partnership under the aegis of the Monitoring Committees, an examination of the physical and financial results always plays an important role in mid-term assessment, even though there may be variations in the priorities granted to the various evaluation objectives. The evaluation of the effects is hampered in particular by problems with indicators and the modelling of effects. This type of evaluation is still often only of relative value in terms of the decision-making process, the management of projects, reprogramming and the preparation of future interventions (see paragraphs 4.16-4.21, 6.5, 6.7, 6.14-6.26).

3.10. In the specific case of Objective No 2, the evaluation revealed the following weaknesses:

(a) a mismatch between programme implementation and Community funding, with significant carry-overs of appropriations from one programme to another (about 12 % on average for the shift from the 1994-1996 period to the 1997-1999 one)(see paragraph 6.27);

(b) the succession of short programming periods does not permit an optimum use of evaluation results (see paragraph 4.23);

(c) indirect effects are still not measured sufficiently to determine the impact of structural interventions, in particular as regards employment (see paragraphs 5.12-5.15).

3.11. Despite failings in the programming, implementing and monitoring systems, the macroeconomic evaluations revealed positive results, in particular in respect of the impact of structural measures (increases in GDP, effects on supply and demand, improvements in terms of employment and convergence and the effectiveness of the CSF instruments), (see paragraphs 7.1-7.14).

3.12. Evaluations of sub-programmes and measures continue to be inadequate. Furthermore, they come up against technical difficulties caused by the dispersion and inconsistency of many measures and sub-programmes (see paragraph 7.17).

4. GENERAL ISSUES

Introduction

4.1. The aim of evaluation is to determine the socioeconomic effects of the measures implemented, i.e. to examine the results obtained, compare them with the resources invested and analyse their cost. Furthermore, all these points must also be looked at in relation to forecasts. However, this evaluation task presupposes a good statistical basis, quantified objectives established on the basis of a system of indicators and the development of a methodological and analytical framework. At the same time, it must be supported by an analysis of the relationships between the means used, final results and intermediate results. The Court found failings at all these levels, and, despite real progress that has helped consolidate the system of evaluation, these have so far limited the usefulness of the evaluations carried out in connection with the Structural Funds.

4.2. Since the 1988 reform, evaluation has been explicitly included within the partnership system (Article 26 of Regulation (EEC) No 4253/88 - described more fully in the amended Regulation of 20 July 1993). During the first programming period and up until 1995, evaluations were nevertheless mostly carried out at the Commission's initiative (5). These evaluations - in particular the ex-post ones - were hampered by a lack of specific development strategies and inadequate monitoring. Moreover, generally speaking, they were not really taken into consideration and discussed in detail by the Monitoring Committees.

4.3. The Commission initially concentrated mainly on setting up its own evaluation mechanism for all the CSFs covering the reform's five priority objectives (6). Among the Commission's initiatives in connection with the development of evaluation and evaluation methods, mention must be made, in particular, of the 1992 creation of the technical group on the 'Evaluation of Regional Policies` and, at a methodological level, the launching of the programme on 'Methods for Evaluating Activities of a Structural Nature` (MEANS) in 1994.

4.4. For the 1994-1999 period, the Commission was responsible for a new evaluation system, which is now systematically applied to the first two phases (prior appraisal and ongoing assessment) and will later be applied to the ex post phase.

4.5. Although ongoing assessment is not explicitly provided for in the SF Regulations, as of 1995-1996 it has been applied to all the CSFs, SPDs and OPs for the 1994-1999 programming period. In concrete terms, it falls under the partnership system, allowing the Monitoring Committees to turn it into an integrated management tool and thus contributing towards a very broad awareness of the idea of evaluation. The Commission set up a system for the phased implementation of ongoing assessment in the Member States (7).

4.6. However, this system did not specify the actual procedures to be used (the definition of project specifications, the criteria for the selection of assessors, the structure of evaluation reports, data collection, quality standards, timetables, etc.).

4.7. At the same time, the Commission organised a series of events accompanied by a number of publications (8). By bringing the SF managers together with experts in the field of evaluation, these events contributed to the development of the idea of evaluation and helped raise the managers' awareness of this subject. Evaluation was presented primarily as a process to be learnt by experience on the job.

4.8. In May 1998, in the context of the preliminary draft budget of the European Communities for the financial year 1999, the Commission presented its 'evaluation report 1996/1997`. In this working paper, it describes the evaluation activities carried out in 1996-1997, gives a list of evaluations already implemented and currently underway and supplies a summary of the results of the evaluations implemented in the 1996-1997 period.

4.9. Expenditure on evaluation work may either be financed directly by the Commission, or co-financed with the Member States. Tables 1 and 2 show expenditure directly settled by DG XVI in the field of evaluation for the 1989-1993 and 1994-1999 programming periods respectively.

4.10. During the 1989-1993 period, as Table 1 shows, around 10 Mio ECU was spent on evaluation. Around two thirds of this expenditure occurred in two fields which come directly under the Commission's jurisdiction and for which it is directly responsible. These were, on the one hand, the Community initiatives under Article 11 of Regulation (EEC) No 4253/88, amounting to 3,1 Mio ECU, and, on the other hand, cross-disciplinary and thematic evaluations amounting to 3,2 Mio ECU, such as those concerning SMEs, telecommunications, the environment and human resources. However, the amounts earmarked for the evaluation of Objectives Nos 1 and 2 of the Structural Funds were relatively small, given the fact that, during this period, partnership was not very well developed in the field of evaluation. As is shown in Table 2, the amount earmarked for evaluation for the 1994-1999 period amounted to 25,2 Mio ECU in April 1998. This increase over the previous period was due to the development of ongoing assessment and its methodology within the partnership system.

4.11. With regard to spending in this field by the Commission and the Member States, which has increased considerably since 1996, the Commission is not always fully informed, in particular where the evaluations are included under the 'technical assistance` measures of Operational Programmes, which may also involve other forms of expenditure. A specific allocation to cover the needs of evaluation could be provided right from the initial programming phase, which would make expenditure more transparent.

Reference framework

4.12. The MEANS programme (1995-1998) (2,8 Mio ECU) has played a specific role in the process of the development of evaluation (9). The procedure applied aims to encourage the development of a uniform and consistent methodology, via an inventory of the methods used in the Member States and the identification of valid examples. This essentially empirical and exhortatory approach does not enable the Commission to decide on the actual procedures to be used for the implementation of evaluation methods or dictate a choice of indicators, in particular those for measuring the socioeconomic impact in the field of regional development. In view of the complexity of the methodological aspects of evaluation, there is no single method applicable to every specific situation. It is nevertheless important that the Commission should be in a position to supply methodological frameworks to enable appropriate guidelines to be drawn up for the implementation of evaluations suited to specific situations.

4.13. So far, the results of the MEANS programme have been used to lay the foundations of a new evaluation procedure, in particular by introducing a common language, and have enabled the preparation of ongoing assessments within the partnership system. Tables 3 and 4 show the generally accepted evaluation procedure. However, the MEANS programme has not yet provided detailed answers to the questions regarding the practical and operational aspects of evaluation. Thus, two fundamental questions remain unanswered: firstly, what is the most appropriate system of indicators for use in formulating an overall assessment of a given Community structural programme; and, secondly, what evaluation approach is most suitable for measuring the socio-economic impact of a given structural programme in the context of regional development? The Commission is not yet able to draw up precise instructions regarding the actual techniques to be applied in each specific evaluation situation. The absence of such instructions is felt particularly during the ongoing assessments of intervention for the 1994-1999 programming period.

Statistical bases and information concerning objectives

4.14. Evaluations (whether prior appraisals, ongoing assessments or ex post evaluations) are often confronted with objectives that have been formulated too generally, even with regard to the objectives of the measures in question.

4.15. The level of development of existing statistical bases is an impediment to the monitoring and evaluation of programmes and measures co-financed by the Structural Funds. Shortcomings are evident at various points, including, in particular, the lack of statistical coverage limited to the areas concerned, the absence of statistics regarding certain aspects and the collection of information at intervals that are inappropriate in relation to the needs of monitoring and evaluation.

Indicators

4.16. The existence of reliable and appropriate systems of indicators is an essential element in the planning process. For evaluation purposes, the structural programmes must contain information on financial and physical achievement as well as on results and performance. These indicators should provide managers with information on the immediate or long-term effects of action taken and also provide a measurement of the progress achieved in relation to the overall objectives of the programmes (e.g. regional growth or the reduction of unemployment).

4.17. The reliability of the financial indicators is closely linked to that of the expenditure certificates sent to the Commission prior to the relevant Community payments. Physical indicators are often considered to be a source of supplementary information and their reliability is not consistent. Existing databases do not always provide a record of actual physical progress and final recipients do not always provide managers with periodic information about the physical state of operations. The ongoing assessment of the Lisboa e Vale de Tejo OP in Portugal reported physical indicators in strict proportion to expenditure without mentioning any variations linked to delays in implementation or other causes. The Member States should improve their monitoring systems and the databases on which their indicators are based.

4.18. Both the measurement of results and impact at sub-programme and measure level and the measurement of impact at programme level are hampered by the fact that neither the causal relationships between the various levels of objectives and between objectives and measures/projects nor the horizontal effects of complementarity and synergy between measures and between objectives can be identified with any certainty. It is therefore difficult to define the specific effect of a given measure on regional development or the impact of investment on demand or, a fortiori, on supply. In this context, with regard to the 1994-1999 CSF, the search for indicators that will serve as an appropriate tool for evaluating results and impact has still not been completed (see paragraph 6.23).

4.19. The prior appraisal of the CSFs is essential to subsequent evaluations. It ensures that all the result and impact indicators have been specified and quantified, thus enabling a judgement to be made on the estimated impact of the programme concerned and comparisons to be drawn with actual effects. In this context, the management by objectives approach has not always been applied in a uniform or satisfactory fashion, which has had an impact in terms of the indicators necessary for evaluation. Thus, the Italian 1994-1999 CSF contains a small number of indicators, a fact which could hamper subsequent evaluation. On the other hand, the 1994-1995 Portuguese CSF, which was approved by the Commission without any major changes, includes too many indicators (about 300), which might also complicate evaluation. This variation in the establishment of indicators from one CSF to another highlights the lack of consistency in programming. Where the system of indicators approved by the Commission during prior appraisal contains a number of general and unquantifiable objectives, they are sometimes revised for purposes of ongoing assessment because they are found to be badly defined and insufficiently functional.

4.20. The structural interventions and the socioeconomic situations upon which they are brought to bear give rise to a wide variety of indicators that are often developed locally. Although partial lists of indicators have been sent to certain regional or national authorities and have already been the subject of debate, the Commission has not drawn up any lists of the indicators that it considers to be both essential and representative in terms of regional development and in connection with the various types of measure.

4.21. The quantification of result and impact indicators should be developed further in connection with monitoring, and, at the same time, this quantification should also be reliable. The fact is that it has been known to be unreliable, thus failing to prevent cases where new jobs were counted twice or to allow a distinction to be made between the gross and net effects on employment. This was discovered by comparing the ongoing assessment report of the Brandenburg OP with the statistics concerning the people registered at the Social Security office.

Taking evaluation into account

4.22. The contribution of evaluations in terms of a more efficient utilisation of Community funds largely depends on the quality of their content, and on when the results are received and decisions are made regarding changes in strategy. However, a number of ongoing assessments for the CSFs, SPDs and OPs for the 1994-99 programming period were finalised rather late, thus delaying possible changes.

4.23. With regard to Objective No 2, the succession of shorter programming periods (as compared with Objective No 1) means that it is particularly unlikely that the dates when programming decisions need to be taken will coincide with those when evaluation results become available, thus making it impossible to learn from the past and adapt the subsequent interventions. The Objective No 2 programming and evaluation cycles are, to a certain extent, out of step. For example, the final version of the summary report on the ex post evaluation of Objective No 2 for the 1989-1993 period was submitted to the Commission in the summer of 1997, while the prior appraisal for the 1997-1999 programming period was underway and the ex post evaluation for the 1994-1996 period had not started.

4.24. Budgetary appropriations for structural programmes and subsequent programmes are rarely influenced by the programme results shown up by the evaluation reports. During the first programming period the decision-taking system very rarely took any account of evaluations, and changes in programmes were generally based primarily on the degree of financial absorption. This was the case for the useful recommendations made by the head consultant in the context of the ex post evaluation of the 1989-1993 Objective No 1 CSF (see paragraph 5.7).

4.25. The more qualitative aspects of performance or programme effectiveness identified in the ongoing assessments or implementation reports did not have sufficient impact on the operational management of the SFs or on the development of the sense of responsibility of the partners affected by the development of the programmes.

4.26. Just as for monitoring, the results of the ongoing assessments of the 1994-1996 Objective No 2 SPDs, CSFs and OPs reveal a divergence between the financing plan and the implementation of the programmes. Because of this situation, the need to utilise appropriations becomes the main concern and as a result the importance of the role of evaluation in the analysis of the area in question may be under-estimated, especially as regards consistency between the objectives, the resources allocated and the impact of structural interventions.

4.27. The evaluations' conclusions and recommendations are often self-evident to the intervention managers, although they may be useful for other parties involved. This is the case for both ex post evaluations (see paragraph 5.8) and ongoing assessments under the partnership system.

The objectivity of evaluations

4.28. The lack of a precise frame of reference is a hindrance to the application of suitable techniques that can be objectively applied in specific evaluation situations so as to produce easily verifiable results and guarantee the independence of the assessors. On occasion, however, the recommended method is not objectively justified, or assessors are all too willing to adapt their approaches to the difficulties encountered or select indicators on the basis of the information available.

4.29. Furthermore, independence from managers does not always appear to be guaranteed, particularly in the field of information (see paragraph 6.11), while evaluation reports do not always give the source of the information on which their observations are based. This failing may raise doubts as to the reliability and validity of the information submitted in the evaluation reports.

4.30. Objectivity should be guaranteed by a verification system based on the quality of the indicators (of implementation, results and impact) used, the reliability of the basic information and the degree of implementation of the methodology. However, the Commission departments analyse the information and results submitted under the evaluation procedure, but do not carry out any on-the-spot checks or verification as to how well the evaluation systems used in the Member States actually work.

The evaluation contracts awarded by the Commission

4.31. The audit of the Commission's procedures for awarding evaluation contracts showed up a number of shortcomings - unsatisfactory selection procedures, excessively vague specifications, delays and low quality reports.

4.32. The minutes of the meetings concerning calls for tender do not always specify the judgemental and analytical factors that determined the Commission's choice of consultants. The Court's audit did not provide evidence to show that the best experts from the most appropriate fields of competence were selected.

4.33. In the case of the ex post evaluation of Objective No 2 and all the 1989-1993 CSFs for Objective No 1, the selection procedure was used to appoint a chief consultant accompanied by other national consultants and coordinating consultants, who then subcontracted the evaluation work to other consultants. In this chain of consultants, the Commission may lose its control of the evaluation procedure, which is supposed to guarantee the consistency, uniformity and quality of the results.

4.34. Specifications are sometimes vague in terms of the description of the work to be carried out and the obligation to achieve results. The wording of the contracts is very general. Often, procedures for carrying out the evaluation work are left open pending a further agreement with the Commission on the methodological framework.

4.35. There are often delays in respect of the schedule for submitting reports. Thus, in the case of the evaluation of all the 1989-1993 Objective No 1 CSFs, the contract provided for the report to be submitted on 28 February 1995, when, in actual fact, it was not completed until November 1995. The Commission provided for no penalties for failure to comply with contractual obligations on deadlines and, on the contrary, made the final 40 % payment on 11 July 1995 under the terms of Article 4 of the contract.

4.36. The contracts do not contain clear, precise guidelines as to the methodology to be followed and the results expected. In consequence, the Commission departments may react to the reports in different ways. For example, some of the reports had been approved, even though they were no more than simple descriptions or vague analyses of the usefulness of the work carried out, were based on general statements by the persons questioned, or produced over-general recommendations. In the case of the ex post evaluation of the 1989-1993 Ireland CSF, one department recommended terminating the consultants' contract because of the bad quality of their report, whereas another department approved it. In the case of the ex post evaluation of the 1989-1993 Portuguese CSF, the departments concerned agreed that, rather than providing an assessment, the report did no more than reiterate the aspects of physical implementation already presented by the national authorities. Similarly, it was pointed out that the ex post evaluation report for the 1989-1993 Spanish CSF was too descriptive and that it was not sufficiently analytical. The ex post evaluation of the 1989-1993 Northern Ireland CSF was judged to be incomplete. The report on the ex post evaluation of the 1989-1993 Italian CSF was considered to be badly structured and organised. Although the evaluation report for the 1989-1993 Greek CSF has chapter by chapter conclusions, there is no general conclusion.

5. THE EX POST EVALUATION OF OBJECTIVE Nos 1 AND 2 CSFs (1989-1993)

Introduction

5.1. Assessors are often faced with the following difficulties:

(a) initial programming that has often been carried out hastily and without a prior appraisal of the programmes, in particular as regards intrinsic economic fundamentals, especially with regard to Objective No 1 regions.

(b) the absence of quantified objectives and result indicators;

(c) gaps in the statistics with regard to the socio-economic situation both before and after Objective No 2 SF intervention, in particular with regard to the industrial fabric, investments and gross regional product. The statistical tools used do not always correspond to the area selected for analysis (boroughs, districts, employment catchment areas);

(d) the fact that certain measures have not been completed;

(e) the existence of exogenous factors that help explain the development of the socio-economic situation;

(f) where the allocation of aid is not linked closely to regional or local needs, the heterogeneous nature of the projects and their objectives;

(g) it is hard to identify the intrinsic effects of the measures and programmes, because of the barrage of measures aimed at the same region or area;

(h) the difficulty of measuring the indirect effects of a programme (in particular as regards employment), because no system of observation was set up for this purpose when the programmes were launched and no detailed inquiry has been carried out on this subject since their completion.

The ex post evaluation of the Objective No 1 CSFs (1989-1993)

5.2. The ex post evaluation of the Objective No 1 CSFs was coordinated by a central consultant and comprises the ex post evaluations of the CSFs concerned that were carried out by national consultants between January 1994 and June 1995. The consultant responsible for coordination developed the methodology that was to be applied, which made use of analytical methods, at both macroeconomic and measure/project levels. The approach followed aimed, firstly, to analyse the strategy and intrinsic logic of the CSFs, OPs and measures, secondly, to provide a preliminary evaluation of physical progress and impact, and, finally, to examine the implementation systems. This procedure is summarised in Annex I.

5.3. Although generally consistent, the proposed methodology does have shortcomings. Thus, it provides that analysis of the intrinsic logic and strategy of the CSFs, OPs and measures should be based on a uniform structure of objectives, linked to a standard classification of activities. This approach might allow comparison between Member States, but it does not necessarily allow objectives to be listed in order of importance on the basis of both the Member States' priorities and the criteria of hierarchy, complementarity, substitutability and synergy. Nor does it allow the identification of all inconsistencies in the strategies specific to each of the CSFs and OPs. The methodology is supposed to provide answers to relevant questions about intentional and unintentional effects, but, in many cases, the physical and impact indicators given in the methodological paper are unable to provide them.

5.4. The Court's audit also found the following shortcomings:

(a) as regards the development of the socioeconomic environment of the CSFs and OPs, a comparison was not always made between the initial situation upon which the Community intervention was to act (1988-89) and the situation at the end of it (1993-94). Data were sometimes out of date, particularly those concerning regional differences;

(b) there was often insufficient analysis for the ranking of objectives, which is required by the methodological guidelines. Either it was not done at all, or it was only done partially -either just for those OPs and measures that were selected for specific examination or without systematically applying the structuring criteria. As a result, the ranking of objectives and instruments sometimes did not produce any improvement in the identification of problems of consistency and the relevance and relative importance of priorities;

(c) adjustments are sometimes made to CSFs and OPs without a thorough analysis of the causes;

(d) financial implementation, implementation systems and physical implementation are generally well dealt with in the evaluation reports. However, in certain cases, either the effects of programmes or measures were not examined except in terms of job creation, or the results were formulated in non-quantitative terms. Sometimes the only quantification of effects is provided by applying macroeconomic models and is concerned with increases in GDP, job creation and competitiveness. This shows the inadequacy of the data provided by managers in the context of monitoring systems;

(e) bottom-up analyses did not generally allow conclusions to be drawn in terms of impact at programme level;

(f) Community added value is sometimes only expressed in qualitative terms (for example, the taking into account of the programme logic, the implementation of an integrated strategy or the effect on the organisation process). Furthermore, it was not always investigated systematically;

(g) anticipated long-term effects were not discussed or were discussed only in qualitative terms.

5.5. The Spanish Objective No 1 CSF was the subject of both an ex post evaluation ordered by the Commission and another one organised by the national authorities and co-financed by the Communities. Both studies were drawn up during the same period and pursued the same objectives but were conducted independently and separately.

5.6. With regard to results and impact, the ex post evaluations highlight the physical achievements of the Objective No 1 CSFs, the macroeconomic impact of the CSFs observed in certain national evaluations, and the progress made in the field of planning -although in this field, the programme-based approach encountered certain obstacles (10). They also stress the CSFs' contributions in terms of innovative organisational structures, financial management, monitoring, information systems, coordination and partnership, although with substantial progress still being needed at each of these levels.

5.7. The chief consultant's 1995 summary included some useful recommendations that the Commission could not follow up completely. One of these was that, when programmes were being drawn up, a clearer distinction should be made between Community structural interventions designed to encourage endogenous growth factors and those aimed at the restoration of a regional equilibrium or a raising of living standards. Another concerned striking a balance between the priority given to infrastructure investments and specific regional characteristics that needed development or finding an equilibrium between the fight against unemployment and the need to increase productivity and competitiveness. Finally, it recommended that regional policies should pay more attention to employment.

5.8. The ex post evaluation of all the 1989-93 Objective No 1 CSFs carried out by the national assessors mostly concentrated on the improvement of the Member States' monitoring and management systems rather than on a real exercise in the assessment of the impact of the SFs.

Ex-post evaluations of Objective No 2 (1989-1993)

5.9. The ex post evaluations of the Objective No 2 programmes for the 1989-1993 period were coordinated by an external assessor in the 60 eligible regions. A general analysis was made for each of the regions, while detailed studies were made of 22 regions. The main aim of these evaluations was to analyse the impact of these structural interventions on the industrial redeployment process in these regions and draw lessons for the next programming phase. A summary of this evaluation approach appears in Annex II.

5.10. Although a common framework was envisaged, the approaches adopted are very heterogeneous. This concerns particularly the priorities adopted and the depth of the analysis, whether it is descriptive or critical, and the methods and techniques used.

5.11. Analysing strategies and the appropriateness of the CSFs and OPs and the changes that they have undergone in relation to the socio-economic environment seldom leads to a reformulation of the structure of the objectives, sub-programmes and measures in accordance with the logic of regional development or redeployment and on the basis of the criteria of ranking, complementarity and substitutability. Whereas many strategic analyses based on the structure of the CSFs and OPs are partial and uncritical with regard to the strategy concerned, a reformulation of this kind should, after due elaboration, make it possible systematically to identify the strategy's shortcomings in terms of consistency, the weighting of priorities, synergy, complementarity and effectiveness vis-à-vis the socio-economic environment.

5.12. As for the impact of the SF interventions, the assessors are not able, on the basis of a sample of projects and an examination of their results, added value and additionality, to aggregate their observations in terms of individual measures and the socio-economic impact of the programme concerned.

5.13. Few impact evaluations were based on a genuine combination of top-down and bottom-up approaches. This combination was, in particular, used for the evaluation of the Objective No 2 interventions in Tuscany. In this case, the top-down approach used a bi-regional (Tuscany/rest of Italy) econometric model to estimate the overall impact of an increase in demand on production by sector, consumption levels and added value, both in Tuscany and the rest of Italy. This model also makes it possible to estimate the multiplier effects by type of intervention (infrastructures, services for businesses, aid for private investment) and the effects on employment (11). The direct effects on employment were estimated using a technique of standard coefficients concerning the various types of intervention. This top-down approach therefore regarded direct short-term effects related to demand. The bottom-up approach, which is based on a sample of projects and measures, is more capable of reflecting long-term, more formative and indirect effects, relating especially to supply and external savings that enable reductions in production costs. Even though these effects are more difficult to isolate and quantify and this approach does not enable an aggregation of the effects in terms of individual measures and programmes (which, however, was the objective of detailed evaluation), qualitative considerations as to the effectiveness of the measures and the degree of additionality of Community interventions constitute a useful complement to the top-down approach. A combination of both approaches can provide real added value in terms of analysis, in particular if it is based on the techniques of modelling or formalisation.

5.14. Impact evaluations that make use of systematic inquiries (such as a number of investigations concerning the additionality of measures or the application of a specific method for estimating net measure-related job creation by comparing an experimental group with a control group) are also rare. However, they do provide real added value in terms of impact evaluation. The effectiveness of measures and programmes would have been more accurately evaluated if assessors had actually conducted the statistical analyses of certain result indicators that they had sometimes said that they wished to carry out. Thus, it is not always possible to find data on the number of companies located in new industrial zones set up with the help of SF aid, or on technological transfers from research establishments to businesses. The meetings with measure and project managers that were generally provided for in the evaluation procedures only seldom made it possible to obtain precise information on the physical implementation, results and impact of the operations.

5.15. In the field of job creation, the results are not clear in terms of the durability of the jobs and the specific contribution ascribable to the structural measures. In addition, the data produced in the evaluation reports is sometimes contradictory.

5.16. With regard to the impact of the Objective No 2 programmes, we can draw the following conclusions from the ex post evaluations:

(a) although the Objective No 2 OPs do not often bring about an improvement in the socio-economic situation, they at least contribute towards avoiding further deterioration (in particular as regards the maintenance of employment (12)) and preparing the ground for industrial redeployment. Indeed, most of the regions were seeking to diversify their economies and their industrial bases and, where possible, cast off their reputations as exclusively industrial areas;

(b) despite the integrated procedure, based on the programme concept, the OPs sometimes consist of a modification of national and regional policies that have already been set up and do not always constitute a programming tool;

(c) the simultaneous existence of the Objective No 2 OPs and the Community initiatives gave rise to confusion and to the overlapping of aid;

(d) SF intervention had positive effects in terms of programming, implementation methods, partnership, synergy, coordination and evaluation. However, the lack of synergy and communication between ERDF and ESF managers and measures was often stressed;

(e) as regards additionality, the SFs make it possible to tackle new target areas, carry out projects which would have remained unrealised without Community aid, speed up implementation, carry out more ambitious projects or give a formative character to initial projects. The additionality of different types of measure (aid for infrastructures, business services, aid for private investment) may vary considerably;

(f) the SFs do not often act as a lever for the involvement of private money or, at least, the evaluations omit this aspect;

(g) the evaluations stress that studies, preparatory negotiations, administrative procedures relating to the adoption and implementation of certain projects and the integration of new national policies all involve delays that are often underestimated.

5.17. With regard to management, the evaluations show that certain techniques used for monitoring the programmes were inadequate, and that the internal changes made during the programming period mainly concerned the maximisation of the rate of absorption of appropriations rather than the revision of regional development priorities.

6. ONGOING ASSESSMENTS (1994-1999)

The development and integration of the principles of ongoing assessment

6.1. Ongoing assessments are provided for in the chapter on implementing provisions, dealing with the improvement of monitoring (included in each of the CSFs, SPDs and other forms of intervention). It is stressed that ongoing assessments involve a critical analysis of the data collected during monitoring, explain any differences in relation to the original objectives and assess the intended results of the measure in question. They also contain an opinion on the validity of the current intervention and the relevance of its objectives.

6.2. In the letters that it sent to the Member States during the financial year 1995, the Commission stressed that ongoing assessments were to provide the Monitoring Committees with appropriate information as to whether the programmes were running smoothly, put forward possible reprogramming measures and present an independent, objective picture of the progress made. In the initial phase, it was also proposed that they should concentrate on the operating conditions of the interventions and, in particular, on the indicators, the management abilities of the persons in charge and the procedures used for selecting projects. Subsequently, with a view to proposing necessary changes and adapting programming, evaluation reports were to look at the relevance of the strategies employed, the validity of intervention objectives, the effectiveness of the programmes in terms of the work carried out, the initial results in relation to objectives, the efficiency of the programmes, the correct application of Community policies (environment, public contracts, etc.) and the measurement of results as regards employment and SMEs.

6.3. In addition, the Commission suggested that evaluations should be general and that thematic evaluations concerning one particular sector or geographical area should be developed in a later phase.

6.4. Certain Member States, in particular Spain (13), France (14), Ireland (15) and the Netherlands (16), also drew up frameworks for ongoing assessment. They all feel that ongoing assessments should look at the following:

(a) the development of the socioeconomic context;

(b) strategies and internal and external consistency;

(c) the implementation conditions, including programming and selection criteria, administrative organisation, the monitoring system and project financing channels;

(d) implementation and the results achieved, taking into account intended results;

(e) initial effects obtained, together with an appraisal of expected effects;

(f) costs, with a view to measuring efficiency, i.e. the relationship between expenditure and results.

It was also felt that, where necessary, ongoing assessment, should also lead to proposals for changes in respect of priorities or intended measures.

6.5. The partnership system was introduced in 1988 as a means for ensuring the effectiveness of the SFs, but it has only played its full role in the system of decentralised management since the start of the ongoing assessment of all the 1994-1999 CSFs in 1995. On this occasion, it was decided that the Monitoring Committees should be responsible for establishing the objectives of evaluations and the required quality standards needed to guarantee the overall consistency of the process, adapting specifications to the characteristics of the regions in question whilst respecting the objectives of the exercise and selecting assessors on the basis of shortlists prepared by the persons responsible for evaluation (members of the Monitoring Committee and Community rapporteurs) after a public call for tenders. Ongoing assessments have made a real contribution to strengthening the partnership system, by allowing it to participate in the causal analysis of structural interventions. Tables 5 and 6 show the operational implementation of ongoing assessments and the significant work carried out by Member States under the partnership system in terms of identifying the impact of the SFs.

6.6. The partnership approach to ongoing assessment has also strongly influenced the institutional structures and the national/regional traditions of the public bodies responsible. It has made for more widespread knowledge of the idea of multiannual programming, the evaluation of which is essential to drawing up critical assessments of structural interventions and preparing any necessary adjustments. In the field of evaluation, partnership is particularly well developed in connection with Objective No 1.

6.7. At present, the aim of ongoing assessment work is to supply overall solutions to all involved, from programme designers to managers and political leaders. This work results in very bulky reports containing information from the programmes themselves, prior appraisals, annual implementation reports and ongoing or ex post evaluations. However, according to information gleaned orally during the audits, only some of the results and recommendations are of any real use in terms of assisting decision-making, managing projects, reprogramming or preparing the next generation of interventions.

Implementation procedures

6.8. Of the 389 intermediate evaluations undertaken, 320 concerned Objectives Nos 1, 2, 5b and 6 (17). The implementation of these evaluations is subject to significant delays in several Member States.

6.9. The intermediate evaluation process was originally envisaged as comprising two distinct stages. The first stage involved an assessment of the scope for evaluation and the identification of related problems. The second consisted of the evaluation itself and the subsequent drafting of a report. In practice, these arrangements were seldom applied.

6.10. As a rule, the Commission insisted on intermediate evaluations being carried out by external experts so as to guarantee not only the credibility of individual know-how but also the impartiality of conclusions about the legitimacy of the strategy, the consistency of socio-economic analyses and the objective quantification of the intended effects. Evaluation requirements in the area of the Structural Funds have created in the space of a few years a network of external assessors known to the Commission and to the Member States, which is advantageous insofar as it generates greater expertise in this area but which focuses on a specific market.

6.11. In practice, the Court's on-the-spot checks highlighted the limitations on the independence of the external consultant. Given the limited time available to the assessors and the consultant's de facto dependency on programme managers for the necessary information, the separation of the respective duties is not very rigorous. Too often, the results obtained in terms of performance merely reproduce and update information from the monitoring system while presenting it as an external contribution.

6.12. Furthermore, the usual procedure of an open invitation to tender based on publication in the Official Journal was not always applied with a view to selecting a consultant to carry out assessments. In the Netherlands, the procedure consisted of asking a limited number of consultants on a list of potential assessors to submit a bid. The selection was then made from these bids.

6.13. Although the Commission regularly participates in monitoring committee meetings and is able to win acceptance for its criteria for selecting consultants, its involvement in this process is inconsistent. Thus, in Belgium, Greece and Portugal the Commission is actively involved in the procedure for selecting assessors within the framework of the technical groups or the ad hoc technical accompanying committee which prepare the choice of consultant and monitor the corresponding implementation and evaluation. By contrast, in France the Commission is officially informed only once the two bids deemed to be the best have been selected; these latter are accompanied by a final proposal in respect of which the Prefect of the Region, in his capacity as contracting authority, asks for any observations to be brought to his attention. As regards the Netherlands, the Commission, which takes part in the work of the national accompanying committee which pilots ex ante, intermediate and ex post evaluations of the regions covered by Objectives Nos 1 and 2 and in the work of the monitoring committees, reacted only belatedly to the content of the draft intermediate evaluation report on the Flevoland Objective No 1 SPD.

Priority given to evaluation objectives

6.14. As regards the implementation of ongoing evaluations, the priority given to evaluation objectives varies, with the exception of the examination of the physical and financial results of the measures, which is an important aspect of any ongoing evaluation. Priority developments often correspond to a genuine regional need, such as in Corsica, where the emphasis is placed mainly on dealing with problems of management, monitoring and the links between evaluation and monitoring. Moreover, the pursuit of certain evaluation objectives is hampered by technical problems, such as the measurement of evaluation and impact indicators.

6.15. Very often, ongoing evaluations do not accord sufficient importance to critical appreciation of the regional development strategy chosen, to the viability of the projects financed by the Structural Funds, to the criteria for selecting projects, and to the effects and impact of the programmes' specific and general objectives.

6.16. Community added value, the analysis of which was one of the objectives of the ex post evaluations of the Objective No 1 and Objective No 2 CSFs for the 1989-1993 programming period, is covered only in exceptional cases by ongoing evaluations.

6.17. Changes in the socio-economic context, where Community assistance takes effect and in respect of which evaluation contracts generally provide for monitoring and updating, are dealt with systematically in certain cases (e.g. by means of very detailed progress charts in Wallonia) and in an unsatisfactory manner in other cases, which has a direct influence on the quality and updating of the analyses of strengths and weaknesses which are generally carried out as part of ongoing evaluations.

6.18. The extent and maintenance over time of the internal and external coherence of all of a programme's components (the objectives of the programme, sub-programmes and measures, and the degree of complementarity and synergy between objectives and measures) are examined only rarely in a systematic manner. Thus, in the case of the Brandenburg OP, the financial commitments in respect of priorities 1.1 (productive investments) and 1.2 (infrastructure) account over the 1994-96 period for 138 % and 242 % respectively of the initial estimate, and the assessor provided no justification for the regional development strategy that was implemented. Synergies are sometimes subject to analyses, which are often incomplete. This aspect warrants more systematic treatment, in particular with regard to synergies between ERDF and ESF activities.

6.19. Similarly, where the reorientation of a programme's priorities and of the allocation of resources is an explicit objective of the evaluation, the evaluation does not always produce a consistent overall presentation of the proposed amendments or an explanatory memorandum. Moreover, the implications of the extensions or curtailments proposed in respect of the measures are not always specified.

6.20. Many of the evaluations examined showed that the assessor had not been able to achieve all of the objectives envisaged in the contract, in particular as regards the evaluation of the measure's impact. Moreover, the time and resources available were not always proportionate to the objectives.

Approaches, methods and techniques

6.21. As contracts often formulate evaluation objectives and approaches in general terms, the choice of approaches, methods and techniques is often left to assessors, who sometimes adapt them during implementation to the difficulties they have encountered. Thus, the assessors occasionally make indicators out of the only information at their disposal.

6.22. Techniques and methods have not always been developed to meet evaluation objectives. Moreover, no genuine consensus exists as to the appropriate operational methodology for evaluating regional development; the Commission merely circulates 'best practices` in general terms without actually coordinating or harmonising their implementation in a concrete way. This imprecision, together with disparities between the Member States as regards programming, implementation and evaluation, results in evaluation reports of variable quality and makes it very difficult to compare results at a European level.

6.23. Measuring the impact of measures thus reveals shortcomings in the techniques applied, in particular as regards the characteristics and complexity of the effects to be evaluated:

(a) a detailed search for even more relevant indicators has not always been carried out, and the envisaged indicators sometimes serve only to measure physical and financial results;

(b) impact indicators have not always been given substance;

(c) evaluation of the socio-economic or regional development impact appears to be hampered by the increasing number of measures, sub-programmes and measures (Objective No 1), whilst many programmes for assisting the conversion and/or revitalisation of industry (OPs, Community initiatives and national programmes) may be implemented jointly. Furthermore, Community and national measures sometimes differ in their application, as do the dates on which measures are launched. Lastly, objectives are often unquantified and are formulated too generally (18). In this context, the links between the diagnosis of the socio-economic problems to be resolved and the objectives of the series of measures, sub-programmes and projects implemented, as well as the links between the objectives of this series, do not always appear obvious, especially when one remembers that the physical and impact indicators attributed to the various levels of a programme do not wholly reflect the programme's evaluation requirements. Similarly, it is difficult to identify the results of each programme and to separate them from the results of other private, regional or national measures;

(d) models for estimating the expected effects and the impact of measures on the basis of mid-term or anticipated results are rarely used. The ongoing evaluation of certain national CSFs was based on an input-output model (Portugal) or on a macro-econometric model (Spain, Ireland), whilst the use of regional models continued to be very marginal. The example of the development model for the province of Hainault (Belgium) used in the context of intermediate evaluations is worthy of mention in this connection. The establishment of models describing the effects of the measures is not very widespread either, except in the field of transport infrastructure. Mention should be made of the application of the input-output model to the Expo 1998 project (Portugal) as part of prior and ongoing evaluations, although the estimates should be considered with caution;

(e) estimates of the effects of measures on employment are also still precarious. The application of models and other analysis and enquiry techniques does not supply adequate information concerning, on the one hand, the net number of jobs created (taking into consideration, in particular the number of jobs that would have been created without the measure or the effects of job substitutions), and, on the other hand, the degree of permanence of the jobs created. Moreover, the application of such models and techniques usually provided no details about new jobs which could not be attributed to the implementation of the programme.

6.24. Some evaluation contracts provide, as part of the bottom-up approach, for enquiries in respect of local economic development partners and final beneficiaries, including companies which receive Community aid, with a view to making a quantitative or qualitative assessment of the contribution made by the Community programme. The enquiries' findings are often unrepresentative in the light of the number of participating companies and partners, and are of limited reliability insofar as they are not always subject to scrutiny.

6.25. As regards result and impact indicators, it was observed that:

(a) the time taken to obtain data and statistics is often long;

(b) the existing data cover geographical areas which do not correspond to the areas in question;

(c) data on companies are not always recorded by the establishments concerned, a fact which distorts local statistics if a single establishment is present in the area but the company itself has premises all over the country;

(d) many statistical bodies are unable to subdivide overall data with sufficient precision to target an area or a given series of problems.

6.26. Where the assessor provides more adequate indicators with a view to measuring impact, whether to compensate for a shortcoming or to replace those originally envisaged, he does not substantiate them and leaves the situation as to their operational nature uncertain. Furthermore, he does not always mention the enquiries still to be carried out which will give concrete form to the indicators, nor does he estimate their cost. It should also be stressed that it is difficult to quantify indicators concerning the programme's aims in the absence of information on historical series.

Implementation of Objective No 2 programmes (1994-1996)

6.27. The results not only of the ongoing evaluation of SPDs, CSFs and OPs during the 1994-1996 period but also of the monitoring procedure highlight the imbalance between the financing originally envisaged and the funds actually used to implement the programmes. Despite numerous adaptations of the programmes during the last programming phase (1994-1996), in order to absorb the financial aid, appropriations were transferred from the 1994-1996 programming period to the 1997-1999 one at an average rate of 12 %. The risk of this situation is that it may lead to an overriding emphasis being placed on the need to use up appropriations and an underestimation of the role of evaluation, in particular as regards consistency between objectives, resources allocated and the impact of structural measures.

7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS

Impact of CSFs and programmes

Results of the models as regards the impact of the Structural Funds

7.1. Given their specific characteristics, the models used for macroeconomic evaluation in the field of the Structural Funds (see Annex III) make differentiated assessments of the impact of the Structural Funds as regards:

(a) the implementation of the Structural Funds' objectives in terms of GDP growth, actual convergence of per capita GDP, employment and unemployment;

(b) observance of the conditions for implementing the Structural Funds, such as additionality and durable development with particular reference to controlling public debt and the balance of payments, to income distribution and to respect for the environment;

(c) the effects in terms of supply and demand;

(d) the effectiveness of Structural Fund instruments, such as public investment, investment in human resources and aid for private companies.

GDP growth

7.2. As Table 7 shows, the results of the principal models used to evaluate the impact of the Structural Funds (HERMIN, OGM and DIO) indicate a positive increase in GDP for Cohesion Fund countries and consequently greater convergence. Thus, cumulative growth in the economies of the Cohesion Fund countries from 1994 to 1999 as a result of structural measures under the CSFs appears, in terms of real GDP, to be 5,5 % for Portugal, 5,9 % for Greece, 2,6 % for Spain and 3,3 % for Ireland. These results assume that the principle of additionality is fully observed; they may thus be modified where the principle is not observed (see paragraphs 8.11 to 8.13). Furthermore, the various macroeconomic models produced sufficiently consistent results for the impact of the 1994-1999 CSFs on real annual GDP growth to be considered plausible. Nevertheless, demand-related effects continue to be predominant in the short term, as the effects on supply become apparent only much later. Similarly, the HELM regional model used for the Hainault region evaluates the impact of the SPD in terms of annual growth in total added value as accounting for an increase of 1,7 % to 2,5 % over the 1994-2002 period. By contrast, in the case of the Italian Mezzogiorno, the impact of the Structural Funds on growth seems very limited, or even close to zero. The Commission's QUEST model, by introducing hypotheses on the expectations of businesses and consumers, produces lower estimates of the effects on growth and employment than was the case with the previous models. The gains shown by this model with regard to the potential for growth and the reduction of unemployment only become apparent in the longer term.

7.3. As regards the average acceleration in annual growth due to structural measures implemented as part of CSFs, Table 8 shows that 1 % of growth in GDP generated by the multiplier effects of the CSFs leads, over the 1994-1999 period, to additional annual growth of 0,2 % to 0,3 % for demand and of no more than 0,2 % for supply. Economically speaking, the predominance, as highlighted by the HERMIN model, of the effects on demand, which would disappear if Community aid were discontinued, contributes to achieving the objectives of Article 130A of the Treaty only if the effects on supply are lasting, thereby increasing the productive capacity of the regions concerned. Insofar as the effects on supply become apparent much later than those on demand, the corresponding multiplier effects may be underestimated.

Real convergence in terms of per capita GDP

7.4. The models used within the framework of the Structural Funds show that convergence is not always guaranteed and that some CSF programming may at best reduce divergence rather than develop convergence.

7.5. In the case of Ireland, Bradley (HERMIN 1994) and Honohan (HERMIN 1997) envisage the possible effect of aid under the 1989-1993 and 1994-1999 CSFs on per capita GDP. Bradley considers that the 1994-1999 CSF's contribution to bringing the Irish economy closer to the EU average is modest whereas, in Honohan's view, the CSFs play a significant role in the convergence process. Beutel (1997) predicts that Ireland will catch up with the rest of Europe even without the 1994-1999 CSF, which would increase the annual growth rate by 0,6 %. Similarly, the annual growth rates of Spain, Portugal and Greece would increase for the 1994-1999 period by 0,4 %, 1,1 % and 1 % respectively due to the Structural Funds, which, in any case, has had a positive effect on convergence in terms of per capita GDP.

7.6. Forecasting simulations based on the OGM model of convergence in real per capita GDP in relation to the EU average provide estimates that stress the fact that, even if the trends projected in 1995 are too pessimistic in the case of Ireland (see Table 9), the CSFs' net contribution to convergence remains modest for the Cohesion Fund countries.

7.7. In the case of Hainault, the HELM model takes aspects of real convergence into consideration for the purposes of ongoing evaluation and considers that the province's estimated annual growth rate would be approximately 1,4 % to 1,6 % higher than the European average because of the Structural Funds; these figures would indeed make it possible for Hainault to catch up with other regions.

7.8. These estimates highlight the fact that, given the differences between Objective No 1 Member States' regional or sectoral productive structures, Structural Fund aid produces variable effects. They also confirm that Structural Fund programming must be improved in order for the corresponding assistance to improve the economy's productive capacity with a view to convergence.

Employment and unemployment

7.9. Table 10 summarises the findings of evaluations of the impact of the Structural Funds on employment and unemployment. All of the studies conclude that the CSFs generate net jobs. These new jobs are often created in the non-traded services sector, whereas the productivity effects of the CSFs are mainly concentrated in the industrial sector. Lastly, some studies note a reduction in the number of industrial jobs due to an increase in productivity.

Effectiveness of CSF instruments

7.10. The HERMIN evaluations for Spain, Ireland and Portugal show that aid for public investment, especially in infrastructure, is more productive in terms of growth than aid for investment in human resources (except for Spain) and for private investment. This trend is illustrated in Table 11 which shows, for each instrument, the increase in GDP due to the 1 % increase in total GDP generated by the CSF over the 1994-1999 period. In the case of Greece, the HERMIN study also concludes that investment in 'hard infrastructure` covered by the CSFs contributes more to GDP growth than investment in human resources. The use of the OGM model (Pereira 1991) to evaluate the effects of replacing aid for private investment or for investment in human resources by additional aid for investment in infrastructure shows an improvement in GDP growth and in the convergence process as a result of the Structural Funds.

Additionality and joint-financing requirements

7.11. The aforementioned models also made it possible to evaluate the significance of additionality and of public joint financing. Thus, Honahan (HERMIN, 1997) considers that, without the public joint-financing requirement, the growth rate of the Irish economy would have been 1 % lower and employment would have experienced a smaller increase. By contrast, Honahan envisages a more appreciable reduction in public debt as a proportion of GDP and in the balance of payments surplus if the public joint financing requirement did not apply. In the case of Portugal, the joint financing requirement leads, according to the HERMIN evaluation, to a total additional gain of 1,3 % in terms of GDP, but the development is less positive in terms of public debt.

7.12. The HELM evaluation for Hainault attaches great importance to additionality. According to the evaluation, the average rates of growth in value added for the 1994-1999 period based on the three envisaged scenarios (regional policy including the SPD, the continuation of regional policy without Community aid and regional policy from which the SPD has been deducted) are 2,5 %, 1,7 % and 0,1 % respectively, and the growth rate in terms of employment would fall from 0,8 % to 0,2 %. The industrial situation would also deteriorate considerably. The HELM evaluation also concluded that, if the rate of subsidy varies, aid does not stimulate investment to the same extent and that considerable investment would take place if levels of aid were lower.

Durable development

7.13. Table 10 shows that the effects of the Structural Funds on the balance of trade, the current-account balance or public debt are not always evaluated. Moreover, the various studies reach different conclusions. Thus, the economic expansion generated by the CSFs leads, in certain cases, to the improvement of public financing through taxation whereas, according to other models, the CSFs create budget deficits.

Role of macroeconomic evaluation in the Structural Funds' decision-making process

7.14. Given the short-term effects of the Structural Funds on demand, ongoing evaluations tend to over-estimate the difficulties experienced by the political authorities in using up Community funds and to underestimate the longer-term effects on supply. Reorientation or reprogramming of the OPs or CSFs may also be biased. The fact is that measures or projects which do not use up Community appropriations quickly enough may be discontinued, especially in the fields of innovation or new technology where implementation is complex, in favour of existing structures which are much less productive. However, in such cases the macroeconomic aspects of sustainable development are not always considered.

7.15. In this context, additional macroeconomic evaluations which take account of sustainability, growth or convergence are essential before any decision is taken regarding initial programming and any necessary changes. However, on several occasions in Spain, Greece and the new German Länder, measures were rescheduled on the basis of ongoing evaluation without serious overall macroeconomic analysis. With regard to Objective 1 regions in Germany, no macroeconomic analysis of the impact of the measures was performed when the intermediate evaluation was made.

Impact of sub-programmes, measures and projects

7.16. Very few evaluations are made of sub-programmes and measures, even though they act as the driving force of Structural Fund programming. The fragmentation, lack of definition and inconsistency of many sub-programmes and measures, and the inadequate quantification of their specific objectives, may help to explain the difficulty of measuring their impact.

7.17. This shortcoming complicates any attempt to identify and measure synergies and the degree of complementarity between individual measures or between one programme and another. Furthermore, the links between a programme's general objectives and the specific objectives of sub-programmes or measures are very often unclear. Thus, in several cases, problems of consistency in the Objective No 1 1994-1999 CSF programmes were noted in the ongoing evaluation reports.

7.18. Furthermore, techniques are seldom fashioned to measure the impact of a sub-programme or specific measure. Thus, the macroeconomic models described above evaluate the effectiveness of the CSFs' principal instruments (public investment, private investment and investment in human resources) only in terms of their respective contributions to GDP growth. Expenditure on infrastructure seems to increase the yield of the private capital which it complements, and would appear to speed up convergence more than other forms of investment (see Table 11).

7.19. Nevertheless, the choice and financing of measures or sub-programmes which make up the programme are not neutral in terms of the impact of regional development on lasting endogenous growth. The evaluations performed by the Commission or by the Member States in the context of their partnership as regards sub-programmes or measures are also insufficient to make it possible to examine the consequences of a specific measure on other areas or sectors or to compare it with other measures with similar objectives. Furthermore, analysis of the impact of the measures or sub-programmes must also take into consideration the initial attempts to offset growth against inter-regional equality which determined the choices made, so as to compare expected gains in efficiency with actual results.

7.20. In principle, the projects' relative costs and advantages are analysed with a view to approving financing. Pursuant to Article 5 of Regulation (EEC) No 2083/93 (ERDF), this analysis is compulsory in the case of major projects the total cost of which is more than 25 Mio ECU for investment in infrastructure and 15 Mio ECU for productive investment. These prior assessments of major projects are part of the examination of applications for Community aid and determine the amount of financing. However, ex post evaluation of them is not provided for in the SF Regulations and may not be required by the Commission for the purpose of establishing whether the socio-economic advantages actually obtained reflect the Community resources originally provided. This deficiency in the process for evaluating major projects renders the very concept of evaluation in this area ineffectual, as a prior assessment can be justified only if an ex post evaluation is actually made. Furthermore, these microeconomic and sectoral studies could also consider the potential impact of the Structural Funds on productivity and thus become an effective instrument for analysing this impact in the short or medium term.

7.21. Synergies and complementarity must also be taken into consideration when projects are evaluated. Current evaluations in this connection are still inadequate or superficial. The harmonisation of synergies and complementarity between projects is essential so as not to weaken their desired impact.

7.22. Although ongoing evaluations often tackle problems relating to programme management, they do not often take account of the criteria which determined the selection of projects, such as their sustainability or their contribution to regional development. However, the programme management system must be able to guarantee that the criteria for selecting projects are applied in such a way as to ensure that the programme produces the best possible results. In particular, projects providing very little added value in terms of development should be avoided even if the fact that they consume appropriations very quickly is an advantage. The project selection system must be more transparent for the purposes of inspection and evaluation.

8. CONCLUSION

8.1. The evaluation of Community structural measures is a legal requirement stemming from Council Decisions and is currently extensively implemented in partnership with the Member States. Evaluation has become a system of aid for the design and management of intervention measures, and contributes to improving understanding of their effects and usefulness. For the 1994-1999 period, more than 250 mid-term evaluations were completed by Monitoring Committees (19) in 1997.

8.2. With the revision of the reform of the Structural Funds, evaluation is entering a phase of maturity which should be consolidated without delay by improving the methodology. Much work still needs to be done to determine the most appropriate evaluation methods for measuring the socio-economic impact of the programmes within the framework of regional development, as the MEANS programme has been unable to provide a satisfactory solution.

8.3. Particular attention should also be paid to indicators at all levels as regards their relevance, the scrutiny of information and the attendant costs. In addition, qualitative aspects should be formalised. Without further progress in these areas, the evaluation process may very well stagnate.

8.4. The same is true of statistical bases, the weakness of which casts doubt on the validity of the indicators and has a major influence on the monitoring and evaluation process.

8.5. The current plethora of indicators of varying relevance is detrimental to the quality of evaluation and prevents it from becoming operational. A small number of impact indicators which are essential for justifying or reorientating a structural policy should be carefully selected bearing in mind the constraints imposed by the regionalisation of data and the material possibilities for gathering information. The reliability of these indicators should be checked. The comparability of indicators and their aggregation on a European scale should be improved.

8.6. In order for full account to be taken of evaluations in the decision-making process and in the operational management of the Structural Funds, steps should be taken to ensure that the evaluations are transparent and objective, that their objectives correspond to requirements and that the finalisation date is adequate.

8.7. In order for evaluation findings to be objective, assessors must remain independent of the managers and public authorities concerned.

8.8. The use of a common frame of reference would make it easier to read and compare the evaluation reports and to share practical experience.

8.9. For each evaluation, more operational objectives should be defined. The terms of reference and the technical specifications, which are generally approved by the Commission, should clearly indicate the most appropriate methodology for dealing with regional, sectoral or national aspects or for specifically covering short- or long-term effects. The strengths and weaknesses of any other methodology should be compared with the Commission's methods. Similarly, the frame of reference for each evaluation should include a minimum list of indicators proposed by the Commission for measuring the performance of the programmes' objectives so as to guarantee a degree of consistency which is essential for the purposes of analysis. Other socio-economic and structural indicators may, of course, be used to complement and refine this analysis.

8.10. Given the diversity of approaches to evaluation, the Commission must require assessors to explain their approach much more explicitly when calls for tenders are being issued, in particular as regards the type of effect or impact which is supposed to be measured and the period concerned, with a view to assessing the exhaustiveness of the methods used. In addition, evaluations should contain a detailed analysis of the links between the programmes' objectives and the priorities and measures implemented to achieve them, and of the corresponding inter-relationship between the objectives and results of the programmes, priorities and measures.

8.11. In this context, the sub-division of evaluations between the beneficiaries concerned needs to be improved so as to increase efficiency and to meet specific expectations, in particular in the area of ongoing evaluations. Similarly, greater control over costs should be sought, as the Commission is not always aware of the volume of expenditure allocated to evaluation within the partnership framework.

8.12. The Commission should also indicate which particular aspects of regional policy are to be evaluated: additionality, convergence, partnership, concentration, employment, endogenous economic development or social welfare. The per capita GDP should not be the only criterion for judging the effectiveness of structural measures.

8.13. It should be possible to base estimates of the programme's effects, impact and Community added value on national and regional development models and on models of the impact of the individual measures.

8.14. The macroeconomic results of structural measures should be refined by taking account not only of the imperfections of the underlying models but also of the side-effects of the measures and their cumulative effect on social welfare.

8.15. Macroeconomic models make a fundamental distinction between the effects on demand in the shorter term and structural adjustments in terms of supply. The programming of Structural Fund assistance should place greater emphasis on improving the productive capacity of the economy so as to ensure that the effects of assistance are permanent and to avoid excessive dependency on demand, which would lead to a situation where Community aid would remain essential in the long term.

8.16. As the effects on supply become apparent only over a longer period, ongoing evaluations generally pay more attention to the effects on demand. The Commission should aim to develop the instruments needed to monitor and evaluate effects on the structure of the economy.

8.17. Macroeconomic models also show that a policy of economic growth is not always compatible with durable development. The relationship between the impact on growth and the effects on durable development should be analysed in greater detail and systematic account should be taken of this duality in the programming of Structural Fund assistance.

8.18. The effects of the additionality and public joint-financing requirement are not easy to discern and are not always examined. More systematic and detailed analyses should be carried out in this area.

8.19. Evaluation per se does not profess to replace a political decision-making process which involves broader considerations, but aims to provide an essential tool for the sound and effective management of Community structural measures which takes account of the corresponding socio-economic results.

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 23 September 1998.

For the Court of Auditors

Bernhard FRIEDMANN

President

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

(1) In particular, these are:

(a) Article 6 of the framework Regulation (Regulation EEC No 2052/88 as amended by Regulation EEC No 2081/93) on the tasks of the Structural Funds and their effectiveness and on the coordination of their activities;

(b) Article 26 of the Funds' coordinating Regulation (Regulation EEC No 4253/88 as amended by Regulation EEC No 2082/93).

(2) 'Prior appraisal` is also sometimes called 'prior assessment`. The Court prefers to adopt the expression 'prior appraisal` as it appears in the Regulations.

(3) Community operations are to complement corresponding national operations or contribute to them. They are set up via close consultation, known as 'partnership`, between the Commission, the Member State concerned and the competent regional and local bodies, social partners and other competent bodies. 'Partnership` is conducted in full compliance with the respective institutional, legal and financial powers of each of the parties concerned.

(4) In its Annual Reports concerning, in particular, the financial years 1990 (chapter 7), 1993 (chapters 6, 7, 8 and 9) and 1994 (chapter 4), the Court had already reported on the numerous shortcomings in the field of evaluation for the 1989 programming period.

(5) This concerns the following evaluations:

(a) For the 1989-1993 programming period:

- prior evaluations of the O1 and O5b CSFs (1990-1991)

ex post evaluations of the O1 CSFs (1994-1995)

- ex post evaluations of the O2 CSFs (1996-1997)

- thematic evaluations (SMEs, telecommunications);

(b) for the 1994-199 programming period:

- prior appraisals of the O1 regional development plans

- prior appraisals of the O5b programmes

- prior appraisals of the 1994-1996 O2 SDPs

- prior appraisals of the 1997-1998 O2 SDPs

- critical analyses of the O1 SDPs and CSFs (1995).

The Commission also requested evaluations by Member State of the impact of European structural policies on the economic and social cohesion of the Union.

(6) DG XXII's memo of 14 March 1991.

(7) (a) The drawing up, in 1994, of the first 'terms of reference` for the on-going evaluation of the CSFs and the Objective 1 forms of intervention. These confirmed the role of ongoing assessment as an aid to decision making for the Monitoring Committees and its implementation within the partnership system. They also specified the procedures for its implementation, in particular the composition and role of the evaluation team;

(b) the 1995 distribution of the 'Common guidelines for monitoring and ongoing assessment` for the Community SFs, which lay down the factors for analysis (overall and specific objectives) and the link between monitoring and ongoing assessment;

(c) the sending of letters to the Member States (1995) reminding them that they were required to carry out ongoing assessment and again specifying the desired implementing procedures.

(8) Such as:

(a) the Brussels conference on the evaluation of European regional policies (16 November 1995) concerning mainly the results of the ex post evaluations of Objective 1 (1989-1993) and prior appraisals for 1994-1999;

(b) the European conference on evaluation methods for SF interventions (Berlin 2 and 3 December 1996) mainly concerning the problems involved in assessing impact at the micro-economic, macro-economic and thematic levels;

(c) the Seville conference of March 1998 aimed at promoting exchanges of experience in the field of evaluation under the partnership system.

(9) This programme is managed jointly by the Commission and an external consultant, whose involvement is financed from budget heading B2-1 8 2 0, for an annual amount of 932 000 ECU.

(10) The programme by programme approach was hampered by the following:

i) the short planning deadlines implying, in particular, the absence of any prior appraisal;

ii) the lack of tangible objectives, quantified indicators for implementation, results and impact;

iii) administrative constraints, in particular where the sub-programmes of a given programme fall under the responsibility of different authorities, or the sectoral and regional approaches overlap;

iv) weaknesses in the Community rules, resulting in particular, in projects rejected for the Objective 1 OPS being accepted in the context of the Community initiatives;

v) the project by project approach of the Cohesion Fund, which also did not contribute towards a harmonious application of the programme by programme approach.

(11) This does not, however, mean that these are job creations, as the econometric model is based on the hypothesis of full employment and companies with production overcapacity during down-turns.

(12) The 8th annual report on the SFs mentions important leverage effects which refer to overall results, such as the creation of jobs or companies.

(13) Evaluación intermedia de programas operativos regionales del objetivo n° 2 (fase 1994-1996). Orientaciones sobre contenidos comunes a recoger en los pliegos de condiciones técnicas, 23 May 1996.

(14) DATAR, Regulatory and administrative timetable.

(15) General criteria to underpin Terms of Reference for Mid-Term Evaluation of Operational Programmes/Community Initiatives.

(16) Evaluatiekader voor de programma's van de Europese Structuurfondsen, Terp Advies- CBEA Pieter Van Run, December 1997.

(17) 36 of which were examined on the spot and/or at the Commission.

(18) E.g. the revitalisation of the countryside, the strengthening of the economic fabric of the region and structural modifications related to regional development.

(19) The Monitoring Committees include representatives of the national authorities concerned and of the Commission.

ANNEX I

Summary of the methodology to be applied as part of the ex post evaluation of Objective 1 CSFs (1989 to 1993)

1. The explicit aim of the ex post evaluation of the Objective 1 CSFs (1989 to 1993) was to:

(a) assess the strategy and objectives selected in the CSFs and to draw the appropriate conclusions for future action. To analyse the consequences for the initial objectives of redefining and adjusting interventions during the implementation period;

(b) take provisional stock of the concrete results of the CSFs and OPs as regards significant measures and make an initial assessment of their impact;

(c) analyse the effectiveness of the implementation mechanisms and draw the necessary conclusions with a view to making future adjustments.

2. Methodological approaches were prepared by the principal consultant, who was responsible for coordinating evaluations made by other consultants on behalf of the Member States concerned:

(a) for the first objective, in order to evaluate the logic intrinsic to the CSFs and OPs and to the adjustments made to the CSFs and OPs since 1989 - and, more specifically, to analyse the causes of these adjustments, the Community added value, internal consistency, external consistency, the relevance of the strategy to existing problems and the appropriateness of financing - the following tasks, based on a top-down approach, were to be carried out:

- collection of information and key data: financial data, ex ante objectives, anticipated output/effects, target groups and areas, main instruments used, number of projects/measures approved and number of projects completed,

- description of the environment and development of the programme (initial situation -current situation); identification of external factors influencing the development of the programme and its success/failure; analysis of the basic problems and causes; formulation of key regional development indicators, including indicators of changes in disparities,

- identification of policy instruments (loans, subsidies and technical assistance),

- prioritisation of objectives, identification of levels of priority and of the links between objectives and instruments;

(b) for the second objective, the analyses to be performed were based on top-down and bottom-up approaches:

- analysis of the output and effects of OPs, measures and projects, account being taken of 'net losses`, and substitution and multiplication effects,

- analysis of the relative effectiveness of measures and combinations of measures; analysis of the causes of successes/failures,

- comparison of output and input,

the aim was also to formulate key and optional indicators;

(c) for the third objective, the aim was to analyse the effectiveness of CSF and OP implementation measures and of a small number of measures, and to identify existing and planned improvements in the planning, implementation, monitoring and evaluation processes; for each Member State, the examination would cover decision-making, planning, financial control, implementation, monitoring and inspection mechanisms, financial channels, partnership, additionality, fund coordination, and greater flexibility and transparency in the simplification and harmonisation process; implementation and monitoring would also be analysed in depth for a small sample of measures/major projects;

(d) ad hoc samples of programmes, measures and projects were chosen with a view to substantiating a small number of physical and impact indicators and to carrying out in-depth analyses.

ANNEX II

Summary of the ex post evaluation approach in respect of Objective 2 (1989 to 1993)

The ex post evaluation of Community assistance under Objective 2 for the 1989 to 1993 planning period, which generally included ERDF and ESF components, comprised:

(a) general comparisons in all Objective 2 regions and areas of the socio-economic situation prior to Objective 2 Structural Fund assistance and of the socio-economic situation after assistance; analysis of the strategy employed in terms of the prioritisation of objectives, priorities and measures, and of indicators, planned and unanticipated effects, synergies between programme components and the utilisation of the various funds. These analyses were to be based on the content of the CSFs and OPs, on annual implementation reports or reports submitted to the Monitoring Committee and on prior and ex post evaluations, where available;

(b) detailed studies covering 22 regions, the aim of which was to:

- study in depth the CSFs, OPs and types of measures adapted,

- analyse institutional structures and programme management procedures,

- evaluate the impact and added value effects attributable to the CSFs and to the OPs,

- evaluate the cost and effectiveness of these effects,

- study the Objective 2 programme's contribution to the regional development process.

These in-depth evaluations employed the bottom-up approach, the aim being - using an examination of operations covering the selection process, typology, management process, the impact and effect in terms of added value, and synergies and additionality - to work upwards from one level to the next, compiling information first about individual measures and ultimately about the whole programme.

ANNEX III

Characteristics of the macroeconomic models used to evaluate the Structural Funds

1. The principal models used in the macroeconomic evaluation of the Structural Funds comprise:

(a) econometric models, such as HERMIN (1) and HELM (2) and QUEST II (3);

(b) calculable general equilibrium models, such as OGM (4) and BBM (5);

(c) input-output models, such as DIO (6) and FIDIMI (7);

(d) structural-residual analysis.

The characteristics of the application of these models in the cases examined are shown in diagram form in Tables III 1 and III 2. Table III 3 reproduces a model of the characteristics of the CSFs.

2. Furthermore, by simulating the evolution of the economies of Objective 1 countries over a period of more than 10 years and on multi-sectoral bases, it is possible to perceive the structural change brought about by the Structural Funds and to evaluate the reliability of the catching-up process which Community assistance is supposed to support and consolidate.

3. As part of its enquiry, the Court had an analysis performed (8) to assess the appropriateness of these macroeconomic techniques for measuring the impact of structural measures and to establish whether they may contribute effectively to the decision-making and strategic aspects of programming.

4. The analysis showed that explanations of the endogenous or exogenous factors upon which the success of the structural programmes depends vary according to the methods and conceptual approaches employed. The fact is that macroeconomic models aim to describe the mechanisms of the economic system of a country or region and to capture or simulate the effects of an exogenous shock such as the assistance provided by the structural programmes. However, no single description of the economic facts is definitive.

5. The HERMIN econometric model takes into account the specific characteristics of each Member State and aggregates effects at a national level. It assumes that economies actually utilise their full production capacity and that the accelerator and multiplier effects related to increases in investment consequently create an expansion in production and employment. This model seems to give satisfactory results in terms of effects on demand. With regard to the effects on supply, taking the Phillips curve into account, the model suggests that a fall in unemployment will bring about an increase in wages whilst limiting the effect on inflation. Also, the effects on productivity are explicitly taken into account by modelling the external saving effects that increases and improvements in human capital and public and private physical capital have on productivity. They are, however taken into account differently from one country to another. The model provides only partial information on durable development, and, in particular, does not consider variations in exchange rates aimed at guaranteeing the stability of the balance of payments or variations in fiscal transfers.

6. The HELM macro-econometric model, which was used for the ongoing assessment of the 1994 to 1999 Objective 1 SPD for the Province of Hainault (Belgium), is one of the few to take the regional development approach specifically into account by considering the effects of endogenous growth linked to long-term productivity, although improvements in productivity due to SF intervention are largely limited to the industrial sector and only concern labour productivity. On the basis of a similar assumption to that of the HERMIN model with regard to the utilisation of productive capacity and the multiplier and accelerator effects of investment on demand, it gives the overall and aggregate effects of the SPD on the Hainault economy, such as overall investment generated, added value or employment in the manufacturing industry and commercial service sectors for the period between 1994 and the years 2002 or 2005. It should be noted that the HELM model does take regional exports into account, but not imports from other regions, thus risking an overstatement of the effects on demand. The figure for demand is all the higher because neither wage adjustments related to the fall in unemployment nor higher costs or capacity constraints were taken into account. The HELM model also gives little information as to the trend of unemployment or the demand for labour. Lastly, it is based on historical statistical data, but these were not cross-checked against any other series of relevant data.

7. The computable general equilibrium models apply more particularly at a national level. They show the long-term effects of Community structural measures on supply. The OMG model, which uses a context of inter-temporal endogenous growth, assumes that the production factor markets are in equilibrium and that economies are working at full production capacity. Changes in demand have no effect on production and employment and the SF programmes can only stimulate growth by increasing productivity. These productivity effects are specified clearly in the production function. The objective of the model is to determine an optimum situation for the country by maximising a function of inter-temporal social well-being. On the basis of the hypothesis of a situation without uncertainty, economic agents, anticipating perfectly the amount of money that will be allocated to them and the period over which aid will be spread, are supposed to take the best possible consumption and investment decisions in relation to these data. Lastly, the OMG model provides for long-term stability of the balance of payments and budgetary equilibrium.

8. For certain sectors, the BBM model expects that labour markets will be in equilibrium and that, consequently, demand will not influence production and employment, while, for other sectors, an expansion in demand is supposed to generate growth. This model also includes technological change - related productivity effects that result in reductions in inputs.

9. The DIO input-output model tries to show the links between all the resources available to an economy and all final consumption. In this way it identifies interdependencies between the various sectors of a given country, linked together on the basis of intermediate consumption. The input-output approach maximises the effects of structural interventions on demand and ignores those on supply (effects on productivity and wages and inflationary adjustments). Durable development is not taken into consideration. The FIDIMI model is a combination of a general equilibrium model and an input-output model which can be used to analyse the effects of Community interventions in Southern Italy on the North of the country.

10. Among other existing techniques for measuring the effects of regional policies, Eurostat uses structural/residual analysis to evaluate the impact of the SFs in certain areas eligible under Objectives 1 or 2. This analysis attempts to identify the residue resulting from the difference between the actual situation and the counterfactual situation produced via the application of estimated national tendencies. However, carried out in this way, the calculation of impact mixes the results of the programme under evaluation, other national and European political measures and the path previously followed by regional policy. Moreover, the studies carried out by Eurostat in 1997 include data for indicators, the most recent of which date back to 1993. This considerably reduces the value of their conclusions.

11. The QUEST II model attempts to analyse the economies of the Member States and their interaction with the rest of the world, in particular with the United States of America and Japan. The model is concerned more particularly with the transmission of the effects of economic policy at the national and international levels. It was initially devised to serve as a simulation tool rather than for predictive purposes.

The QUEST II model now pays more attention to the effects of demand than its predecessor and takes into greater consideration interactions in terms of stocks. However, it gives little or no attention to the implications in terms of changes in the behaviour of economic agents.

>TABLE>

>TABLE>

>TABLE>

(1) HERMIN: Econometric model based on the original HERMES model (Harmonised Econometric Research for Modelling Economic Systems), but whose minimum size (Min) distinguishes it from this latter, whence (HER)MIN.

(2) HELM: Hainaut Economic Lead-in Model.

(3) QUEST II: a Multi-Country Business Cycle and Growth Model - Economic Paper No 123 - October 1997. DG II - Economic and Financial Affairs.

(4) Optimal Growth Model (OGM); see Pereira, 1991 and 1995.

(5) Bourguignon, Branson and de Melo model (BBM); see Lolos and Zonzilas, 1992.

(6) Dynamic Input-output model (DIO); see Beutel, 1995, 1996 and 1997.

(7) FIDIMI Consulting Company, Evaluation of the macroeconomic impact of the Objective 1 CSF (Italy) and of the implementation mechanisms.

(8) The Court has had the services of an external assessor who was highly specialised in the field of macroeconomic and European economy models.

COMMISSION'S REPLIES

I. GENERAL COMMENTS

The Commission accepts that the evaluation process, being something entirely new for many parties, is still rather rough and ready as regards its scope and the use that can be made of it, but insists that remarkable progress has been made in the space of just a few years.

Conscious of the difficulties that still have to be overcome, the Commission has redoubled its efforts on the organisational and methodological front to achieve the most satisfactory situation possible, engaging in a permanent dialogue with all those in the different countries of the Union who are studying and carrying out research in this area.

There are three main phases in the development of the assessment process for Structural Fund intervention, corresponding to the three programming periods:

1989 to 1993 period

The regulations on the first reform of the Structural Funds were not adopted by the Council until December 1988 and the national plans submitted at the beginning of 1989 only partly reflected the requirements for strategic vision, choice of objectives to be achieved and quantification of expected results.

The ex post evaluations therefore attempted to remedy these shortcomings by comparing the results achieved with the reworked objectives. This considerable task suffered from its own limitations, which are reflected in the reports dealing with the interventions in Objective 1 and Objective 2 areas. Nevertheless, it represents the first attempt at coherent analysis of the impact of this reform on economic and social cohesion.

1994 to 1999 period

At the request of certain Member States and of the Commission, this first experiment was followed by a tightening of the rules on monitoring and evaluation and the identification of three stages: prior appraisal, ongoing assessment and ex post evaluation. There can be no doubt about the success of the ongoing assessment. Going beyond what was strictly required by the regulations, the mid-term assessments were organised for the first time by and at the level of each Monitoring Committee, thus bringing assessor and the beneficiaries of the assessment closer together. The reports produced were presented to and debated in each committee, with the result that the process served an educational and operational purpose that studies organised and carried out by the Commission could not hope to fulfil.

There may have been some discrepancies between the methods employed, the choice of assessors, the cost of the work and the level of detail of the reports, even though coordination and general supervision were provided by the Commission's evaluation coordination unit and its national contacts in the technical evaluation group. However, despite certain differences connected with the discretion allowed to each Member State under the subsidiarity principle, the Commission believes that this process played an essential role in propagating a genuine climate of evaluation. It proved to beneficiaries that this was an operational tool for improving the management of interventions financed from both national and Community sources. Future evaluations can only benefit from this attitude in that the quality of their conclusions depends as much on the quality of the information in the possession of and supplied by the project managers as on the expertise of the independent assessors.

2000 to 2006 period

The new proposals for regulations aim in particular to reduce still further the factors limiting the scope of the evaluation process, out of concern primarily to increase the efficiency of the deployment of Structural Funds. The following proposals are made in this spirit:

- the prior appraisal should be properly incorporated into the plans which Member States are required to submit containing a strict quantification of the objectives to be attained,

- the results of evaluations already carried out in the previous period, either on a particular topic (thematic evaluations) or by a specific programme, must be taken into account,

- there should be a close correlation between mid-term reprogramming and the conclusions of the ongoing assessment.

In a bid to ensure greater effectiveness in the use of Structural Funds, the Commission has proposed that 10 % of the Structural Fund aid be disbursed at mid-term, depending on performance, the criteria for assessing performance being closely linked to the quality of the programming, follow-up, prior appraisal and ongoing assessments.

Finally, with regard more specifically to methodology, the Commission plans by the autumn to publish the work carried out under the MEANS programme (1) in order to ensure greater harmonisation of evaluation within the Union. This will involve:

- a detailed description of specifications and the methodology for prior appraisal, ongoing assessment and ex post evaluations,

- a survey of the literature on different assessment methods and the conditions for their application,

- an analysis of systems of indicators by area of intervention. Harmonisation of concepts and a list of suggested indicators,

- methodology and organisation of subject-based evaluations,

- a multilingual glossary of the main concepts of evaluation,

- an outline of the most innovative national or Community projects.

It will be clear in what follows that the Commission endorses the Court's comments in some cases. But it is important to take account of the difficulties and limitations inherent in any evaluation exercise, even if these are being gradually overcome or superseded.

Nor should we lose sight of the fact that evaluation is first and foremost a technical aid to decision-making and management; it is not intended to take the place of political decision-makers. It serves to shed light on the consequences of choices to be made or decisions already taken. Moreover, there is a need for a degree of flexibility as to the methods used in order to allow for the different contexts from one evaluation to the next.

Finally, the Commission would point out that since the start of the Fund reforms in 1988, the respective responsibilities of national administrations and the Commission for Structural Fund management in general and evaluation in particular have been constantly shifting. The Member States are increasingly required to assume prime responsibility and to carry out their own evaluation.

II. SPECIFIC COMMENTS

2. THE SCOPE OF THE AUDIT

2.3. The Court did not include in its audit countries like Germany and the United Kingdom that have a very different attitude to evaluation.

3. SUMMARY OF THE MAIN OBSERVATIONS

3.2. The Commission believes that the evaluations for which it does not have the figures on expenditure are marginal in terms of costs. Because ongoing assessment is a new and growing field and is carried out at a decentralised level, the volume of expenditure devoted to it is changing. Expenditure on evaluation is broken down by country, by objective and by type of intervention. The figures are also published in the Commission's annual evaluation overview.

The Commission will continue its efforts to obtain a systematic overview of the amount of expenditure set aside for evaluation out of the sums awarded in technical assistance and managed under the partnership system.

3.3. Under the terms of the regulations and in deference to the subsidiarity principle, the Commission has no authority to dictate a precise framework for evaluation indicators and methods.

In an effort to develop national practice and improve the quality of evaluation, the Commission has:

- set up the MEANS programme and distributed eight booklets on methodological questions,

- organised three European conferences on evaluation techniques and practice,

- distributed documents such as the frame of reference for prior appraisals and a guide for evaluating major projects,

- set up Technical Evaluation Groups in which the Member States participate and organised several seminars.

3.5. There is an objective problem in trying to coordinate the ex post evaluation with the different stages of planning structural interventions. Ex post evaluations should really be carried out two or three years after the end of programming.

In order to solve this problem the Commission has promoted ongoing assessments and thematic assessments. The thematic assessments carried out in 1992 and 1993, for example, produced material that could be used when preparing the programming for 1994 to 1999.

The ongoing assessments will be available for the preparation of programming after 2000, because they have already been completed, and evaluation overviews are currently being introduced for each objective to supplement and update the ongoing assessments. Four thematic evaluations have also been started.

The Commission has introduced this provision into its proposals for amending the Regulations.

3.6. A distinction should be made between the objectivity of the evaluations and the adequacy and quality of the data. The question of data is often highlighted in evaluations. The reports almost always pinpoint this problem. However, there has been a distinct improvement in the situation as regards data, largely thanks to the demands of evaluation.

The Commission is not satisfied with the current situation as regards data, but this should not be regarded as a criticism of evaluation but of the monitoring systems. The Commission would stress that the problem of the relevance of monitoring data and indicators is a matter primarily for the follow-up audit.

The Commission is continuing its efforts to improve the situation and will act on the Court's comment about a need for more systematic evaluation of the reliability of data. The work carried out in connection with the audit and control will also help to consolidate the improvement in the situation which has been apparent in the field for several years.

3.7. The departments concerned comply with the conditions for awarding contracts prescribed in the relevant Directives and with the procedure laid down by the ACPC. However, it is not possible to negotiate the remit before signing the contract. This sometimes means that clarifications have to be made which upset the timetable, but the problem is marginal.

The Commission alone cannot be blamed for this problem. The inadequacies of some evaluation reports may also be due to incompetence on the part of the consultants or the poor quality of the data available. Even when the same terms of reference are used as a basis, some reports in a country may be excellent while others are unsatisfactory.

3.8. The Commission shares the Court's views about the difficulties experienced by the first evaluations organised in a climate of indifference in the Member States.

The ex post evaluations of Objectives 1 and 2 were carried out by consultancies with an international reputation and by highly qualified experts using a common methodology. The second part of paragraph 3.7 justifies both the measure and the evaluation exercise, despite these constraints. The Court's criticisms refer to the evaluations carried out in 1993 to 1994, but on the Commission's initiative a great deal of work has been done since then on developing methodologies. The Court's criticism of the ex post evaluation of Objective 2 is also valid in part. However, a more realistic view should be taken of the possibility of devising a precise quantitative measurement of impact or Community added value.

3.9. The ongoing assessment took the form of an analysis of the implementation of the interventions. Effects are more likely to be assessed in the context of the evaluation overviews and ex post evaluations. These evaluations, however, do not always involve the use of econometric models - which tend to be very expensive and are not necessarily the most appropriate way of measuring impacts.

Evaluation is a management tool and, as such, just one of several aids to the decision-making process.

3.10. The Commission found that dividing Objective 2 programming into two periods did not correspond adequately to the needs of the areas in question and made the implementation of programmes more complicated. The Commission's proposals in March 1988 for a reform of the Structural Funds provided for a single programming period for Objective 2, as is the case for the other objectives.

3.11. - 3.12. In order to avoid having an unmanageable number of evaluations of programmes or sub-programmes a more selective approach may sometimes be needed in certain key areas.

The Commission shares the Court's opinion on the need to cut the number of measures and sub-programmes or even programmes. It made a number of specific proposals to this effect in the course of the Fund reform.

4. GENERAL ISSUES

Introduction

4.1. On several occasions the Commission expressed its regret at the failure of certain Member States to quantify their objectives adequately. The fact that national public expenditure is not administered by objective also hampers the development of management systems based on indicators. Consequently it is less a matter of a defect in the methodological or analytical framework than a problem of programme design and the sophistication of national statistics.

4.3. The Commission would point out that:

- in 1994 it sent Member States its common guidelines on monitoring and ongoing assessments,

- in December 1994 it presented the Technical Evaluation Group (consisting of those responsible for evaluation in the Member States) with a framework document on the purpose and content of and procedure for ongoing assessments. This initial meeting was followed by four further meetings of the same group,

- in 1995 it distributed to the various departments copies of a MEANS document on how to organise ongoing assessments in the context of partnership. Three of the five chapters of this document deal with formulating the specification and invitations to tender, selecting assessors and supervising the quality of the evaluation,

- in 1996, before receiving the first evaluations, it distributed to the departments involved a MEANS document on assessing the quality of evaluation reports, including a detailed schedule.

4.5. It should be recalled, however, that some evaluations were carried out under partnership arrangements even in the period 1989 to 1993. Thus Member States and the Monitoring Committees actively participated in several thematic and ex post evaluations initiated by the Commission.

4.10. The Court comments on the small amount of money earmarked for Objectives 1 and 2 evaluations in the first programming period.

Because almost none of the Member States had carried out an evaluation, the Commission itself took the initiative of conducting a first evaluation of all Objective 1 CSFs and 15 Objective 2 SPDs at the beginning of the period. In addition to the thematic assessments, which also relate to the period 1989 to 1993, the Commission carried out an ex post evaluation of all CSFs and SPDs for Objectives 1 and 2 in the period 1989 to 1993.

4.11. The question of knowledge of costs was raised in paragraph 3.2. The only evaluations that might not be included in the financial overview of evaluations drawn up by the Commission are the small number initiated solely by the Member State and funded from technical assistance. The amounts involved are marginal compared with all the other work.

Frame of reference

4.12. - 4.13. The MEANS documents were intended as educational tools for administrations which are often still developing their technical expertise. The priority for the Commission was to raise the average standard of evaluation, not the level of experts who are already specialised.

Statistical bases and information concerning objectives

4.14. - 4.15. The Commission shares the view of the Court.

Indicators

4.16. - 4.21. The Court acknowledges the Commission's efforts to develop indicators. The regulations were tightened up in 1993, and an exercise to quantify disparities was launched with each Member State. The question of quantifying objectives has been a priority for the prior appraisal and some progress has been made: the 1994 to 1999 interventions are distinctly better quantified than their predecessors. The first task of the ongoing assessments has been to improve the indicators and their effectiveness.

The Commission would stress that the problem is not just one of increasing or reducing the number of indicators but the difficulty the monitoring systems have in regularly quantifying the agreed indicators that already exist for the programmes. Nor is it a question of a lack of coherence in programming, for it is the right of the regional and local authorities, under the principle of subsidiarity, to set their own priorities and determine the scope of programming in their countries.

Progress is, however, being made, under the pressure and demands of evaluation.

The Commission's view is that a simple methodological framework is not enough and it therefore intends to take more far-reaching steps as part of the Fund reform.

Taking evaluation into account

4.23. The Commission agrees that the dual programming cycle did not make coordinating evaluation and programming any easier and for this and other reasons it proposed a single programming period for the future. Nevertheless, the 1997 to 1999 programming drew on an ex post evaluation of the 1989 to 1993 period and on a prior appraisal of the SPDs carried out on the Commission's initiative and relating to matters under its authority and priorities for the Union.

As for the ex post evaluation of the 1994 to 1996 period, the Commission believes that the evaluations carried out in the context of reprogramming and the thematic evaluations of SMEs and R& D cover much of this ground. The Commission is currently discussing with the Member States the launch of an evaluation overview in the second half of 1998 covering the period since 1994. This overview would be used for future programming.

4.24. - 4.27. The Commission is currently drafting a report describing, for each country, the quality of evaluation and its contribution to modifications to interventions. A summary will cover the key points of the mid-term review. The information available to the Commission does not support the Court's comments relating to widely divergent situations and hundreds of programmes.

The Commission will assess the results when the report on the mid-term review is presented. For the time being, the Court's position seems to be premature.

The Commission would also point out that evaluations are useful not only for managers but also in the interests of greater accountability to the budgetary authority, the European Parliament and the public.

The objectivity of evaluations

4.28. - 4.30. In these paragraphs the Court does not make sufficient allowance for the complexity of the partnership system, despite the positive comments in paragraphs 6.5 to 6.7.

There is more to the issue of objectivity and independence of evaluations than the existence of a precise frame of reference for applying the methodologies and checking data on the spot. The quality of the evaluation relates more to degree of professionalism, professional ethics and the interplay with the interests of the Member States.

The Court fails to mention that three of the eight criteria in the MEANS brochure No 5 on assessing the quality of evaluation reports explicitly refer to the appropriateness of the methodology, the reliability of the data and the credibility of the results.

The Commission is constantly striving for improvement in the standards and requirements, bearing in mind the reality on the ground. It will be turning its attention more to assessing the operation of evaluation systems on the ground.

The evaluation contracts awarded by the Commission

4.31. - 4.32. The Commission applies the standards and procedures for awarding contracts laid down by the ACPC. Experts are chosen on the basis of selection and award criteria published in the Official Journal. Their application and the results of this exercise feature in the report to the ACPC.

As regards the degree of precision of the contracts it should be borne in mind that the Commission is paying for expert advice and has every interest in ensuring that the contracts are drafted in such a way as to enable the experts appointed to make the best use of their skills.

4.33. The method used for Objective 1 is not appropriate for Objective 2, which comprises 60 programmes. Given the expenditure involved and the number of teams, it would lead to a fragmentation of the invitation to tender that would go against the letter and the spirit of the Directive on the procedure for awarding public works contracts.

The Commission complied with the rules in these cases and was careful not to lose control of the practical arrangements for implementing the evaluation. To this end a steering committee was set up to accompany the evaluation and meetings were organised periodically with the evaluation teams. The reports were subsequently checked with the departments in charge.

4.34. The details of the specifications depend on the needs and information requirements of the requester. The requester drafts them and the evaluation units carry out checks. The terms of the contract, with the exception of the description of work to be carried out are standardised by the ACPC.

As pointed out earlier, there is no generally accepted methodology for programme evaluation. This being the case, and in order not to lose its control over the approach adopted, the Commission discusses the methodology, its relevance to the questions raised and the applicability of the techniques proposed. The general methodological approaches form part of the specification, but the precise details and techniques are left open to discussion.

4.35. In this specific case the Commission, with the agreement of the consultants who produced the evaluation, tried to improve the quality of the report by organising a meeting on the content and the lessons to be learned with all the units and officials concerned.

The final report was submitted in May 1995 (three months after the scheduled date), accepted after official consultation in June 1995 and payment was made in July 1995. The report criticised by the Court was in fact a clarification by the consultant involving no additional budget.

4.36. The Commission stresses that the terms of reference were structured around three principal, well-defined questions: the relevance of the strategy, the results and effectiveness in terms of output and the effectiveness of the implementing strategies (pages 2 to 4 of the document).

The Commission explained its position on the question of methodologies in paragraph 4.12. The various departments are consulted and the lead department decides whether or not to accept the report.

5. THE EX POST EVALUATION OF THE OBJECTIVES 1 AND 2 CSFs (1989 to 1993)

Introduction

5.1. The Commission endorses the Court's comments on the difficulties experienced with these evaluations connected with the design of the first-generation programmes.

The ex post evaluation of Objective 1 CSFs (1989 to 1993)

5.4. Some of the Court's comments on this point should be qualified.

(a) If data on regional disparities were out of date this is a criticism of the statistical office rather than the methodology of the evaluation.

(b) The assessors used the sampling techniques generally employed for evaluations, combined with case studies for greater detail in specific instances. These techniques, properly used, can provide useful lessons.

(e) The Court states that bottom-up analyses do not generally allow conclusions to be drawn in terms of impact at programme level. The impact on Community policies, the impact on improving management, transparency, etc. cannot be top-down or based on econometric models.

(g) The Court criticises aspects that were not within the remit of the evaluation.

5.7. The Commission confirms that the evaluations in question were fully used in the context of its duty of accountability to the budgetary authority and as part of the work on the report on Economic and Social Cohesion.

5.8. One of the three objectives of the remit for the evaluation was to suggest ways of improving the monitoring and management system. The relevance of the strategy was also analysed and a performance assessment drawn up which enabled the effectiveness of the interventions to be analysed.

The ex post evaluations of Objective 2 (1989 to 1993)

5.13. The systematic use of econometric models is not justified in all cases, either because the level of aid is low or because the necessary regional statistical series are unavailable.

Most of the assessments of the policy appear in the report on Economic and Social Cohesion. The Commission has suggested modifications as part of its new proposal for the reform of the Structural Funds from the year 2000.

5.16. The Commission has already included a number of the recommendations of this evaluation in its proposals for the new regulations.

6. ONGOING ASSESSMENTS (1994 to 1999)

The development and integration of the principles of ongoing assessment

6.1. - 6.20. The Commission shares the opinion of the Court in this part of the report. It would clarify a number of points:

- Paragraph 6.15: The Commission believes that critical appreciation of the development strategy, the impact in terms of objectives, and project viability are matters more specifically for the prior appraisal or the ex post evaluation.

Paragraph 6.16: The purpose of the ongoing assessments is to adjust the interventions and improve management. Analysis of the Community added value will be part of the evaluation overview or ex post evaluation. However, the evaluation of the impact of Community priorities on the modification of interventions will in several cases contribute valuable information about Community added value.

Paragraph 6.18: As far as possible, the Commission tried to obtain a picture via the ongoing assessment of the synergy between operations financed by different Funds, particularly in countries with separate programming, as in the case of the German Objective 1 SPDs. To this end the assessors appointed for ERDF projects produced a summary of the evaluations carried out under each Fund. The Commission agrees that the regional evaluation cited by the Court was particularly weak, but it is not representative of all the evaluations carried out in that country. It is correct to state, for example, that the ongoing assessment did not analyse the changes in the indicative funding plan resulting from a request by Brandenburg to change its OP. As for the high rate of absorption of funds allocated in the case of the projects referred to, a number of changes were made to the measures in the Monitoring Committees of 12 June 1996 and 5 August 1996.

Approaches, methods and techniques

6.21. The Court criticises the vague formulation of interim objectives in the contracts. The Commission gave guidelines in certain cases, but these were not followed at the decentralised level.

Bottom-up methods are used for evaluating effectiveness and efficiency. In the case of large CSFs, where the consequences of reprogramming on global impacts had to be determined, use was made of econometric modelling techniques in Ireland, Spain, Greece, Portugal and Italy.

6.22. The Commission drew on the best expert opinion in Europe and the United States of America to identify the most effective ways of evaluating programmes.

Objective methodological difficulties and, in many cases, inadequate monitoring systems, limit the comparability of results at European level.

6.23. The aim of ongoing assessments is not usually to measure the impact of programmes. This is a deliberate choice because the impact is not quantifiable in the first phase of the intervention, the sequence being first implementation, then results and finally impact. The question of the evaluation of the overall impact of large CSFs was discussed at paragraph 6.21.

7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS

Impact of CSFs and programmes

7.1. - 7.15. In this part of the report the Court presents the results of different econometric and macroeconomic models but without any explanation of their scope or nature, which could lead to confusion. These data are presented in a structured manner in the report on Economic and Social Cohesion, with appropriate background information.

Impact of sub-programmes, measures and projects

7.16. - 7.19. The Court appears to criticise the lack of evaluation at the level of sub-programmes or measures, yet the programme evaluation, which is generally bottom-up, is necessarily based on the evaluation of measures and sub-programmes.

Similarly, the current thematic evaluations that have been launched by the Commission are concerned more specifically with certain categories of measure and/or sub-programme across the board, as in the case of measures for small businesses, R& D, the environment, etc.

8. CONCLUSION

8.1. - 8.19. The Commission believes that the Court's conclusions provide constructive criticism on issues which the Commission has been working on for years and will continue to develop.

The Commission admits that the Court has detected weaknesses but it has also noted the progress that has been made. The steps taken by the Commission under the aegis of the Structural Funds, reinforced by the SEM 2000 exercise, are helping to develop a culture of evaluation which did not exist before. There are few regions that have not carried out evaluations and that have not committed themselves to this process.

The subsidiarity principle and the difficulties and heterogeneity of geographic locations and subject areas are both factors to be taken into account.

(1) MEANS = Methods for evaluating activities of a structural nature: a multi-annual programme to analyse and propose methodologies of evaluation.