COUNCIL RESOLUTION of 28 June 1983 on a Community plan of action relating to the evaluation of Community research and development programmes

THE COUNCIL OF THE EUROPEAN COMMUNITIES

Having regard to the Treaties establishing the European Communities,

Having regard to the communication from the Commission containing a Community plan of action relating to the evaluation of Community research and development programmes,

Having regard to the opinion of the Scientific and Technical Research Committee,

Whereas the Council has stressed that the aim of Community research should be to produce results capable of contributing to the economic, social and other objectives of the Community and its Member States and has requested the Commission to prepare specific proposals for a system for evaluating the results of common R & D programmes;

Whereas the Commission has addressed to the Council a communication on the "Exploitation and evaluation of research results" ; whereas the Scientific and Technical Research Committee has approved the general guidelines on evaluation put forward by the Commission in that communication;

Whereas the evaluation methods and procedures which the Commission intends to apply in the plan of action have been successfully tested during an experimental phase ; whereas the Commission's evaluation approach has been supported by evaluation experts and users of evaluation results at various levels within the Community institutions and within the Member States;

Whereas evaluation is an integral part of the continuing process of R & D policy and programme formulation, execution and revision;

Whereas evaluation constitutes a key element in the implementation of the framework programme for the Community's scientific and technical activities, which will contain a specific reference to it;

Whereas evaluation can effectively contribute to ensuring that timely decisions are taken on the establishment of R & D priorities, the choice of research areas and allocation of resources in order to meet the needs of new scientific and technological developments;

Whereas it is necessary to strengthen existing evaluation methods, and in particular the evaluation of the results of Community research programmes, in order more effectively to contribute to the R & D decision-making process;

Whereas it is necessary to stimulate activity within the Community in the field of evaluation in order to develop methods which can satisfactorily contribute to the achievement of the evaluation objectives;

Agrees with the importance attached by the Commission to the evaluation process,

Takes note with appreciation of the three year plan of action proposed by the Commission for the evaluation of Community R & D programmes which is attached to this resolution, and recognizes in particular the validity of the methodological approach the Commission intends to follow,

Notes the administrative, planning and budgetary provisions, associated with the plan of action, as envisaged by the Commission,

Invites the Commission to pursue its activities as proposed in this field and, if necessary, to make a further communication to the Council, in the light of the experience acquired.

Community plan of action relating to the evaluation of Community research and development programmes The evaluation of Community research and development programmes

PLAN OF ACTION

The evaluation activities included in his plan of action will play a key role in the context of the framework programme which the Commission proposed in its communication to the Council in December 1982 on "Proposals for a European scientific and technical strategy - Framework programme 1984 to 1987". The plan of action covers a phase of three years starting in January 1983. At the end of this phase the Commission will review the results of its actions in this field and or the basis of this review will, if necessary, make a further communication to Council on the implementation of a fully operational evaluation system, taking into account also developments in the framework programme.

This phase will enable the Commission to apply the results of its previous experimental period on a broader and more systematic basis. The evaluation methods used during the first test cases will be progressively applied to Community research and development (R & D) programmes at the appropriate time.

At the same time the Commission will further refine methodological and procedural aspects on the basis of more research in the field of evaluation and from the experience acquired through additional applications of the method.

The plan of action covers the following activities: 1. Continued strengthening of existing internal evaluation methods carried out during the implementation of R & D programmes in order to control effectively work progress and to adapt to changing needs, priorities and developments. The Advisory Committees on Programme Management (ACPMs) play a key role in this continuing internal evaluation, as does an effective monitoring and reporting system. ACPMs should provide the Commission with their evaluation of the results of the R & D programmes on a systematic basis, prior to any decision concerning programme-revision or extension.

2. The retrospective assessment of the results of Community R & D programmes carried out by external independent groups of experts. 2.1. The evaluation method is based on the principle of the assessment ex-post of the programmes, performed, programme by programme, by external groups of independent experts.

To a certain extent also some ex-ante evaluation aspects are included since the evaluation panels are expected to provide, on the basis of the retrospective assessment, recommendations for the future orientations of the programmes.

The evaluation method must be adapted to the nature and implementation procedures of the programme being evaluated and will therefore remain flexible.

A distinction should be made between the different methods of implementation used for the research: (a) For cost-sharing programmes (indirect actions) the "peer-evaluation method" will be applied. The objectives, which will vary to take into account the nature of the programme and the needs of the users, will, in general, cover the following aspects: - determination of the practical contribution of the results of the programme to progress of R & D in tile appropriate field within the Community, to Community objectives and to the socio-economic development of the Community in general,

- evaluation of the effectiveness of the management and of the resources utilized,

- recommendations on ways of exploiting research results and on the future orientation of the programme.

(b) For the programmes of the Joint Research Centre (JRC), the evaluation method which the Commission intends to apply will consist of two levels.

The first level encompasses the relevance and impact of the research results of the JRC, programme by programme, within a global strategy where the direct, indirect and other forms of actions are simultaneously used to reach a given target, i.e. within the framework of action programmes.

The first level covers the following aspects: - determination of the practical contribution of the results of the programme to progress of R & D in the appropriate field within the Community, to Community objectives and to the socio-economic development of the Community in general;

- recommendations on ways of exploiting research results and on the future orientation of the programme.

The second level deals with the problems of proper management practices : this second level of evaluation uses a number of permanent control bodies which are "built-in" in JRC and Commission structures : Advisory Committees, ad hoc expert groups, and, at overall level, the Governing Board, as well as the various control commissions of the Commission, of the European Parliament, etc.

Whenever a difficulty is identified, the Commission takes advantage of specific contributions such as technical or financial audits performed by specialized bodies or companies of consultants upon specific request.

(c) For concerted actions, including COST actions, the lighter "Hearing" method will be applied to take into account the more limited size, in terms of Community input, and scope of these types of research actions. The evaluation objectives will cover the following aspects: - assessment of the value and impact of concerted actions and of the benefits derived from the concertation,

- assessment of the effectiveness of the management and coordination of the actions,

- recommendations on the future orientation of the programme.

(d) Combined evaluations

Where it is felt that Community activities in a particular research area covered by different methods of implementation require simultaneous evaluation, the peer-evaluation method will be used. This procedure will be necessary in particular once the "action programmes" concept is implemented.

2.2. The evaluation panels should be relatively small to permit informal working procedures, numbering generally six to nine members, with appropriate member composition to include the necessary blend of competence.

Each panel will be free, within general guidelines, to establish its own specific evaluation methods and criteria.

The evaluations will take place approximately mid-way through each four-or five-year programme taking into account the results of the previous programme and the partial results of the current programme.

The results of the evaluations should be rapidly published and widely distributed. The reports will be submitted to the European Parliament, the Council, the Economic and Social Committee, the ECSC Consultative Committee, the Court of Auditors and other advisory bodies (CREST, ACPM, COMAC, etc.). Reports will also be distributed to other interested organizations, institutions and governmental bodies within Member States, including, in particular, potential users of research results in the industrial or other sectors.

Systematic feedback on the evaluations will be encouraged in order to ensure that programmes are meeting the real needs of the users. This will be done inter alia through oral presentations and discussions with users on the results of the evaluation and the organization of seminars on the impact and utility of current evaluation methods and procedures.

3. The Commission will carry out studies on certain evaluation aspects and encourage research in this field within the Community. In order to provide the panels with the most effective tools to carry out their tasks, the Commission will contract out specific studies in order to develop or improve certain methodological aspects. Given the relatively undeveloped state of the art in this field, this constitutes an important continuing task not only for the success of the Commission's own evaluations but for their application in other organizations and institutions in Member States. The Commission will widely distribute the results of its research.

4. The Commission will encourage the exchange of information in this field within the Community through the organization of workshops and seminars and by progressively establishing an informal Community evaluation network. The Commission will endeavour to keep abreast of national and international developments in this field. It intends to encourage the exchange of information on evaluation covering both its own experiences and those of other organizations and institutions in Member States. As part of the establishment of a network of experts in this field (national and international), it will organize periodic workshops and seminars on specific topics (e.g. assessment of the long-term benefits of research results or technology assessment methods), the results of which will be published.

Administration, planning and budget

In order to effectively carry out this plan of action and to ensure the necessary detachment from continuing programme-management activities, the Commission is maintaining and will be reinforcing a unit, independent vis-à-vis the R & D programmes, responsible for evaluation activities. The Commission intends to keep its administrative structure as light as possible in order to reflect the flexible and adaptable nature of the proposed evaluation strategy.

Over the next three years it is planned to perform approximately seven or eight evaluations per year and organize two seminars and a conference.

To carry out its plan of action the Commission envisages utilizing appropriations entered in the general budget of the European Communities under Chapter 72 - General and preparatory projects in the field of scientific and technological research (Article 721).

At the end of the three-year phase, the Commission will, if necessary, make a further communication taking into account the results of this phase and the requirements for the subsequent fully operational phase.