INTRODUCTION

The purpose of this toolkit is to present practical tools supporting the evaluation of projects aimed at increasing the employment of young people (aged 15-24), including those who face difficulties in transition between school education and work (NEETS).

 

The main recipients of this study are NGOs and other entities which want to analyze one of their projects in the abovementioned area. Such evaluation may be aimed at:

 

  • more efficient adaptation of the project to the needs of its beneficiaries and the labour market
  • measurement of the project’s effectiveness in achieving project goals and results
  • assessment of the project usefulness for the participants and sustainability of the results
  • examination the project impact on a wider group of people who did not participate directly in it
  • measurement of project effectiveness in terms of resources engaged in the project

 

This toolbox is a supplementary material to the course Towards better projects – blended learning course on evaluation, available on the MOODLE platform at our webpage (for free). While during the course, you can gain extensive knowledge about evaluation, the toolbox is designed to develop evaluation skills and support in using the acquired knowledge in practice. This is achieved, among others, by question sets, tables and tool templates facilitating the design and planning of evaluation, gathering the necessary information, and then formulating conclusions and recommendations aimed at improving the activities or projects carried out by the organization.

 

The toolbox has been developed by the Jerzy Regulski Foundation in Support of Local Democracy in Poland, in cooperation with Forschungsinstitut für innovative Arbeitsgestaltung und Prävention e.V. (Germany), Channel Crossings (Czechia), and PEDAL Consulting (Slovakia),within the framework of the Youth Impact project, financed by the EEA Financial Mechanism and the Norwegian Financial Mechanism. The project seeks to provide tools and services to improve the ability and capacity of Youth Employment and Entrepreneurship Support Actions implementers to efficiently evaluateimpact of their activities. The action will be carried out in the years 2019-2022.

 

The activities in this project are aimed at developing the evaluation competences of entities that support employment and entrepreneurship of young people.

GLOSSARY OF PROJECT TERMS

 

General objective – the future, expected state or effect of the organization’s activities conducted within the project, planned to be achieved within a specified time.

Example: Increasing by 2022 employment among young mothers who were unemployed in 2020 and benefiting from social assistance benefits in town X.

 

Specific objective – a planned state that will be achieved as a result of the implementation of certain activities. It should be consistent with the general objective and contribute to its achievement.

Example: Increasing by the end of 2021 the professional competences of young mothers who were unemployed in 2020 and benefiting from social assistance in town X to the level expected by employers from this town

 

Activity – actions aimed at a specific target group, which contribute to the creation of the planned outputs, and then the outcomes and achievement of the project objectives.

Example: Training in dyeing fabrics of 20 young mothers who were unemployed at the start of the project and benefiting from social welfare benefits in town X.

 

Sample selection – selecting from the population cases that will form the sample. It is conducted in a specific way (random or non-random) based on the sampling frame, i.e. a compilation (list) of all units making up the population from which the sample is drawn.

 

Generalization – generalizing the results obtained in the study of the sample to the entire population (i.e. also to units that have not been tested). In other words, based on the results from the sample, we conclude that the characteristics / opinions of the entire population are similar.

 

Logic model of change – a comprehensive tool for project planning and subsequent management of its implementation. It depicts the logic of intervention linking the individual elements of the project with cause-and-effect ties (see chapter).

 

Project logic matrix – a table used to determine the methodology of measuring selected project elements such as individual products, results or wider impacts. The matrix defines the indicators by which a given element will be measured, the measurement method, and assumptions (see: chapter).

 

Monitoring – ongoing collection, analysis and documentation of information during the project on the progress of its implementation in relation to the planned schedule of activities and budget.

 

NEET (not in employment, education or training) – the name of the group, mainly young people who remain outside the sphere of employment and education, i.e. people who do not study, work or prepare to practice, dut to various reasons (discouragement, life crisis, disability, parental or family responsibilities).

 

Impact – the effects of activities, outputs and outcomes of the project, contributing in the long term (apart from possible other interventions and factors) to changes affecting a wider community than the direct recipients of the project.

Example: improving the living conditions of children raised by women who got a job thanks to the professional competences acquired in the project.

 

Studied population – group of individuals (e.g. specific people, organizations, companies, schools, various institutions) that is the subject of the research / object of interest of the researcher.

 

Output – a short-term effect of an action in a material form (of a countable nature), e.g. a thing, an object, an event. These may be goods or services transferred to the project recipients, which are to contribute to the achievement of the planned outcomes.

Example: Training, training materials, certificates confirming the acquisition of professional qualifications in the field of dyeing fabrics by project beneficiaries.

 

Project (intervention) – a set of activities aimed at producing the intended outputs, which, when used by the target group of the project, should bring the planned outcomes.

 

Research sample – a collection of individuals selected in a specific way from a population for the purpose of research. Depending on the method and purpose of selecting these units, different types of samples can be distinguished.

 

Representative sample is a sample that reflects well (represents) the studied population and allows to accurately estimate its features through generalization.

 

Outcome – direct and immediate effects / changes that arise at recipients as a result of the implementation of specific project activities.

Example: project beneficiaries obtain gainful employment related to dyeing fabrics.

 

Output indicator – all goods or services created during the project implementation. It informs about the implementation of activities that resulted in the creation of measurable products.

Examples: The number of issued certificates confirming the acquisition of specific professional competences, and in other types of projects it may be e.g. the number of people who have achieved a certain level or an increase in the level of social competences according to the selected test, the number of sample cover letters and CVs prepared by the training participants, the number of textbooks prepared or good practices.

 

Outcome indicator – .informs about the degree of implementation of the changes envisaged in the project, which were to take place in the life situation of the beneficiaries as a result of their participation in project activities and the use of outputs produced at a given stage of project implementation.

Example: The percentage of beneficiaries of the project who, one year after the end of the project, have gainful employment related to the professional skills acquired in the project or better (declaration of the respondent).

 

Impact indicator – shows the delayed effects of activities, outputs and outcomes of the project that go beyond its immediate recipients. These effects usually cover the social environment of the target group of the project and may result from the accumulation of various activities (including non-project activities).

Example: percentage of project beneficiaries whose household does not benefit from social assistance benefits (due to non-compliance with the income criterion) 18 months after the end of the project.

I. THE BENEFITS OF EVALUATION

 

There are many ways to understand evaluation. According to the approach applied in the Youth Impact project the main goal of evaluation is to assess the effects of project activities. The assessment is based on facts and methodology of social sciences with regard to the change to be exerted by the project.

Our approach largely refers to the methods specific to impact evaluation. It is a data-based reflection on the activities and their real effects. It allows you to understand the determinants of the ongoing change, and focus on the sustainability of the achieved outcomes and the impact of the project that goes beyond its direct recipients. This evaluation also allows for the formulation of recommendations supporting project management, which contribute to the effective and efficient implementation of its objectives, as well as the mission of the organization, and take into account the needs of various stakeholders.

Such approach to evaluation enables to determine the value of a given project and to understand the reasons for successes and failures in its implementation. It is also a good management tool for organizations focused on social mission and other “learning” institutions.

 

BENEFITS OF THE WELL-DONE EVALUATION:

  • it allows you to PREDICT different DIFFICULTIES before you start the project (ex ante evaluation) or notice problems at an early stage of its implementation (ongoing or periodic evaluation), and also allows you to plan actions minimizing identified risks.
  • PROVIDES RECOMMENDATIONS concerning an ongoing or completed project: it indicates HOW TO IMPROVE a given project to better meet the needs of its recipients, ACHIEVE more durable and useful OUTCOMES, to HAVE A WIDER IMPACT, to FULFILL THE PLANNED OBJECTIVES using less resources.
Example: The use of the efficiency criterion in the evaluation makes it possible to assess whether, for example, a sufficient number of people were involved in the implementation of the project, and sufficient time and financial resources were allocated to the implementation of individual activities. On this basis, a decision can be made to strengthen the team in terms of staff or to properly plan project tasks.

 

  • INCREASES THE MOTIVATION OF EMPLOYEES – involving the project team in evaluation, especially at the design stage, as well a in discussing the results of the evaluation study, increases the sense of agency, emphasizes the relationship between the work performed and the goals of the project, the mission of the organization and one’s own values.
  • INCREASES THE KNOWLEDGE OF EMPLOYEES – from issues related to project management, through substantive aspects related to its subject, to knowledge of the mechanisms of the changes taking place.
  • influences INCREASING THE LEVEL OF CONFIDENCE AND COOPERATION WITH PARTNERS (ALSO IN FUTURE PROJECTS), thanks to taking into account the PERSPECTIVE OF EXTERNAL stakeholders in the EVALUATION.
  • enables to demonstrate the achieved results and improves cooperation with SPONSORS and GRANT SUPPLIERS, encouraging them to finance subsequent projects.
Example: In the application for co-financing, justifying the need for the project, you can quote the results of the evaluation that concerned an earlier / similar project. Providing reliable data may help you convince funders that your project will be effective, and therefore worth funding.

 

  • serves to PROMOTE THE ORGANIZATION.
Example: The case studies developed as part of the project evaluation can be used on social media to promote the activities of the organization. These can be, for example, stories of young people who, thanks to your support, acquired new competences and then found a satisfying job or successfully run their own business.

 

  • IT ALLOWS YOU TO ASSESS IF AND TO WHAT SCALE THE EXPECTED EFFECTS OF THE PROJECT WERE REALLY A RESULT OF PROJECT ACTIVITIES. Moreover, it makes it easier to decide whether a given PROJECT IS WORTH REPEATING, DISSEMINATING, or directed to a different target group.

 

Overall, evaluation has many benefits. Introducing it to everyday work can be a very useful support for managing an organization – educating and motivating staff, raising funds, strengthening credibility and improving the image, and above all, the effective fulfillment of the assigned mission.

II. PREPARING THE EVALUATION

 

“You can’t do “good” evaluation if you have a poorly planned program”.

Beverly Anderson Parsons (1999)

 

This toolkit, we concentrate on impact evaluation. We present practical ways of conducting evaluation focusing primarily on the effects of activities undertaken within a given project. The subject of the analysis are the effects of project activities (outputs, outcomes, impact) and their compliance with the project theory. The project theory defines the concept and plan of a project, including its objectives, activities, expected outputs, outcomes and impact, and how they will be measured, and what resources are needed to achieve them.

 

The most important element of the project theory is the so-called logic model of change that compiles information on what the project organization needs to accumulate (inputs / resources), the work it needs to do (project activities), and the effects it intends to achieve. The logic model of change for individual projects is created according to the following scheme.

 

The methods of measuring the outcomes of the project and the related assumptions are sometimes specified in a separate table called the project logic matrix. The logic model and logic matrix should be part of the project documentation.

 

In practice, it happens that the logic matrix or even the logic model of change have not been developed or are very selective. Lack of assumptions indicating what and how to measure to determine the success of the project makes it impossible to carry out a correct evaluation and thus verify whether the planned change took place and if it did – whether it occurred as a result of the implementation of project activities.

 

What to do if there is no logic model of change in the project documentation?

In such a situation, it is necessary to recreate the logic of change behind the project, e.g. on the basis of interviews with the management and project staff, as well as already existing documents such as strategy / project implementation plan, justification for its implementation, application for co-financing, partnership agreement, etc. The following table may help you to reconstruct the logic of the project.

 

Breaking down the logic of the project in such a way allows reflection on the ways of demonstrating the level of achieved effects (outputs, outcomes and impact). This goal is served by defining the indicators by which we will measure the progress of the project. An indicator is an observable attribute (feature) that enables the measurement of the phenomenon to be measured. Each indicator has a measure (quantitative, qualitative) which informs about the degree / intensity of the occurrence of a given phenomenon. In order to measure the change that has occurred as a result of the project implementation, you should determine the values ​​of a given indicator at the beginning and at the end of the project, i.e. the baseline value and the final value. It is also good to know the minimum value of the indicator, if such a value was defined, which at the beginning of the project was considered as confirmation of the achievement of a given project outcome. More information on indicators can be found in the online course (module 2).

Tool 1 (RE)CONSTRUCTION OF PROJECT LOGIC

Tool 2 TABLE OF INDICATORS OF PROJECT EFFECTS

Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.


Empty tab. Edit page to add content here.

Empty tab. Edit page to add content here.