ENTREPRENEURSHIP TOOLKIT 

Your leverage to better projects supporting youth entrepreneurship 

 

TOOLKIT PDF

TOOLKIT E-BOOK

 

Authors: PEDAL Consulting s.r.o, Bratislava (2021)

 

1. Introduction

 

The purpose of this toolkit is to briefly present practical tools supporting the evaluation of youth           entrepreneurship support actions. In other words, the aim of the toolkit is to support evaluation of projects aimed at increasing the entrepreneurial skills of young people (aged 15-24) who, more than other age groups in the Visegrad region, face difficulties in transition between school education and work.

The main recipients of this toolkit are business support agencies, consultancy firms, business incubators and accelerators, social enterprises, NGOs and other small or medium-sized entities, which want to analyse one of their projects or statutory activity in the above-mentioned area. Such analysis will contribute to:

• Measuring the project’s effectiveness in achieving its goals and results,
• Assessing the project’s usefulness for the participants and sustainability of the project results,
• Assessing whether project activities should be continued or even scaled up,
• Evaluating the project impact with special regard of the project stakeholders and a wider community (their social environment),
• Measuring the project’s efficiency in terms of resources engaged for its implementation and delivery of results. 

This toolbox is a supplementary material to the course ‘Towards better projects – blended learning course on evaluation of entrepreneurship support actions’, available on the e-training platform HERE. The main idea behind it, is to guide the course participants to its practical application. To do this, we have developed a number of resources to help in this effort. The toolbox is intended to equip organizations, which aim to develop and/or increase the entrepreneurial skills among young people with practical tools to measure and continuously improve their impact. 

Lastly, this toolkit is primarily targeted to helping you to get the most out of your entrepreneurship      projects through evaluation.

The toolbox was developed by PEDAL Consulting (Slovakia), in cooperation with FIAP e.V. (Germany), Jerzy Regulski Foundation in Support of Local Democracy (Poland) and Channel Crossings (Czechia), as part of the Youth Impact project, financed by the EEA Financial Mechanism and the Norwegian Financial Mechanism. The Youth Impact project seeks to provide tools and services to improve the ability and capacity of implementers of Youth Employment and Entrepreneurship Support Actions to efficiently evaluate the impact of their activities and is being carried out in the years 2019-2022.

 

1.2.Characteristics of the targeted age group

 

As pointed out above, this toolkit is targeted at imparting practical tools to organisations supporting the evaluation of projects aimed at increasing the entrepreneurial skills of young people who face difficulties in transition between school and work. The situation of these young people most often referred to as NEETs (Not in Education, Employment or Training) can be described as follows:

• Their long-term unemployment rate was at 5.9% in 2017 (EUROSTAT).
• They have an uneasy position when striving for entrepreneurship. They have very limited access to financial resources, bank loans and credits because their bank history is very short or not existing. Therefore, they have no or very little capital to base their own businesses or enterprises on.
• Those who want to start their own business have a very disadvantageous position regarding experience. Either just after graduating from secondary or tertiary education with no or very little experience in the field of their degree, or with just a few years of work practice, the demands of competitive market and complexity of running own enterprise may often exceed far what they are able to invest.
• As a result of the COVID19 pandemic, many young people have lost projects, or were on the cusp of entering the workforce, have been exposed to the risk of missing out on building much-needed skills and experience during the crucial early stages of their careers.

It is therefore against this background that we set up this toolkit to enable organisations that are running Youth Entrepreneurship Support Action projects to help them assess and evaluate their projects and remedy any redundancies that may affect their quest to support young entrepreneurs to build scalable and innovative businesses that improve lives.

2. What is our approach to evaluation? What are the benefits? 

 

If you have an already existing youth entrepreneurship support action (project or programmes aimed at supporting youth entrepreneurship), or thinking about starting one up, then the approach to evaluation presented in this toolkit can be of use for you to find out and demonstrate if you are making the desired impact. 

Even though there is no single definition of the term evaluation, the UNDP defines evaluation as an assessment, of an activity, project, programme, or institutional performance. Evaluation should be conducted as systematically and impartially as possible. It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors, and causality using appropriate criteria.  An evaluation should provide credible, useful, evidence-based information that enables the timely incorporation of its findings, recommendations, and lessons into the decision-making processes of organizations and stakeholders (UNDP, 2019). Evaluation differs from monitoring and audit. For more information see the Toolkit section. 

Evaluation is a great idea if you want to: 

• Find out how well you are achieving your goals and results,     
• Learn if the needs of the target groups were identified properly or if your project is tackling real problems,
• Find out what impact your project or activities are having on your participants and others,
• Assess whether or not the project outcomes have justified the effort,
• Improve the outcomes of your partnerships. 
• Identify opportunities to expand the scope of your operations,      
• Use results to promote the project,
• Report progress to stakeholders or senior management,      
• Attract funding, as well as more or different partners.

Our approach therefore is to help you evaluate your project and show its benefits; give you some handy tools to use in your evaluation; and prescribe simple and effective methods of using evaluation to grow your operation or activities.  

And do not worry if you have an existing project and are yet to do any evaluation. It is never too late to start! Remember though that you cannot conduct every type of evaluation on the late stage of the project. 

The primary benefits of evaluation are:

• The use of impact evaluation allows you to evaluate not only whether the changes planned in the project have actually taken place, but also whether and how far the expected results of the project were actually the result of project activities.     
• The evaluation conclusions contribute to the optimization of actions for a specific project. It allows you to predict difficulties before the project starts (ex-ante evaluation) or notice problems at the early stage (ongoing or mid-term evaluation).
• It provides recommendations at the end of the project (ex-post evaluation) – these recommendations could help management in making informed decisions about specific activities sessions or even future projects. 
• The scope of possible use of evaluation in management depends primarily on the selected evaluation criteria. For example, the use of the project efficiency criterion allows us to assess whether a sufficient number of people, money and time were involved in the implementation of the project or whether the project team lacked any competences or other resources were insufficient. On this basis, you can decide to strengthen your team, budget or project planning.
• The results of the evaluation can be a tool for the promotion of your organisation and your achievements. For example, case studies illustrating the stories of successful recipients or data showing an increase in their competences can be presented on social media, on a website or during public presentations to promote the effectiveness of the organisation.
• Results of evaluation research can be used in communication with sponsors, including in grant applications. For example, including data on effectiveness, sustainability, relevance, utility, and other features of implemented activities in answering standard questions in application forms for financing similar projects.
• Evaluation can also be helpful in recruiting volunteers, especially when its results show the effects of work in the organisation, the extent to which the organisation has improved the lives of certain people or communities.
• Engaging the participants in the evaluation of a project, can also increase their enthusiasm and motivation. For example, thanks to their impact on the scope, evaluation criteria, as well as participation in commenting on preliminary results, their sense of agency increases, it illustrates the link between the work performed and the project’s objectives, organisation and own values. Participation in the evaluation process increases the participant’s knowledge and empowerment in many areas affecting the effectiveness of organisation: from management issues to substantive aspects related to the subject of the project.
• The involvement of project partners in evaluation – especially those who work with its beneficiaries. For example, involving the external partners (e.g., employers, institutions such as labour offices, associations of the group influenced by the project) of the project in the evaluation allows to diagnose problems, which limited the project effectiveness, relevance, or utility. In addition, describing and resolving the questions bothering these partners in the evaluation report can significantly increase the level of trust and cooperation with them also in next projects.
• In the field of projects related to vocational activation, employment and entrepreneurship of young people, the participation of beneficiaries is of key importance for the success of their evaluation – it allows to assess how project activities influenced the expected changes in the life of recipients. In addition, it can help in planning the next project to meet their needs and contribute to the achievement of the objectives.
• Evaluation is based on the methodology of social sciences and therefore provides reliable answers to questions relating to a particular project and the organisation implementing it.
• Project evaluation may also be suitable for comparing and finding out which individuals and with what characteristics are able to start a business and why others are less likely to do so.

In summary, evaluation has many benefits. The problem, however, is that knowledge about them is often abstract, not based on experience. Meanwhile, the systematic use of evaluation and its various techniques can support managing an organisation, strengthening its image, training, and motivating staff, obtaining funds, and above all effectively fulfilling the mission of the organisation.

3. How to use this toolkit

 

How you use this toolkit depends on where you are in the ‘life cycle’ of your youth entrepreneurship support action. 

If you want to start a project and build in regular evaluation from the beginning, you can use this toolkit in the planning stages of your project (ex-ante evaluation). 

If you already have an existing project but not sure how to measure the impact of what you are doing, then this toolkit can help you identify the information you need, show you how to gather it, and what to do with it once it has been collected (mid-term or ongoing evaluation). 

If you have an existing project and have been evaluating what you are doing, then you might want to focus on the sections that can refine your approach or follow up the references at the end of this document (ex-post evaluation). 

4. Type of youth entrepreneurship support action 

 

There is no single ‘right’ model of youth entrepreneurship support action, or only one proper youth entrepreneurship project. 

Some are targeted to different categories up on the entrepreneurship value chain that is, idea stage or growth stage. Others are more targeted to specific segments of the population for example, unemployed youth, women, immigrants and people with disabilities, or victims of forced migration. 

What your project looks like will depend on what you want to achieve and how you want to achieve it. In turn, how you go about your evaluation will depend on the type of project you have and what the evaluation objectives are.     

Once you are clear about the kind of project you have, or want to have, then you are in a better position to know what and how you want to evaluate and what you want to get from the evaluation.

Your project is bound to or may change over time (based on your experiences), and evaluation can      identify the best way to move forward. For example, it might show if your project has the right impact which you want to achieve. Or that you need to: 

• Improve a certain element of the project
• Redefine existing roles and responsibilities 
• Bring in additional resources
• Change programme structure
• Bring in a new partner
• Expand an existing project
• Develop a new project
• Plan better new programs

5. Types of evaluation according to the stage of evaluation

 

There are different evaluation types that vary mainly depending on the stage of the project. These types of evaluation are ex-ante evaluation, mid-term, on-going evaluation, ex-post evaluation (Types of evaluation, 2013).

 

Table 1: Types of evaluation according to the stage of project 

Stage of Project	Purpose	Types of Evaluation
Conceptualization Phase	Helps prevent waste, adjust a project to its recipient´s needs, and identify potential areas of concerns while increasing chances of success.	Ex-ante evaluation
Implementation Phase	Optimizes the project, measures its ability to meet targets, and suggest improvements for improving efficiency, relevance, effectiveness.	Ongoing or mid-term evaluation
Project Closure Phase	Insights into the project’s success/failure, utility, sustainability and impact, and highlight potential improvements for subsequent projects.	Ex-post evaluation

SOURCE: adapted from Nanda, 2017 

 

Below we provide brief characteristics of selected types of evaluation. 

 

Table 2: Characteristics of selected types of evaluation

Ex-ante evaluation
This evaluation is used before project implementation. It generates data on the need for the project and develops the baseline for subsequent monitoring. It also identifies areas of improvement and can give insights on what the project’s priorities should be. This helps project managers determine their areas of concern and focus and increases awareness of your project among the target population prior to launching it. When to use it:  New project development Project expansion	Helps make early improvements to the program. Allows project managers to refine or improve the project.
Mid-term or ongoing evaluation
Process evaluation occurs once project implementation has begun, and it analyses how effective, relevant and efficient your program’s procedures are. The data it generates is useful in identifying inefficiencies and streamlining processes and portrays the project’s status to external parties. When to use it:  When project implementation begins During operation of an existing project	Provides an opportunity to avoid problems by spotting them early. Allows project administrators to determine how well the project works.
Ex-post evaluation
This evaluation is conducted after the project’s completion or at the end of a project cycle. It generates data about how well it delivered benefits to the target group. It is useful for project administrators to justify the project, show what they have achieved, and lobby for project continuation or expansion. When: At the end of a project At the end of a project cycle	Provides data to justify continuing the project. Generates insights into the effectiveness and efficiency of the project.

SOURCE: adapted from Nanda, 2017 

6. How do I evaluate a project’s impact?

 

Just as there is no single ‘right’ model of a type of entrepreneurship project, there is also no single ‘right’ way to go about an evaluation. You might want to adopt more informal or more structured approach. Either is fine if it meets your needs. In any case, a certain amount of planning is of course important to achieve useful/meaningful results.

As this toolkit is aimed at organisations implementing or developing entrepreneurship projects targeted at young people, discovering if you are making an impact, measuring and continuously improving it can be achieved through a specific type of evaluation – the impact evaluation. 

Impact evaluation can be understood and conducted in various ways. Keeping in mind that this type of evaluation is key in analysing the changes that occurred thanks to your project. Based on the results of the impact evaluation, you will be able to adjust and/or improve the existing project as well as plan better new projects. However, this evaluation is most often used for analysing causality to conclude how much observed change is due to the project activities and influence.

Table 3: Characteristics of impact evaluation

Characteristics of impact evaluation
When?	At the end of the project After a defined period of time when the project has ended At pre-selected intervals in the project
What?	Assesses the intended change in the target population’s well-being Accounts for what would have happened if there had been no project
Why?	To show proof of impact by comparing beneficiaries with control groups (not participating in a project) Provides insights to help in making recommendations for preparations of new projects and/or to policy and funding decisions
How?	A macroscopic review of the project, coupled with a survey of project participants, to determine the effects achieved. Insights from project officers and suggestions from project participants are useful, and a control group of non-participants (very similar or identical in respect to some features with project recipients) for comparison is needed.
Questions to ask	What changes in project participants’ lives are attributable to your project? What would those not participating in the project have missed out on?

Source: Nanda, 2017

7. Phases of Evaluation 

 

It may help to think of the evaluation process as a cycle involving preparation, information gathering, information analysis, information use, further preparation and so on. 

In the framework of Evaluation of Entrepreneurship Support Actions Life Cycle below, which forms the basis for this section of the toolkit, these phases have been called: 

• Preparing – the success of your evaluation depends largely on the thinking and decisions you take in this phase,
• Gathering information – you need to collect the information most relevant to your purpose, and the way that guarantees it is reliable,
• Analysing information – you need to organise and interpret your information and identify your key findings, 
• Using information – this phase is about sharing the findings and making decisions about the future based on the outcomes of the evaluation process.     

 Figure 1: Evaluation of Youth Entrepreneurship Support Actions Life Cycle

 

In the following sections we explore each of these steps a little further and provide you with some support tools to help you along the way.

8. Preparing your evaluation

Figure 2: Evaluation of Youth Entrepreneurship Support Actions Life Cycle – Phase 1

Before you start preparing an evaluation, you should think about the entrepreneurship support action you want to evaluate and define:      

• What does this project try to achieve and why:
  • Is it meant to develop certain entrepreneurial skills? Which skills and how many people are to get them?
  • Is it focused on developing new businesses? How many businesses are to be established and supported? What level of their development is meant to achieve? 
• What are the risks and assumptions:
  • What are the skills gaps of the target group? 
  • What is their motivation and attitude to entrepreneurship? 
  • What resources are available? What obstacles for the business development are to be overcome?
• And how can you tell whether the project is a success once it’s implemented:
  • What are the indicators of achievement of the project goals?
  • How will you know these results are due to the evaluated project (e.g., how do you know if the increased number of new businesses is a result of the project?

Planning a good youth entrepreneurship support action can be a tricky task. Designing a good project requires understanding the situation in the region, defining a real problem that can be solved by the evaluated intervention. Other vital requirements are identification of the target groups and their needs, specifying objectives that are attainable within given timeframe and available resources, designing the most appropriate activities. The activities must be designed to serve the production of outputs and outcomes, which lead to the achievement of a defined objectives and a high-level goal.

Furthermore, the project implementers need to see the bigger picture during the project implementation, keeping in mind key questions: what are they trying to achieve, why and how they want to prove? 

• Do they plan to develop certain entrepreneurial skills? 
• Ensure there will be a certain number of business plans developed? 
• The number of new businesses will increase by some %? 
• What are the risks and assumptions (e.g., what are the skills gaps of the target group? 
• What is their motivation and attitude to entrepreneurship? 
• What resources are available to achieve planned outcomes and how you can tell whether the project is a success once it is implemented?
• And finally how will you know the observed effects are caused by the evaluated project?

Understanding the logic of the project is the key for preparing yourself to design an evaluation and provides you with information on the data produced within the project, which can be used in the evaluation. Often the Logical Framework Matrix or Logframe is prepared as part of seeking financial support for the project. Otherwise, you can reconstruct the project logic yourself by asking key questions about seven key areas of the project:

• Purpose – why has this project been initiated? What problem is to be tackled? What change is expected? 
  • If the results have been achieved, then certain effects may result in the target group. For example, the increase in knowledge can lead to a change in the participants’ behaviour.
• Outcomes – what results are expected? (e.g., increase in knowledge/entrepreneurial skills among participants, establishment of own firms by the project participants)
• Outputs – what are the deliverables?
  • Direct results of the activities are described. Following the “if-then” logic, this means that if an activity is carried out, then certain results are expected. Examples are business plans developed within the project; skills certificates issued to the project participants. 
• Activities – what actions have been planned to deliver the outputs? (e.g., workshops, trainings, mentoring)
• Indicators of achievement – how will we know if the project has been successful?
• Means of verification – how the reported results can be verified?
• Risks and assumptions – what are assumptions which underlie the structure of the project and what are the risks for achieving?

The template of the Logical Framework Matrix can be found in section 12.3.1. 

8.1 Concept of evaluation 

Similarly to developing a project, good evaluation requires answering a set of key questions that will help you define the concept of evaluation. 

Answering the 5 “W” questions is a good way to start: 

WHY are you going to do the evaluation? It is necessary to define the purpose of evaluation as the next parts of the concept depend on this question.

WHAT are you going to evaluate and what resources are needed? The subject and scope of the evaluation has to be clear, otherwise you might end up doing an evaluation for which you do not have the resources, expertise or time. The evaluation questions and evaluation criteria need to be formulated at this stage.

WHO will conduct the evaluation? Is it going to be a self-evaluation (staff implementing the project conduct the evaluation), internal evaluation (evaluation is conducted by own staff not participating in the project implementation) or external evaluation (evaluator from outside). Each type has its pros and cons. 

WHEN are you going to do the evaluation? Before the launch of the project (ex-ante evaluation)? During the project implementation (mid-term or ongoing evaluation)? Or after the end of the project (ex-post evaluation)? 

HOW will you conduct the evaluation? What sources of information will you use? What methods and tools of gathering necessary data will you use? What tools? 

The following sections will guide you through these questions. 

8.2 Be clear about the purpose subject, and scope of your evaluation

What is it that you want to find out? What do you want to learn from the evaluation results? How do you plan to use the results? 

EXAMPLE

Defining the purpose of evaluation

For more than 25 years, Junior Achievement Slovakia (JA Slovakia) has been helping teachers develop entrepreneurship, economic thinking, and financial literacy of students in Slovakia. 

The mission of JA Slovakia is to help teachers to develop entrepreneurship, economic thinking, and financial literacy among students of primary and secondary schools. This is done primarily by the means of experiential learning, in which experienced professionals are involved. JA Slovakia is a member of a worldwide network of 115 JA organizations and a member of a network of 41 JA Europe organizations. For 100 years, this network has been bringing education and skills development in job readiness, financial literacy, and entrepreneurship around the world. 

The YEEAs run by the organization 

In the programs ‘Applied Economics’ and ‘Entrepreneurship in Tourism’, students have the opportunity to run their first real business in a student company. Skills for employability and creating their own idea are developed in the ‘Skills for Success’ program. The development of ethical aspects of business and the moral values ​​of the individual is the subject of the ‘Ethics in Business; program. Pupils’ financial literacy is increased through the ‘More than money’ and ‘Me and money’ programs, which are created in accordance with the National Financial Literacy Standard. The youngest ones can prepare for their future profession through the ‘Fundamentals of Business’ program.

The scope of evaluation in JA

JA Slovakia has long used the available evaluation options taking into account their time and financial constraints. The main purpose of the evaluation is to assess the growth of the participants´ knowledge by testing their skills at the beginning of the education / school year and at the end. The aim is to compare the results and determine the progress over a period of about 10 months to see if the programme is addressing the needs of the target group or if any improvements are needed. 

Source: Bednárová, 2021; Junior Achievement Slovensko, n.o., n.d.

The subject and scope of the evaluation needs to be defined clearly. Are you going to evaluate a specific entrepreneurship support project? Or only a part of it? What exactly do you want to learn? 

Do the project goals, activities match the goals and priorities of the organisation?

EXAMPLE

The evaluation will focus on the programme ‘More than money’.

Financial literacy test: Pupils involved in the ‘More than money’ program take a ‘central entrance’ test at the beginning of the school year. The aim of testing is to determine their initial level of knowledge in the field of financial literacy.

At the end of the school year, students take a central exit test. Its aim is to verify the level of knowledge achieved by students after completing the program.

Source: Bednárová, 2021; Junior Achievement Slovensko, n.o., n.d.

8.3 Define the evaluation criteria and questions 

The next step is to decide about the evaluation criteria that are linked to the purpose of evaluation. The criteria provide a perspective through which you can look at your project and formulate the evaluation questions.

The criteria describe the desired attributes or aspects of the project which you want to verify/assess: e.g. an intervention should be relevant to the beneficiaries’ needs, coherent with other interventions, effective in achieving planned objectives and outcomes, deliver results in an efficient way, and have positive impacts that last (OECD, n.d.). 

You will also need to identify some key questions (evaluation questions) that you want the evaluation to answer. For example, you may want to ask how participants have benefited from being in the project under evaluation, or if it achieved what was expected to do, or if all parties involved in the project participated as planned. 

Criterion: RELEVANCE

Evaluation question: IS THE INTERVENTION DOING THE RIGHT THINGS?  

The extent to which the intervention objectives and design respond to beneficiaries’ needs

Criterion: EFFECTIVENESS 

Evaluation question: HAS THE INTERVENTION ACHIEVED ITS OBJECTIVES?

The extent to which the intervention achieved, or is expected to achieve its outputs, outcomes and objectives.

Criterion: EFFICIENCY 

Evaluation question: HOW WELL ARE RESOURCES USED?

The extent to which the intervention delivers, or is likely to deliver, planned outputs and outcomes in an economic and timely way. 

Criterion: UTILITY

Evaluation question: HOW USEFUL ARE PROJECT OUTCOMES FOR ITS RECIPIENTS?

The extent to which project outputs and outcomes were useful for their recipients.

Criterion: IMPACT 

Evaluation question: WHAT DIFFERENCE DOES THE INTERVENTION MAKE?

The extent to which the intervention has generated significant, positive or negative, intended, or unintended, higher-level effects.

Criterion: SUSTAINABILITY 

Evaluation question: WILL THE OUTCOMES OF THE INTERVENTION (ITS BENEFITS) LAST?

The extent to which achieved outcomes will be sustainable in time, after the project completion and financing.

Criterion: COHERENCE 

Evaluation question: HOW WELL DOES THE INTERVENTION FIT? 

Synergies and interlinkages between the evaluated project and other projects implemented by the same organization or a wider programme within a given project is implemented.

All your evaluation questions need to be related to the scope and objective(s) of the evaluation. Keep it simple and achievable. Below are some examples of possible evaluation questions and related evaluation criteria proposed for an existing project. 

EXAMPLE 1.

Project outcome: development skills for success – basic skills for employability and a proactive approach to entrepreneurship development, own work, the ability to solve problems, identify, design and develop their own idea.

Evaluation questions: To what extent have the planned outcomes been achieved (EFFECTIVENESS)? Do the project goals, activities and outcomes match the objectives and priorities of the organization (COHERENCE)? Are the project outcomes useful for its recipients (UTILITY)? Are the project outcomes sustainable (SUSTAINABILITY)? How efficient was the use of the project resources (EFFICIENCY)?

EXAMPLE  2.

Productive post-project pathways: 

Objective: To improve youth transition from university to entrepreneurship debut options. 

Evaluation question: How, and to what extent, is the project helping the youth in successful transition to entrepreneurship or to work (IMPACT)?

EXAMPLE 3. 

Tangible results for the lower qualified participants: 

Objective: To equip participants with lower academic qualifications with the necessary skills to start enterprises in their communities by the end of the project. 

Evaluation question: How is the project able to harness the entrepreneurship skills of these young people to help turn their ideas into profitable enterprises (UTILITY).

Source: own elaboration

In table you can find examples of evaluation criteria and questions. 

Table 4: Evaluation criteria and questions

QUESTION	EVALUATION CRITERIA
Was the project budget sufficient to achieve the plan correctly?	Efficiency
To what extent have the planned objectives and results been achieved?	Effectiveness
How are project resources used?	Efficiency
What went wrong and why?	Effectiveness, Impact
Do the project goals, activities match the goals and priorities of the organization?	Coherence
Are the results obtained permanent?	Sustainability
What helped the implementation of the project?	Efficiency
What slowed down the implementation of the project?	Efficiency
How does the community perceive the project?	Impact
to what extent were project results useful for its recipients?	usefulness
To what extent was the project relevant to the needs of its recipients?	relevance

Source: own elaboration

Having defined the questions your evaluation should answer, you ought to proceed with setting the evaluation indicators that are key for measuring project effectiveness.

8.4 Project indicators

Project indicators measure effects against project goal and objectives. In such context an indicator is used as a benchmark for measuring intended project effects. Indicators can be quantitative (e.g., a number, an index, ratio or percentage) or qualitative (depicting the status of something in more of qualitative terms, e.g., whether a local start-up ecosystem is more developed after the project than it was before the beginning of the project – according to opinions of key informants or judging by the quality of local regulations concerning business activities). Indicators can show if your project has produced the expected outcomes. Why defining indicators is important in the evaluation process (Selecting project indicators, 2013)?

1. At the initial phase of a project, indicators are important for the purposes of defining how the success of the intervention will be measured and what level of given indicator or its dynamics should be considered as satisfactory for meeting the respective objective.
2. During project implementation, indicators help assess project progress and highlight areas for possible improvement. 
3. At the final phase, indicators provide the basis for which the project assessment may become dubious.

There are three types of project indicators that are widely acknowledged and can be used when conducting evaluation based on such criteria as effectiveness, efficiency or impact:

1. Process indicators: are used to measure project processes or activities. For example, this could be ‘the number of training activities organised in period XY’.
2. Outcome Indicators: measure project outcomes. Outcomes are results of a project. For example, it could be ‘how the level of entrepreneurship skills was improved’.
3. Impact Indicators: measure the long-term impacts of a project, or simply the project impact, e.g., ‘the number of new start-ups established by youth entrepreneurs’.

Also, other criteria (relevance, utility, coherence) and evaluation questions related to them can have their own indicators, but any appropriate indicator must have particular characteristics (Bureau of Educational and Cultural Affairs, n.d.) that are listed in table 5.

Table 5: Characteristics of indicators

Characteristic	Description
Specific:	Probably the most important characteristic of indicators is that they should be precise or well defined. In other words, indicators must not be ambiguous. Otherwise, different interpretations of indicators by different people implies different results for each.
Measurable	An indicator must be measurable. If an indicator cannot be measured, then it should and must not be used as an indicator.
Achievable / Attainable	The indicator is achievable if the performance target accurately specifies the amount or level of what is to be measured in order to meet the result/outcome. The indicator should be achievable both as a result of the program and as a measure of realism. The target attached to the indicator should be achievable.
Relevant	Validity here implies that the indicator actually measures what it is intended to measure. For example, if you intend to measure impact of a project on development of specific entrepreneurship skills, it must measure exactly that and nothing else.
Time-bounded	The system [monitoring and evaluation system and related indicators] allows progress to be tracked in a cost-effective manner at the desired frequency for a set period

Source: adapted from Selecting project indicators, 2013; Bureau of Educational and Cultural Affairs, n.d.

8.5 Identify the type of information you need 

Measuring success may require the collection of information before, during and after your project. You have to identify the types of information you will need earlier on and ensure that they are ready and available when you need them.

There are two broad categories of information to consider:

• Quantitative: this is information that is concerned with counting and measuring things, like attendance, sessions, or scores.
• Qualitative: this information is concerned with people’s feelings, thoughts, perceptions, attitudes, behaviour change and beliefs, and may include things like improved participant attitudes to specific project sessions identified through observation, interviews, and feedback forms.

Below are some ideas for how you might capture, through questions, the different categories of information.

Sample questions for evaluating a project 

Knowing what the main purpose of evaluation (see the previous section) is, you can think about the specific questions to be used in the tools to collect qualitative and quantitative information (e.g. tests, questionnaires, etc.)

Questions you can use in tools gathering quantitative data 

• How many trainings focusing on the development of entrepreneurial skills were organized? 
• How many staff from the organisation and business community were involved in the mentoring program? 
• What was the increase in the number of participants that occurred after involving the local business community in the mentoring project? 
• What was the increase in number of participants choosing sessions from the project because of the mentoring program? 
• How many participants have gained positive outcomes from participating in the activities?     
• How many participants were there at the beginning of the program and how many participants completed the program?

Questions you can use in tools gathering qualitative data 

• What were your feelings about having a mentor before you joined the project? 
• How do you feel about the experience now that you have been mentored for a year? 
• What (if any) parts of the experience did you find most enjoyable? 
• What (if any) parts of the experience did you find challenging? 
• What advice would you give other participants coming into the mentoring program next year?     
• Why did some participants terminate the program? 
• Please think about the period after you started participation in the project. Have you observed any significant change in the way you live your life in this period? 
• (If yes) Please mention the areas in which you observed these changes. Please say what was the main factor that caused each of these changes?
• Please think about the period after you started participation in the project. Have you observed any significant change in the way your start-up was functioning in this period? 
• (If yes) Please mention the areas in which you observed these changes. Please say what the main factor was, that caused these changes. 
• (If the project was mentioned as main factor) Which specific workshops moved your start-up the most? 
• Is there anything in the project you think should be done another way (concerning the organization, lecturers, times, etc.)? What would you like to be different?
• What should be different (if anything) so that you can make more use of contact with other participants, to benefit from being part of the community? 

8.6 Be clear about your stakeholders

In any youth entrepreneurship support action, there are multiple stakeholders. Stakeholders can vary from project representatives of project partners including business incubators, accelerators, universities local or regional administrative authorities, employers’ organisations, donors, mentors, trainers, coaches, project staff and participants. 

You will need to think about the audience for your evaluation, who your stakeholders are, if and how some of them can be involved in the evaluation process, what information they can provide you with or what evaluation criteria and questions would be important for them.

If the evaluation conclusions are to be implemented than it is vital to identify the stakeholders´ evaluation needs in the preparation phase. Otherwise, you might not be able to satisfy them, as the results of evaluation will not provide information expected by the stakeholders. 

If possible, integrate stakeholders into the evaluation. This allows a comprehensive insight and change of perspective. Especially the involvement of the target group (beneficiaries) is an important issue, because you are going to be in touch with them anyway, so they are already there. Participative evaluation is a simple tool to ensure integrating stakeholders into the evaluation.

8.7 Identify potential sources of help 

Before gathering your information, think about the kind of help you may need and when you may need it.

Potential sources of help 

• An independent person, such as someone from another organisation carrying out similar projects, stakeholder, or business, could help during the preparation and planning part of your evaluation. 
• External experts could assist with the evaluation concept, design of surveys and data analysis, as well as methodological supervision of your research tools (interview scenarios and questionnaires). 
• Some information gathering might best be done independently by a third party (e.g., collecting and analysing some data by an external expert, statistics collected by public authorities, etc.). 

Depending on the nature of the project, its stakeholders could participate in the designing of the evaluation concept, support data collection, help in interpretation of the results and formulating recommendations bringing about their experience and expertise in the project. 

8.8 Identify the sources of the information you want to gather 

To get a good perspective about the participants in the project, you could think about using internal information such as: 

• Attendance records 
• Retention rates 
• Post-project tracking data (e.g., data on developments of businesses launched by the participants, data on project beneficiaries employment status, participation in mentorships etc.) 
• Participant portfolios on participants’ businesses developments launched during the project (e.g., technology, social enterprise, business, novel ideas) 
• Participant behaviour records (e.g., time-out, commitment, retention) 
• Participant’s, parent’s, professional assistant’s or instructor’s opinions on particular aspects of the project, including their needs, satisfaction 
• Participant achievement data (e.g., pre-tests and post-test results).

From perspective of youth entrepreneurship support action implementer, you could think about: 

• Interviewing corporate volunteers 
• Records of financial and in-kind support 
• Level of media coverage or reach of the communication and dissemination efforts 
• Asking participant about their satisfaction (e.g., using a survey and interviews)
• Numbers of participants directly and indirectly impacted by the activities or projects 
• Sales figures or other evidence of marketing success 
• Interviewing the project staff.

To ensure objectivity of the evaluation results, you should also consider information that could be available from external sources. These can include statistics, surveys, or analysis developed by: 

• labour offices that might keep statistics of the number of graduates registered as unemployed persons in the monitored period;
• municipalities that can monitor the entrepreneurship activity in the respective region;
• secondary, tertiary and other educational institutions, which may provide statistics collected on their graduates / alumni clubs;
• government level / ministry of labour, social affairs and similar, that may systematically approach the issue of NEET employment and entrepreneurship; 
• non-governmental organisations that deal with youth (e.g., NGOs collaborating with universities, such as AIESEC, IASTE, ELSA),
• or networking spaces, such as community centres, leisure centres, co-working spaces, etc. 

Make the most of existing information:

The history of the project could be traced through such documents as: 

• Planning documents, especially Project Logic or Logframe, grant application documents etc.
• Communications – emails, records of phone conversations between partners 
• Original timelines and budgets 
• Business or strategic plans 
• Minutes of meetings 
• Evidence of community consultation 
• Memos, and 
• Financial records.

Other information could be gathered by interviewing people who have been involved since the early days of the project. 

You could find out what people remember about the beginning of the project: 

• their initial expectations and motivations to join the project, changes in expectations in the course of the project and to what extent the expectations were met 
• early roles and responsibilities 
• expected and actual challenges and 
• proposed ways of addressing these.

In this way you can build up a picture of how the project has evolved and if it is still serving its originally intended purpose. 

There are many different methods of gathering data from different sources of information, each with its own advantages and disadvantages, such as desk research, interviews, case studies, observations, and surveys. See the section 9, ‘Gathering Information’, for more ideas on how to collect information from a variety of sources.

8.9 Design of impact evaluation 

Impact evaluation is the type of evaluation that focuses on factors which caused the observed change in target group of the evaluated project. Using a combination of the following strategies can support the conclusions drawn (Peersman, 2015):

• estimating what would have happened in the absence of the evaluated project, compared to the observed situation,
• checking the consistency of evidence for the causal relationships described in the project logic framework,
• ruling out alternative explanations, through a logical, evidence-based process.

There are three designs that allow for implementing these strategies. Experimental designs and quasi experimental designs are based on the principle of comparing the situation before and after an intervention in two groups – a treatment group consisting of participants benefiting directly from the evaluated project and the other group, which includes individuals of similar characteristics who were not supported by this intervention/project.

1. Experimental designs – in which ‘the other group’ is called a control group and the assignment to this group as well as to the treatment group is based on random mechanism. Due to these features this design is often called randomized controlled trials (RCTs). 

The main precondition for an RCT is that the number of individuals interested in your project is greater than the number participants you can provide support to. The treatment group and the control group should be similar in terms of their features as age, education level, employment status, etc. Randomized selection can be conducted in various ways, e.g., computer generated assignments, or a lottery. The main principle is that all individuals have an equal chance to be selected to both of the groups.

1. Quasi-experimental designs – in which ‘the other group’ is called a comparison group and is constructed by using various techniques to secure optimal similarity or controlled difference to the treatment group. A selection to both groups is based on non-random mechanism (e.g.  the compared groups contain only the people who were close to the project admission border, and they are selected from the project recipients and from candidates who were not included in the project). 
2. Non-experimental designs – which look systematically at whether the evidence is consistent with what would be expected if the intervention was producing the planned impacts (e.g., sequence and timing of project activities and effects go as assumed by project’s logic), and also if non-project factors could provide an alternative explanation to the observed effects.

 

9. Gathering Information

Figure 3: Evaluation of Youth Entrepreneurship Support Actions Life Cycle – Phase 2

This section of the toolkit looks at: 

• Different ways of gathering information 
• How you might benefit from a particular source of information by using respective evaluation methods. 

An evaluation can use quantitative or qualitative methods, and most often includes both as they can complement each other and mutually balance their weaknesses (CTSA, 2011). 

9.1 Quantitative methods

These methods provide quantitative information concerning which measures the scale (how much), the intensity (to what extent) and frequency (how often) of the examined phenomena, for example effects of the project implementation (e.g., the number of people who used knowledge and skills acquired in the project in order to develop and implement their business plans.

Quantitative data can be collected by surveys (using e.g., questionnaires, pre-tests and post-tests, observation, review of existing documents and databases or by gathering clinical data), using various communication channels. Especially the COVID-19 pandemic resulted in wider use of on-line communication channels, methods (e.g., computer assisted web interview – CAWI) and tools (e.g., mentimeter). 

Surveys 

Surveys that are based on self-administered questionnaires (paper or online) are quick and inexpensive ways of finding out what people think, do and what is their situation. Surveys questions can be also posed by pollsters during telephone or face-to-face interview, but such methods are much more expansive.

For example, you can survey project recipients to see what they have gained from the project and what they have gained as a result of this exposure to business or industry experiences. You should survey project participants to find out what they have learned from participating in the respective activities, including those developed by themselves within the project, and if they have any suggestions for improving either outcomes or processes. You may also survey parents or guardians of the youth participating in a project to see what impact the project has had on their children. If you want to show the change caused by the evaluated intervention, the information should be collected before and after the implementation of a project (e.g., the level of knowledge or certain skills that were developed in the project). 

Remember that the questions in your evaluation tools (that are self-administered questionnaires or questionnaires navigated by pollsters) must first of all be strictly connected with the chosen evaluation criteria and evaluation questions. Once your evaluation tools are ready, you can conduct your own online survey using free online software.

The quantitative data are easier to analyse than the qualitative ones and can be generalized. The generalization means that the findings can be applied to a wider group (population) than those participating in the survey (sample), provided that the sample is big enough and accurately represents the population. If collected properly, the data are reliable, and their precision can be estimated. However, collecting quantitative data can be challenging due to difficult reach to sampling frame data and contact details of sampled people, as well as difficulties in reaching respondents in the sample, their deficit of time and motivation, not to mention serious consequences of imperfections in the questionnaire design. Also, there are limitations in the type of information you can get from the quantitative data. It does not provide you with insights and explanations of the context and more complex problems, such as causes and consequences of the studied issues. 

To show the change caused by the evaluated intervention, the data should be collected before (pre-test) and after (post-test) the implementation of a project, for instance the level of knowledge or skills that were developed in the project, number of businesses that were registered. For that purpose, various tests and databases can be used. 

9.2 Qualitative methods 

These methods serve a better knowledge, deeper understanding and explanation of the project process and its effects. Qualitative data can provide you with in-depth information answering questions asking about what happened, in what way went well and why (e.g., what went right and wrong or what did not go so well and why, and what were the factors that caused these effects) contributed to the result achievements, etc.  

To collect qualitative data, you can use in-depth interviews (individual or group ones, so-called focus groups), observation, and case studies as well as desk research that is analysis of documents including testimonials, diaries, staff journals, logs, etc. Let’s review them briefly.

 

Individual interview 

An interview is basically a conversation and can be more or less structured (based on a scenario with a prepared set of questions), semi-structured (with prepared questions that you might add to, adapt or omit as the interview progresses) or unstructured (where interviewees talk with only the occasional prompt questions from an interviewer). 

Interviews can be adapted to fit in with the availability of your interviewees – during project hours, face-to-face or by phone or internet communicator, onsite or offsite. 

How you conduct your interviews will depend on the project you are evaluating. For example, participants in a mentorship or apprenticeship project might have less formal, unstructured conversations about how the activities or project or sessions are going. When interviewing participants in an idea phase of the project you may want to use semi- structured interviews by phone or in person. 

Interviewees can be given some guiding questions (but not the actual scenario – set of interview questions) beforehand as this allows them time to think about their responses.

 

Focus group 

A focus group is a kind of an interview in a small group (5-8 persons) with a moderator and a group of participants. Focus groups usually invite a range of people to discuss a common topic. They can be useful if you want to bring together, say, a few participants, parents, staff, trainers, mentors, project stakeholders or partners to discuss their different perspectives on a project, its impact and how they think it may be improved. 

It can help to have one person facilitating the discussion (a moderator) while a second person takes notes. The most convenient way (same as for individual interviews) is to ask interviewees for consent for recording). Then, you can summarise conducted interviews in writing, or verbally, the findings from the discussions.

 

Observation 

Observation is data collection method based on careful and systematic experiencing (by seeing, hearing) various events and phenomena. It can be supported by observation sheet or a checklist that focuses the observer’s attention on certain issues, such as the activity level and interactions between training participants (Bartosiewicz-Niziołek, M., Nałęcz, S., 2021). Observation is data collection method based on careful and systematic experience of various events and phenomena by observing participants’ behaviours in a natural situation. It can be conducted without any tool (free description observation) or be supported by an observation sheet. This tool focuses the observer’s attention on certain issues (e.g., activity level and interactions between training participants, trainer’s engagement, a way of presenting issues, involving trainees, and conducting training). 

You can also use observation checklists or make notes to record activities of participants or staff as well as their responses. An observation sheet enables you to record the dynamics of the interaction between participants and staff or among participants. To measure participant engagement, you might use a simple yes/ no or tick/cross system to record participants’ body language or verbal comments, the kind of questions they are asking, the amount of confidence being shown, or the extent to which they are following staff or mentor suggestions. You also can use a simple tally to record, say, the number of participants engaged in activities at various points during the course of the project.      

An observation checklist/observation sheet should be focused on the information needed to answer the evaluation question and ought to be prepared some time before coming to the observation site.

It is a good idea to gather other sources of information besides observed information for a more complete picture of participant outcomes.

 

Case study 

A case study is an in-depth analysis of a particular story or experience. It can be a powerful illustration of how an activity or project has had an impact at an individual, chosen project or activity. Case studies capture the story behind the statistics or other information and are particularly effective when combined with various sources of information. 

For example, a case study might concern a particular participant, highlighting the skills and knowledge gained, and the benefits of participation identified by these persons. By bringing together the participants’ voice with other evidence, it is possible to build up a broad picture of the impact the project causes and understand the processes which contributed to the results. This other evidence could include comments you have gathered from the business coordinator of the project, other business staff who contribute their skills, mentors and several parents or guardians of the participants involved.

 

Desk research (analysis of documentation)

Documents can be a useful source of information about the beginnings of your project as well as provide information on the activities carried out, project beneficiaries and their progress etc. Keep in mind the purpose of the evaluation when selecting materials for reviews. Especially, in the case the logic framework is missing, such documents can help you develop it at later stages of the project. 

Relevant materials might include early planning documents such as business or strategic plans, communications between partners and advisory services, financial records, minutes of meetings, letters of agreement or memoranda of understanding and the like. 

If you are already in a well rolled out project spanning a period of time (or has been active for a number of years consecutively), it is likely you will have access to some of these documents and they can be included in your evaluation (e.g., reports, needs assessments, knowledge or skills tests, trainers’ and trainees’/interns’ write-ups, checklists). Also, you may gather information from individuals who have been involved in the project from the beginning. You could find out what they remember about the reasons for starting a particular project, session, or other activity, what they hoped to achieve and how well they think the early objectives and expectations have been fulfilled.

One of the simplest documents you can use is a checklist that is a quick and easy way of recording basic information. It is used when there are specific items or actions in a project to be recorded.

You can use a checklist to: 

• Summarise actions or activities that participants have engaged in, 
• Track participants’ progress over time,
• Record observations – more on these below 
• Show the tasks a participant has completed, or which aspects of a task have been done. 

Checklists may be maintained by project or support action staff, by participants or by all parties. An example of a checklist can be found in the later pages of the toolkit.

If you are just starting out in a youth entrepreneurship support action, keep your early records that provide relevant information about the project. They can be a useful source of information to track progress of your activities or interventions.

When analysing documentation, you can also use desk research method (analysis of such documentation as diaries, journals and logs including personal reflection) that can give you a rich picture of people’s experiences. Entries can show how a recipient’s thinking or understanding has changed as a result of participating in a project. They might show what the participants were expecting at the beginning and whether or not these expectations have been met. They can show if a trainee/intern has become more confident or gained particular skills or knowledge. Staff journals can also track student development or the staff’s own growth in professional understanding. In turn, partners may choose to keep a log of meetings and activities to track progress or document what has happened. These reflections can be less or more formal. This kind of information may be especially useful for evaluations focused on the change or growth over time. It is worth remembering to obtain consent from the owners of these diaries, journal, and logs to ensure compliance with the General Data Protection Regulation.

You can also make use of a testimonial or reference that enables people outside the project to show their support for it. Testimonials offer evidence about the strengths and weaknesses of a project. They are typically, but not always, a response to your request. Like case studies, testimonials might provide a more complete picture of what has been achieved from the perspective of a project participant or even someone outside the project. A letter from the local council, appreciating the use of an innovative business solution to solve a community problem, for example, might be used to demonstrate the impact of the project in a particular area. Another example could be if a local chamber of commerce passes on their feedback from businesses regarding the conduct and performance of participants during mentorship or apprenticeship activities delivered as part of a collaborative project.

 Moreover, visual records can be also analysed as a part of desk research. These recordings may serve as evidence of engagement in an activity or show the impact of a project on participants. Photos and videos can set the scene for an evaluation by showing the environment in which the project is taking place and the people involved. Depending on the nature of the activities that participants are involved in, photos can show progress towards an end product and the end product itself. Consent (where necessary) should be obtained from participants (and from mentors and other stakeholders) prior to taking these images. Where applicable, it must be taken into account who and for what reason the images were taken to give the images context.

 

Skills assessments

In order to show participants’ progress, some youth entrepreneurship support action implementers may also routinely conduct tests and other assessments that can be used within desk research. There are several tests that could be taken in these circumstances that can show student performance levels before and after engagement in a session or series of activities. However, it may be difficult to show there has been a change in performance as a result of your project activities alone. Keep in mind the external factors that might have contributed to or counteracted in achieving the observed results. You can find more on skill testing in the section 12.4.

9.3 Advantages and limitations of some common information collection methods 

You may not be entirely sure about which information collection methods you should use for your evaluation. Some will be more suited to your purposes than others. 

To help you decide what information would be best for your evaluation, look at the following table, which shows the most important advantage and one key limitation for each information collection method. However, it is very important to remember that in evaluation research you should always use both, qualitative and quantitative methods in order to obtain complementary information and thus, stronger evidence and the full picture of an evaluated project or its effects.

Table 6: Advantages and limitations of collection methods

Information Collection Methods	Main advantages	Main limitations
Individual interviews	Allows you to capture a range of in-depth participants’ perspectives	Can be time consuming when it comes to conducting and analysing the data from the interviews; requires skilled interviewer
Focus Groups	Interactions among participants provide a wider perspective and enable confrontation of various opinions	Can be difficult to arrange (face-to face version), and time consuming (as above); requires skilled moderator
Case Studies	Serves as an example and can provide an in-depth and holistic picture of impact on participants	These are about particular instances and not generalisable
Observation	Allows you to analyse the beneficiaries’ reactions in natural context to see participant involvement first-hand	Shows only the observable       behaviour not what participants are thinking
Desk research (analysis of documents, such as diaries, journals, and logs, testimonials, photographs, and videos, tests and other assessment)	Can give a context to the      project, including early expectations and objectives	Early planning documents can sometimes be hard to find
Surveys	A reliable, comparable, and generalizable, as well as quick and low-cost way of capturing views, behaviour and facts	Does not allow for more in-depth perspectives

Source: own elaboration 

Remember that in evaluation research different kinds of methods should always be used together as they complement each other. The use of both, qualitative and quantitative methods enable overcoming their limitations.

9.4 EVALUATION TOOLS 

Evaluation tools are assigned to respective methods of collecting information from given sources. See the table below to find out what tool corresponds to a given method.

Table 7: Overview of evaluation methods and tools

EVALUATION METHODS	EVALUATION TOOLS
Quantitative method – survey	Self-administered questionnaire (paper, online) Questionnaire navigated by a pollster
Qualitative methods: Interview (individual, group one) Observation Desk research	Interview scenarios Observation sheet or check list Instructions for the analysis of documents

Source: own elaboration

Most often evaluation tools are prepared from scratch according to the evaluation concept as questions included in tools must correspond to evaluation purpose, criteria, and questions. Some exemplary questions that you can use in order to develop your own tools were mentioned in section 8. 

10.Analysing Information

 

Figure 4: Evaluation of Youth Entrepreneurship Support Actions Life Cycle – Phase 3

This phase is about analysing the information you have gathered in order to make sense of it. Thanks to this phase you should learn the answers to your evaluation questions. This is a critical step toward identifying whether the evaluated project has made a difference and achieved the desired outcomes and impact, met recipient’s needs, was useful and effective. 

You will need to analyse information that is both quantitative and qualitative in nature. Quantitative information is essentially numbers, such as first of all data collected with using surveys, but also test scores, attendance and/or absenteeism figures, or the number of participants who meet minimum benchmark standards. Qualitative information, on the other hand, captures people’s views, observed behaviours, experiences and perspectives. 

By analysing a variety of information and using different methods you can get a reliable picture of your project and highlight any measurable improvements it has brought about, as well as capture how stakeholders feel they have benefited.

Analysing your data by looking for patterns or trends may tell you what elements if your project has been a success and which one you could change to improve your outcomes. For example, if you and your partners run a project to enhance participant engagement you could track their performance over time by analysing the changes in attendance or final outputs and outcomes. 

To fully understand the impact of your project, you may also want to analyse feedback from the project staff or professional advisors and mentors on student attitudes. If you are an entrepreneurship academy tasking with NGOs to provide mentoring support to participants to an entrepreneurship course, you opt to measure success both in terms of better participant engagement and staff attitudes about their employer and task.

10.1 How to analyse the collected information 

There are 4 steps in the data analysis that need to be conducted to be able to draw findings and interpret the results of evaluation. 

• Anonymization – make sure that any personal or sensitive data are removed from the data sets (e.g., names, contact details, dates of birth, etc.),
• Check the quality of information – completeness and consistency of the data, correcting obvious mistakes, removing information in case their accuracy cannot be verified, selection of data relevant for the evaluation,
• Coding and categorizing the data using a set of codes (e.g., symbols or names of the categories of data),
• Analysis and interpretation. 

 

What are the steps for analysing qualitative data?

Analysis of qualitative data includes examining, comparing, and contrasting, and interpreting patterns and consists of the following steps: identification of themes, coding, clustering similar data, and reducing data to meaningful and important points, such as in grounded theory-building or other approaches to qualitative analysis (NCVO a), n.d.). Here are the main steps of this analysis:

• Carefully read through and get to know your data.
• Identify themes or categories that are relevant for your analysis. Name each of the categories (the names represent the “codes“). The codes can be defined prior to the analysis or when you start working with the data. 
• Go through the transcripts of interviews, focus groups or open-ended responses in surveys and highlight key quotes. Provide the corresponding code to each highlighted quote. If there are more person’s coding, try to code and compare some parts at the beginning. This way you can ensure the data is coded in a consistent way.
• Create groups of quotes (cluster them) with the same code. 
• Identify specific patterns that have become obvious after clustering the data.  You can also realize; additional data needs to be collected or the validity of some data needs to be confirmed using other sources of information. For example, if there are contradictory quotes in the data you collected, you might decide to contact the facilitator of the training to obtain additional data. Also, you might need to do additional analysis. For example, some categories need to be divided into sub-categories. 
• Use relevant quotes to describe findings and interpret them.

 

What are the steps for analysing quantitative data?

Analysis of quantitative data is based on performing summary collected from questionnaires and tests. Before you can do so, you need to make sure your data is in a format you can use. This will depend on the software you plan to use (e.g., MS Excel or google spreadsheets). If you have, for example, collected information using paper questionnaires, you will need to enter these into a spreadsheet or database. Then remove any mistakes, e.g., blank responses, duplicates, any obvious errors and make sure that each of your variables is in the right number format, e.g., dates are formatted as dates, numbers as numbers, amounts of money as currency (NCVO b), n.d.).

Please see the table in the section 12.5.3 and 12.5.4 that provides descriptions of common techniques and calculations you could use. For these simple calculations you can use a spreadsheet program like MS Excel to organize and analyse your data. After completing the initial calculations, you may identify areas where more detailed analysis could provide you additional insight.

In the next step you should think of the form to present the data. Keep in mind that the presented data should be easy to present. In general, presenting your data in a table or a chart is recommended, especially in the case of data that is most important for people using your evaluation. 

Now you should be able to draw key findings. There are some things you should consider (Cottage Health, n.d.):

• Is 80% (for example) good or bad? How do you know? You may be able to decide on this by comparing your data to the previous year’s data, or to other similar interventions.
• Are there any other patterns, themes, or trends? For example, does one group consistently achieve more, or less, than other groups?
• Can you explain some of the less common responses? You may need some qualitative analysis to help you here.
• Is there anything in the data that has surprised you?
• Do you know anything about why some of the results are as they are? For example, can you link your percentages to qualitative data that explains why some people achieved an outcome while others did not?

 

10.2 What to keep in mind when analysing collected information

Tips for analysing information 

• You can convert your information from words into numbers to make analysis easier. This might be done by categorizing responses under a set of headings or themes that capture the relevant aspects of the project.
• This approach will allow you to summarise large amounts of information, such as frequencies (counts), per cents or rankings. It also lets you look at more than one characteristic (or ‘variable’) at a time if you want to know how they might be related. For example, you might be interested in looking at how attendance in a project or achieved outcomes differs for girls and for boys. 
• Depending on the quantity and kind of information collected you can analyse this yourself, seek help from a colleague or partner, an external expert/ organisation specialized in data analysis, or use a statistical software package to help. There are also other computer programs for analysis of quantitative and qualitative data – both of which can be expensive and require some advanced skill to operate.
• In some cases, you might be able to compare your results with national or regional results. However, if you do take this approach, remember that any comparison needs to take into account the context in which the data were collected.
• Patterns, trends, and themes can help you answer your evaluation questions, identify any unanticipated outcomes, and reveal possible gaps in your information. 
• Sometimes your analysis will involve comparing what people say or write at the beginning of an activity or project with what they say and write afterwards. (Such pre- and post-testing information collection can use quantitative or qualitative data.) 
• Your project might even lend itself to more regular collection so consider ways that data can be collected at multiple points during your project and then analysed to best identify your achievements. 
• Your analysis should be ‘fit for purpose’, that is, it need only be suitable to provide the evidence that your project has or has not achieved the desired outcomes. For a one-off project this might be fairly basic pre- and post- comparisons, showing if and how things have changed. For longer-term projects, or where further funding is being sought, you might need more depth to your analysis to identify if there are sufficient measures of success to justify continuing your project and project.

 

How to ensure Ethical Data Management

Now that you have collected all this data and are looking forward to using it, it is important that you follow the legal provisions that pertain to ethical data management. Below is an excerpt with how you could ethically manage the data that you have gotten from your evaluation activities.

Following Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data and repealing Directive 95/46/EC (General Data Protection Regulation), there have been implemented a full-fledged process of mapping the GDPR-based requirements, with the purpose of aligning the project`s interventions to the applicable legislative corpus. 

The ethics appraisal process is breveted to fully transpose the Article 19 of Regulation of Establishment (EU No 1291/2013), stating that all the research and innovation activities carried out shall comply with ethical principles and relevant national, Union, and international legislation, including the Charter of Fundamental Rights of the European Union and the European Convention on Human Rights and its Supplementary Protocols. 

Considering the large-scale data management pertaining to the evaluation related activities, the implementation phase is ensuring that the data collection process is aligned to the acquis communautaire, with due respect to the techniques for exploiting data falling under the GDPR incidence. For this purpose, you are strongly advised to appoint an in-house Data Protection Officer, in charge of safeguarding the fully compliant data processing.

 

10.3 Drawing conclusions and recommendations

 

What is the difference between findings, conclusions, and recommendations? Follow the steps described below to draw correct, evidence-based conclusions and recommendations (Kuhn, 2020). You can use the table used when analysing the data and simply add new columns for findings, conclusions, and recommendations:

Findings are validated data without any interpretation. Using expressions based on the data, e.g., “x number of females out of y number of participants attended the training” keeps your findings as clear and undiluted as possible. In the case of qualitative analysis, it can be hard not to jump directly to conclusions. However, anything else at this stage could be just an unfounded assumption or hasty conclusion that can harm the validity of the analysis. 

Once the findings are complete, the person analysing the data can move to the next step – drawing conclusions. Conclusions answer the question “so what?” that can arise when looking at the findings. Conclusions should be accompanied by interpretations or explanations, created by looking at the findings in the bigger picture. The more evidence-based explanations you can provide, the easier it will be to translate them into data-driven reports.

The next step in the analysis chain, answering the “now what?”. To build on what works or improve what does not in a project, what would be recommended to do? Recommendations need to be specific and actionable.

 

Example 1 – too broad recommendation: Invest more resources in staff

 

Example 2 – a good recommendation: Invest more resources in improving operational enabling environment factors, specifically on capacity-building for staff around project management and business management best practices. If capacity building efforts are paired with establishing and training on standard operating procedures for core communications and operations processes, this can greatly improve efficiencies.

11.Using Information

Figure 5: Evaluation of Youth Entrepreneurship Support Actions Life Cycle – Phase 4

 

This phase in the cycle is about sharing your evaluation results and making some informed decisions about your project and its future directions. 

11.1 Sharing your findings 

• Sharing may be for compliance purposes, but it can also be a valuable opportunity for others to have their say about the evaluation findings. 
• Sharing may help stakeholders engage with the project, influence the next steps of the project, or influence the partners to go back and have another look at their analysis. 
• Who you decide to share your findings with, and how you choose to do so, will depend on the expectations of each party in the project and on your reason for evaluating in the first place.
• If funding is involved and you need to satisfy a group or organisation, you may want to write up your evaluation findings providing answers to the questions of interest with that of this audience. 
• If you want to show the world or just local community, what a great project you have then the findings can be made public. For example, you could put a summary in your respective newsletters or annual reports or website.
• Visualisation of evaluation results can help deliver the message to the recipient. Well-designed infographics can illustrate the main features and relations in a simple and clear way. Of course, the type of visual elements should be selected according to the type of the recipients.
• Share data with trainers, mentors, and lecturers. They can thus improve their services and change them.

11.2 Making informed decisions 

You have conducted your evaluation, shared your findings and are now in a position to make informed decisions about whether to go on as you are, or to make changes to your project, or both. 

For example, an evaluation might show that your evaluated youth entrepreneurship support action      should continue to work collaboratively with the same partner but on a new challenge or opportunity. Or an evaluation might show that a similar activity could be carried out but with a different partner or set of mentors. 

This is the point where you and your partners need to talk openly about the future of the evaluated youth entrepreneurship project.      

Below are some examples of questions that could guide this stage in your evaluation:

• Will you continue to do things in the same way? 
• Do you need to evolve or enrich the project? 
• Could your model be replicated elsewhere? 
• Has the project run its course? If so, what is the best way to finalise the project? 
• How will you convey your decision to stakeholders? 
• Which part of the project was the most beneficial for recipients and which the least?

12. Toolkit

This Toolkit contains templates, tables, and checklists that you can use to help you at different stages in your evaluation (the list is not exhaustive).

• Evaluation and Development Cycle Checklist
• Tools to use in the Preparing phase of the evaluation
• Tools to use for Gathering Information
• Tools to use for Analysing Information
• Tools to use for Using Information
• Soft skills required in successfully conducting a youth entrepreneurship project

 

12.1 Difference between evaluation, monitoring and audit

Table 8: Monitoring, audit, and evaluation

	Monitoring	Evaluation	Audit
Definition	Ongoing analysis of project progress towards achieving planned results with the purpose of improving management decision making	Assessment of the efficiency, impact, relevance, utility and sustainability of the project’s actions.	Assessment of: – The legality and regularity of the project expenditure and income -Compliance with laws and regulation -Efficient, effective and economical use of project funds
Who?	Internal management responsibility	External firms (research/consulting), internal specialists or self-evaluation	Usually incorporate external agencies
When?	Ongoing	Usually at completion (ex-post) but also at the beginning (ex-ante), mid-term, and ongoing.	Ex-post, completion.
Why?	Check progress, take remedial action, and update plans.	Learn broad lessons applicable to other      projects and projects. Provides accountability and development.	Provides assurance and accountability to stakeholders.

Source: European Commission – Europe Aid Cooperation Office, 2004

12.2 Evaluation and Development Cycle Checklist

Table 9: Evaluation and Development Cycle Checklist

Addressed	Yes (ii) No (x) Or Unsure
PREPARATION What do you need to find out? What information will help you? Do you have clear and shared objectives for the evaluation? Is there a key evaluation question (or questions) to guide your evaluation?  Have you identified the information that you need to gather?  Do you have the skills and knowledge to gather it? Who will conduct the specific evaluation tasks? Is any external support needed?  Do you know your stakeholders? Do you know if, how and under what conditions they can be involved in the evaluation? Do you know their expectations of the evaluation?  Do you know what relevant information is already available? Do you know who can help you to find the information you need? Will each ‘partner’ have an opportunity to contribute to the evaluation?
GATHERING INFORMATION How will you gather your information? Is your focus on gathering relevant information rather than a lot of      information?  Do you know with what methods the information is to be gathered (quantitative, qualitative)? Have you established a process for gathering information (existing and/or additional)? Who will collect it and how long it will take?
ANALYSING INFORMATION How will you analyse the data?  What quantitative and qualitative data do you need to analyse?  What analytical methods will you use?  Do you have internal capacities to analyse the data?  What does the information you have gathered tell you? Have you identified the main themes, patterns and trends (over time)? Are you clear about the main outcomes from your project? Are there any additional (i.e. unanticipated) outcomes from the project? Have you identified ways in which your Youth Entrepreneurship Support Action      might be improved?
USING INFORMATION What can you learn from the evaluation results?  Does your project respond to the needs of its recipients/local communities? To what extent the objectives and expected outcomes were achieved?   What needs to be improved in our existing or future projects?  What internal processes need to be improved?  Have the communication channels and tools to share and promote what you have learned? Is there scope to expand or build on the project? Have you provided relevant feedback to your key stakeholders (answer to the questions of their interest in a written or verbal form)? Have participants in the relationship been invited to discuss the findings (where necessary)? Is the status of the project – complete, ongoing etc – understood? Have the stated objectives for the evaluation been achieved? Do you need to make changes to the project? Have you agreed how you will proceed next with the project?

Source: own elaboration 

 

12.3 Tools to use in the phase of preparing the evaluation

12.3.1 Logical framework matrix 

 This is a tool with which you can reconstruct the logic behind the project you want to evaluate. 

Figure 6: Logical framework matrix

Source: Department for International Development, 2007

The table below provides additional information on the elements of the Logical framework matrix.   

Table 10: Difference between inputs, activities, outputs, outcomes, and impact

Inputs/ resources	Inputs are those things that we use in the project to implement it, including human resource, finances, or equipment. Inputs ensure that it is possible to deliver the intended results of a project.
Activities	Activities are actions associated with delivering project goals. In other words, they are what the personnel/employees do in order to achieve the aims of the project.
Outputs	These are the first level of results associated with a project. Often confused with “activities”, outputs are the direct immediate term results associated with a project. In other words, they are usually what the project has achieved in the short term. An easy way to think about outputs is to quantify the project activities that have a direct link on the project goal. For example, project outputs could be: the number of community entrepreneurship training carried out, the number of meetings with successful entrepreneurs or the number of educational publications published
Outcomes	This is the second level of results associated with a project and refers to the medium-term consequences of the project. Outcomes usually relate to the project objectives.  For example, it could be the improved level of knowledge about business-related topics or establishment of a start-up.  Nevertheless, an important point to note is that outcomes should clearly link to project goals.
Impact	It is the third level of project results and is the long term or wider than project participants consequence of a project.  Most often it is very difficult to ascertain the exclusive impact of a project since several other interventions can lead to the same goal. An example would be increased number of new businesses in the region, reduced unemployment or NEET rate in given a city.

Source: Difference between inputs, activities, outputs, outcomes and impact, 2013

12.3.2 Candidate Outcome Indicators

Indicators can be developed for various stages of your project. The figure 7 provides examples of outcome indicators of an Employment Training/ Workforce Development Program (The Urban Institute, What Works, n.d.). 

Depending on the complexity and length of your programme, you can divide your programme into several stages and develop outcome indicators for individual stages of your programme. 

igure 7: Example of candidate outcome indicators

   

Source: The Urban Institute, What Works, n.d.

12.3.3 Identifying your evaluation stakeholders

In any Youth Entrepreneurship Support Action there can be multiple stakeholders. Knowing your stakeholders can help you identify what kind of questions they would like to answer with the findings of the evaluation, in what form and when they may need it. 

This stakeholder inventory is intended to help you identify who may be interested in your evaluation. Think about who needs to participate in the evaluation and who you will need to share the findings with so that the recommendations can be implemented.

Table 11: Stakeholder inventory

Potential stakeholders of the evaluation	Yes / No
Project recipients (who presently participate in the project )
Graduates (recipients who already finished participation in the project)
Trainers
Mentors
Innovation Hubs (representative)
Business Accelerators (representative)
Private Equity and Venture Capital Firms
Project staff
Coaches
Angel Investors
Researchers
Local Administrative Authorities (representative)
Project partners
Volunteers
Other institutions executing similar projects (representative)
Local employers
Who else? (Add those that specifically apply to you)

Source: own elaboration 

Having identified your stakeholders, you should also think about the added value they could bring to your evaluation and their motivation to get engaged in the process. Based on this information you can decide about the type of information, key messages, as well as communication channels to be used. 

Stakeholders can be categorized into 4 groups in terms of their influence, interest, and levels of participation in your project (Product plan, n.d.).

Figure 8: Power-Interest Grid

Source: Product plan, n.d.

1. High power, high interest: the most important stakeholders. You should prioritise keeping them happy with your project’s evaluation – keep them informed, invite them to important meetings, ask for their feedback.
2. High power, low interest: Because of their influence, you should work to keep these people satisfied. But because they haven’t shown a deep interest in your project, you could turn them off if you over-communicate with them. You should consider the frequency of contacting them and the amount of information provided.
3. Low power, high interest: You will want to keep these people informed and check in with them regularly.
4. Low power, low interest: Just keep these people informed periodically, but don’t overdo it.

12.4 Methods and tools to use in the Gathering Information phase of evaluation

Below you will find a number methods and tools that you could use in this phase of evaluation.

12.4.1 Identifying the appropriate methods to collect the needed information

Keeping in mind the purpose of your evaluation, identify the information collection methods and tools that will best suit this purpose. Summaries of each kind of the tool and method collection can be found in the section 9 “Gathering Information” above.

Table 12: Checklist to identify which methods can be used to collect needed information 

Method of data collection	Which Information is most relevant to your purpose?	Which Information will be easiest to collect?	Which Information will you need some help with, to gather?	What kind of information will you need? (eg. locating documents, identifying participants, analysing information, dissemination activities)	Who can give you this help? (eg. a colleague, independent experts, a partner’s colleagues, project participants, an external evaluator)
Desk research (document analysis/ review, such as reports, checklist,  diaries, journals and logs, testimonials, tests and other assessments)
Observation
Individual interviews
Focus Groups
Case Studies
Skill assessments (e.g., tests, self-assessments)
Survey

Source: own elaboration

12.4.2 Personal potentials and objectives – self assessment tool 

This tool is important in assessing the participant’s potential and objectives to find the right fit for them in the project and also be able to help them harness their potential. It is a guided enquiry where some of the participants (especially victims of forced migrations like refugees, immigrants, or people with disabilities) may not be able to understand all questions and may require guidance from the instructors to provide the required information.

These and personality assessments were inspired by tools available at the platforms mfa-jobnet.de and Psychometrics (visit www.mfa-jobnet.de and https://www.psychometrics.com) and with necessary modifications to fit this evaluation module, however the original source entails more tests that can help you assess the employability of a candidate. 

 

Table 13: Self-assessment tool to assess personal potentials and objectives

Education/Skills
In which areas have I been active in my life so far? What have I learned there?
What have I learned there?
What skills have I gained from this? What skills have I developed in my everyday activities? What general competencies have I gained through my professional and extra-professional experiences?
What professional or other training do I have?
What diplomas, certificates etc. do I have?
Which of my degrees are recognized in the country?
Which are not?
Where have I worked before?  a) In the home country b) In another EU state      c) In other countries What other work experience do I have?
What in the training/work has particularly appealed to me, what have I been interested in, what have I been enthusiastic about?
What other qualifications do I have (e.g. language courses, computer training, driver’s license etc.)?
What skills and competences have I acquired in different places and countries? (Not only professionally, also in everyday life)
Where do I use them today?
What do I want to learn?
Which of my skills can I put to good use in the country I live in?
Are there areas in which I have completely different skills than current     colleagues and friends?
Is there a profession would I like to take up now?
If no: What do I want to do for living      instead?
What expectations and ideas did I have when I came to the country?
What is different than I expected?
Is that  a) a disappointment? b) a challenge for me?
How can the contacts I have be useful to me in implementing my ideas and plans?
Self-assessment
What have I successfully achieved in my life?
What have I always been particularly good at?
What has always been fun for me?
Which of my competences/skills relevant for my future professional life do I want to develop further? What do I want to become even better at?
What do I not like doing at all?
Which of my skills were particularly helpful/important/successful during my stay in this country?
External assessment
What abilities/strengths do others see in me?
What have I been able to inspire / impress others with?
For what have I received a lot of praise / recognition?
Work
Which 5 characteristics should a job have so that it fits me?
How do I concretely imagine my future work?  Where? With whom? What exactly is my work?
What is my goal?
What can I do to achieve my goal?
What do I need help/support with? What exactly do I need?
Which difficulties/obstacles can appear on my way?

Age:

Gender: 

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

 

12.4.3 Self-assessment sheet:  Social skills and personal values

This questionnaire is suitable for pre and post measurement of a participants’ social competences and attitudes to assess their progress in the project. It should however be noted that self-assessment questionnaires are not the only tools for measurement of progress as they may be subjective and sometimes the change measured in this kind of tools may show a decrease only because the trainings made the beneficiary aware of his/her weaknesses in particular area. However, self-assessment can be complemented by other forms of external assessment done by project staff (for instance trainers, mentors, psychologists). 

Table 14: Self-assessment sheet to assess social skills and personal values

Name, first name:

Date:

	++	+	0	–	—
Social skills
Communication skills
I pay close attention to what and how others say something.
Others tell me that I understand them well.
Even in larger groups I can express my opinion in a way that is understandable for everyone.
Team spirit
It is important to me that a team works well: that is why I share important experiences and knowledge with my team colleagues.
But I also like it when I can learn from others.
If it is important for the group, I can put my personal interests last.
I actively participate in group work, e.g. by considering how best to divide the work.
Ability to deal with conflicts
Before I get too upset about something, I prefer to talk about things in a quiet minute and in a calm mood.
I have no difficulty with it when other people have a different opinion than I do.
In conversations I can easily tell whether it is a factual problem or whether two people personally do not get along well with each other.
When conflicts arise, I mediate or work towards a solution that all parties can live with (I don’t just want to win myself).
Ability to take criticism
If someone criticizes my performance or even individual behaviour, I think about whether they could be right.
If I have something to criticize about someone else, I explain it very specifically in a friendly tone.
I understand that other people sometimes make mistakes.
Dealing with people
I like to approach other people.
I usually remain calm and objective even when other people get on my nerves.
If I notice that someone else is getting upset, I can calm them down.
Personal values
Reliability
I always arrive punctually for appointments.
If I cannot keep an appointment, I apologize in time.
I always deliver work orders on time.
I don’t need to be constantly monitored: if I have a task, I think about fulfilling it myself.
Sense of responsibility
Of course, I take responsibility for what I do.
I already take care of my health.
I am very careful not to put anyone in danger.
I handle the equipment or materials entrusted to me with great care.

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

 

12.4.4 Questionnaire for self – assessment of own competences

This questionnaire is suitable for pre and post measurement of a participants’ progress in the project. It should however be noted that self-assessment questionnaires are not the only tools for measurement of progress as they may be subjective and sometimes the change measured in this may show a decrease only because the trainings made the beneficiary aware of his/her weaknesses in particular area.

+++ is particularly true; ++ is true; + is less true.

Table 15:Self-assessment sheet to assess own competences

1.Self-Competence	+++	++	+
Independence
I make my own decisions
I form my own opinion and represent it
I take responsibility for my actions
I plan and carry out a work without external help
I call for      outside help – if necessary
I can assert myself
Flexibility
I can adapt to new situations
I can do different tasks side by side
I am open for new or unusual ideas
I can easily switch from one task to another
Creativity
I find solutions for problems
I can help myself
I try out new possibilities
I have imaginative ideas
I can achieve a lot with little means
2. Social Competence
Communication skills
I express myself clearly in spoken and written form
I ask if I do not understand something
I can listen
I do not judge and interpret hastily
Ability to handle conflicts
I can say no
I accept other assessments
I can offer constructive criticism and react appropriately to criticism
I recognize tension and can talk about it
I can deal with my strengths and weaknesses
Ability to work in a team
I accept decisions made
I work out a solution together with others
I can also stand back in a group
I share responsibility for work results
I can take responsibility in the group
3. methodological competence
Learning and working technique
I know where and how I can obtain information
I can concentrate well
I have perseverance
I divide my strengths correctly
Work organisation
I create a work plan and control it
I recognize connections in my work
I foresee consequences and can estimate them
I recognize the essence of a thing
4. professional competence
Expertise
I know technical terms
I know the rules and norms of my work
I have technical / linguistic knowledge
I have a good general education
Practical knowledge
I implement specialist knowledge
I carry out work properly
I bring in what I have learned
Total

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

12.4.5 Tests used to measure skills development

There are a number of tests that have been already developed and can be utilized to help in the evaluation of entrepreneurial skills of Youth Entrepreneurship Support Actions beneficiaries (start-up founders). Measuring the skills of your participants helps you in planning and optimising your project. 

These psychometric tests were inspired by tools available at the platforms mfa-jobnet.de and Psychometrics (visit www.mfa-jobnet.de and https://www.psychometrics.com) and with necessary modifications to fit this evaluation module, however the original source entails more tests that can help you assess the employability of a candidate. 

Below we provide some tests you can use:

• Inspire test,
• Awareness test,     
• Skills test.     
• Networking test.

Inspire Test (IT)

Every entrepreneurship project has different requirements for its successful execution, and the IT allows a beneficiary to specify the importance of the personality match for the project that is being rolled out by the organization and the ideal scores that they should receive. These ratings are then used to examine how well a beneficiary’s project fits with the requirements of the whole project activity.      

IT is performed by way of either personal interviews or practical exercises in teams.

Benefits of the IT

The IT provides a comprehensive measure of a participant’s personality, showing how they will:

• Complete their project,
• Interact with people,
• Solve problems,
• Manage change and,
• Deal with stress.

Sample Interview questions

ENERGY AND DRIVE 

Energy

• Tell me about a project you previously worked on that required a lot of energy and commitment. What were your responsibilities? What was the end result? 
• Name some of the most demanding things you have done. How did you manage them? 
• Everyone runs out of energy at some point, tell me about a time when you had to work on a task that was too demanding. What did you do? What happened in the end? 

Ambition

• Tell me about a time when you needed to compete hard to be successful. 
• Tell me about some difficult goals you set for yourself, and how you reached them. 
• Describe a situation in which you adopted a non-competitive attitude in order to be successful. 
• Have you ever had a project with little room for advancement? What was that like for you? 

Leadership

• What experience have you had with leading people? What was that like for you? What was positive about the experience? What would you do differently? How could you have been a more effective leader? 
• Tell me about a time when you needed to convince people to follow you. What did you do? Were you able to get people onboard? 
• Tell me about an occasion when you encountered difficulties with your team. What were the difficulties and how did you overcome them? 
• Name a time when you took on a leadership role without being asked. 
• Give me an example of a difficult leadership role you took on. 
• Tell me about a time when you had to follow someone else’s lead. 

IT process:

Tests will create benchmarks for the project. A beneficiary’s performance will be assessed, and a beneficiary report will be made.

• Create Benchmarks – observers will work with beneficiaries to create benchmarks for the project activity. One of the most effective ways to identify these requirements is to gather information from people who know the project well. These individuals who are familiar with the project and can speak about the knowledge, skills, and characteristics necessary for someone to emerge successful in the project.
• Assess participants – use an assessment tool to administer participant assessments.
• Assess Person – after participants have completed the Inspire module test the observer can generate a report which will indicate how well the participant’s personality traits match with the requirements of the project activity.

The IT report does provide among others; Overall project activity fit score – which is quickly sorted based on the candidates overall fit with project benchmarks, Participant profile that helps identify participants’ specific areas of fit or misfit,     

In-depth narrative description of the participant’s shared working space behaviours, helping to target areas of uncertainty in follow-up interviews and

Profile validity – assesses the extent to which the questionnaire was completed honestly rather than in an overly positive or unusual way.

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

Awareness test (AT)

Youth Entrepreneurship Support Actions have the goal of developing entrepreneurial skills. Therefore, the objective of this test is to provide overview of the different competences and soft skills that are to be developed with the measures to be evaluated and the skills to be tested.

1. Self-management – Readiness to accept responsibility, flexibility, resilience, self-starting, appropriate assertiveness, time management, readiness to improve their team’s      performance based on feedback and reflective learning.
2. Team working – Respecting others, cooperating, negotiating, persuading, contributing to discussions, awareness of interdependence with others.
3. Cultural tolerance / sensitivity – being able, respecting people of other cultures, not discriminating against people of other cultures.
4. Business and customer awareness – Basic understanding of the key drivers for business success and the importance of providing customer satisfaction and building customer loyalty.
5. Critical thinking – collecting and analysing information objectively, making a reasoned judgment.
6. Problem solving – Analysing facts and circumstances to determine the cause of a problem and identifying and selecting appropriate solutions.
7. Communication – The ability to effectively tailor messages for the purpose and audience and use the best tools available to communicate them.
8. Time management – Ability to plan and organise one´s time among different activities.
9. Flexibility – Ability to work within permanently changing environment and desire to look for win-win solutions.
10. Resilience – Ability to work under pressure. 
11. Efficiency – Applying the 80/20 rule and other techniques for yielding higher results in less time. Switching between different chores and progressing effectively day-to-day.
12. Networking – Growing a network facilitates business opportunities, partnership deals, finding subcontractors or future employees. It expands the horizons of PR and conveying the right message on all fronts.
13. Branding – Building a consistent personal and business brand tailored to the right audience. 
14. Sales – Being comfortable doing outreach and creating new business opportunities. Finding the right sales channels that convert better and investing heavily in developing them. Building sales funnels and predictable revenue opportunities for growth.

Helping a beneficiary in understanding their personality type is the first step to personal and entrepreneurial growth. The AT helps beneficiaries to understand their strengths, their preferred working styles, and ultimately helps them see their potential. Used individually to provide self-awareness and clarity of purpose, the AT also helps to create a better understanding and appreciation between their teams – enabling them to work better together.

AT can be performed in either personal interviews or practical exercises in their teams.

Benefits of the AT:

• Greater understanding of oneself and others.
• Improved communication skills.
• Ability to understand and reduce conflict.
• Knowledge of your personal and work style and its strengths and development areas.

 

Interview questions

SOCIAL CONFIDENCE

If the project requires an individual to be self-assured and at ease with people in all types of social situations, consider some of the following questions: 

• Give me an example of a time when you had to interact with a group of strangers for a work-related function. What did you do? What was the result? 
• Tell me about a time when you had to exercise social confidence to accomplish a goal. 

 

 

PERSUASION
If the project requires someone who is comfortable with negotiating, selling, influencing and attempting to persuade people or trying to change the point of view of others, consider some of the following questions: 

• Give me an example of a time when you persuaded someone to purchase something? What did you to do convince them? 
• Tell me about a time where you changed someone’s mind. What did you do/say that made them see things your way? 
• Name a time when you used negotiating to get what you wanted. What was the result? 

 

INITIATIVE

If the project requires someone with a high level of initiative to identify new opportunities and take on challenges, consider some of the following questions: 

• Give me an example of a time when you completed a project without any support from others. What was the outcome? 
• Tell me about an opportunity you identified that others missed. 
• Describe some new challenges that you took on without encouragement from others. 
• Name some new responsibilities you took on voluntarily. 
• When you identify a potential opportunity what do you need before you will begin working towards it? 
• What do you prefer, stable project responsibilities or frequently changing responsibilities, and why?
• Tell me about some occasions where you have shown initiative. 
• Give me some examples of when you have shown initiative. 

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

 

Soft-skills test (SST)

This test demonstrates the difficulty in developing the competences listed in awareness test and consequently propose specific knowledgebase (lessons) that can do this.

The SST is used to help participants and their team members get acquainted with each other’s conflict styles, identify potential challenges, and set goals for how they should handle conflict as a group. With established teams, the SST helps team members make sense of the different conflict behaviours that have been occurring within the team, identify the team’s challenges in managing conflict, and find constructive ways to handle those challenges.

SST may be performed either as personal interviews or practical exercises in teams.

The SST describes five different conflict modes and places them on two dimensions:

• Assertiveness – the degree to which a person tries to satisfy their own needs
• Cooperativeness – the degree to which a person tries to satisfy other people’s needs.

Benefits of the SST

Why do organisations use the SST?

• It is easy to complete, the short questionnaire takes only a few minutes and can be administered online before your training or in paper booklet onsite.
• Delivers a pragmatic, situational approach to conflict resolution, change management, leadership development, communication, participant retention, and more.
• Enables your organisation to open productive dialogue about conflict.
• Can be used as a stand-alone tool by individuals, in a group learning process, as part of a structured training project workshop.

 

Interview questions

DEPENDABILITY

If you are looking for participants (beneficiaries) with a high level of dependability, consider some of the following questions: 

• Tell me about a project you couldn’t finish on time. What happened? What would you do differently? 
• Are you comfortable leaving a project unfinished if something else comes up? 
• Give me an example of a task that you needed to work beyond your normal hours to complete. What was that experience like? 
• Can you describe a time when you had to shift priorities and leave a task you were working on unfinished? What happened? How did you complete the first task? 
• Name a time where it was difficult for you to complete your task. What happened, and how did you resolve the difficulties? 
• Describe a time when you needed to work extra hard to get your tasks done on schedule. 

 

PERSISTENCE
If the project requires someone with a high level of persistence, consider some of the following questions: 

• Tell me about a difficult task that you recently completed. What made it difficult? How did you manage to work through the difficulties/obstacles? 
• Describe a time when you had a large number of boring/dull/uninteresting tasks to complete. How did you motivate yourself? 
• Give me an example of a project that you gave up on because it was no longer worth the resources to complete. 
• Tell me about a time when you showed a high level of persistence.
• Tell me about some obstacles you have overcome that took a lot of persistence. 
• Give me an example of something you gave up on because you did not think it worth the effort. 

If the project mostly involves tasks that can be completed quickly and has few obstacles to overcome, consider some of the following questions: 

• Give me an example of a project that you gave up on because it was no longer worth the resources to complete. 
• Describe a time when you had a lot of boring work to complete. How did you motivate yourself? 
• Have you ever worked on a project that only required you to do easy work that had no challenges? What was that like for you? Was it pleasant or unpleasant? 

 

ATTENTION TO DETAIL

If the      project involves tasks that require a lot of detailed information or research, consider some of the following questions: 

• Describe a project you worked on that involved a lot of detailed work. 
• What is the most detailed work you have had to complete? 
• What is worse for you, completing a project late, or completing it on time with imperfections? 
• What kind of tasks have you had to do in the past that required you to pay close attention to details? 

If the project does not involve examining a lot of detailed work, but requires someone who focuses on global problems, best practices or issues, consider some of the following questions: 

• Tell me about a time when you ignored the details and focused on the big picture. 
• Tell me about a time when people on your team focused too much on details and missed the big picture. What did you do to help them broaden their focus? Is there such a thing as spending too much time looking at the minor details? 
• What experience do you have with determining strategy/looking at the big picture/setting large goals and priorities? 

 

RULE-FOLLOWING
If the project has a lot of operational procedures and rules that need to be strictly followed, consider some of the following questions: 

• Describe your past experiences of working in a very structured, rule bound environment. 
• Describe your experiences of working in an environment with no structures or rules on how to work. 
• Can you tell me about an occasion where you needed to ignore rules or procedures to get your work done successfully? 
• How do you determine when to ignore procedures/rules? Are there situations where you believe they should be followed all the time? How do you determine when that is? 

If the project has few to no procedures and rules, and requires the individual to determine the best way to complete their tasks, consider some of the following questions: 

• Tell me about some ineffective rules that were still followed in your previous innovation centre or business incubator. 
• Can you tell me about an occasion where you needed to ignore rules or procedures to get your tasks done successfully? 
• How do you determine when procedures/rules can be ignored? Are there situations where you believe they should be followed all the time? How do you determine when that is? 
• How often do you encounter rules/procedures that you think should no longer be in effect? 
• How comfortable are you working on tasks when you have not been given any direction/instruction? Do you enjoy the freedom? Do you wish you could get feedback to ensure you are doing your tasks correctly? 

 

PLANNING
If the project involves a lot of short and long-term planning, consider some of the following questions: 

• Tell me about a task you completed that required a significant amount of planning. 
• Give me an example of a long-term goal or plan that you established. Did you meet your goals? How effective was your plan? 

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

Networking test

The main objective here is to improve Youth Entrepreneurship Support Action’s networking activities and knowledge exchange, enabling the setting up of successful innovation ecosystems across Europe.

Benefits of the NT?

• It links your interests and preferences to various task activities, project settings and careers. 
• The NT gives you results that you can benefit from if you are just starting out in your business.

NT is done either by way of personal interviews or practical exercises in teams.

Interview questions

PROBLEM SOLVING STYLE 

INNOVATION

If the project requires someone who is creative and innovative, consider some of the following questions: 

• Tell me about a problem you solved in an innovative way. 
• Name an original/creative/new solution you came up with to solve a problem. 
• When addressing a problem do you first look at what was tasked in the past, or do you come up with an entirely new solution? 
• What are the benefits/disadvantages of using past solutions? 
• What have been some of your most creative ideas at task? 
• When you don’t understand something, do you ask until you understand?
• Do you often question what is considered normal?
• Do you like to watch people? Do you often question what is considered normal? 
• Do you like to watch what is happening in the world as part of innovation? 
• Do you observe how people behave in situations that may affect you start-up (e.g., where, and how to eat, drink, dress, shop, etc.). Do you observe the world around you? Do ideas for new products and services come to mind?
• Are you adventurous, happy and looking for new experiences?

If the project does not require much problem solving, or the problems addressed only require incremental changes or practical solutions, consider some of the following questions: 

• When addressing a problem do you first look at what has tasked in the past, or do you come up with an entirely new solution? 
• What value do you see in sticking with the established ways of doing a task? 
• What are the benefits/disadvantages of using past solutions? 
• What have been some of your most practical solutions to task problems? 
• Would you describe yourself as innovative or practical? 

 

ANALYTICAL THINKING

If the project requires analysing a large amount of information and a decision-making approach that is logical, cautious, and deliberate, consider some of the following questions: 

• How much information do you need to feel comfortable making a decision? How do you get that information? 
• Tell me about a decision you have made through extensive information gathering and discussion with others. How did it work out? Did you need to be as cautious as you were, or could you have made the decision more quickly? 
• Would your friends describe you as analytical and calculating or intuitive and spontaneous? Why? 
• What process do you go through before you make a decision? 
• Tell me about an important decision you needed to make quickly.

If the project requires quick decision making that does not allow for the extensive gathering of information, consider some of the following questions: 

• How much information do you need to feel comfortable making a decision? How do you get that information? 
• Tell me about a decision you have made based on your gut feelings. How did it task out? How comfortable are you making decisions that way? 
• Would your friends describe you as analytical and calculating or intuitive and spontaneous? Why? 
• When have you had to rely upon your intuition to make a decision? 

 

DEALING WITH PRESSURE AND STRESS 

SELF-CONTROL

If the project requires the individual (beneficiary) to have a high level of self-control, consider some of the following questions: 

• What do you do when you get frustrated with others? Tell me about a time when you were frustrated with a team member. 
• Give me an example of when you maintained your composure in a difficult situation. 
• Give me an example of a time when you were angry with someone at task. What did you do? 
• Describe a previous task experience that had you frequently dealing with upset people. 
• What experience have you had dealing with irate customers? 
• Tell me about a time when you had to deal with an upset customer. What did you do? What were the results? How did you feel afterwards? 

 

STRESS TOLERANCE

If the project requires regular results in a high level of stress, consider some of the following questions: 

• What do you do to alleviate stress? 
• How do you tolerate stress? 
• Name a time when you had to do a task under extreme stress. 
• What types of activities do you find stressful? 
• Name a time when you had difficulty coping with stressful tasks. What did you do to get yourself through that time? 
• What type of stress do you find very hard to deal with? 
• Are there stressful activities that you cannot cope with? What are they? 
• What do you find to be the most stressful? 
• What kinds of extreme stress have you had to execute tasks under? 
• What have been some of the most stressful things you have been involved in? 

Source: adapted from Stellenbörse für MFA und Arbeitgeber (n.d.) and Psychometrics Canada Ltd. (n.d.).

 

12.4.6 Business Model Canvas

The Business Model Canvas is a good tool for assessing a start-up at the beginning and end of the project – to measure business development.     

Figure 9: Business model canvas

Source: Strategyzer, n.d.

 

Summary of steps on how you could use the business plan format in assessment of Entrepreneurship Project Actions:

Development of a business/ project idea from the point of view of a start-up

• Which idea/ plan is working best for me?
• What makes it different from others?

Requirements of the founder

• What are my qualifications, strengths, and weaknesses?
• Which partners do I have, or do I need?

Overview over the market

• What is my target group and what do these people need?
• What do my competitors offer?

 

Good strategies for marketing and distribution

• How could good advertisement look like?
• What is my distribution area and roads?
• Do I have a good price strategy?

Set up of the organisation /company

• How many employees do I need and what should their skills/ qualifications be?
• How does the legal form of my company look like?

Analysis and weighing up of the possible chances and risks

Financing

• How much are my potential customers willing to pay?
• How high are my capital requirements?
• Do I have a financing- and investment plan?
• Are there alternative funding opportunities? Grants, donations.
• How high are my costs of living?

Acquisition of further documents

• Do I need other documents, for example expert opinions, contracts or curriculum vitaes?
• Legal requirements?

 

Want to try it on your own? Fill the Business Model Canvas here below.

Figure 10: Template of a Business model canvas

Source: Strategyzer, n.d.

 

12.5 Tools to use in Analysing Information

Below are some tools you could use in analysing the information that you have gathered above.

 

12.5.1 Assessing the outcomes (‘the what’) of your project

The questions below are provided simply as guidance in assessing the outcomes of your project. 

You may have additional or different questions you want to ask depending on the nature of your project and your reason for evaluating. 

• In what ways have participants benefited from being in your project? (e.g., re-engaged with learning, gained accreditation, increased or improved industry specific networks, identified a realistic vocational pathway, gained entrepreneurship skills, gained identity documents etc.) 
• How do you know this? (That is, what information or evidence do you have to show that the participants have benefited?) 
• What other benefits have you delivered (and to whom)? 
• Have there been any surprising or unanticipated outcomes?
• How do you know if it is your project that is making the difference? (i.e. there might be other things that are going on in the organisation at the same time as your project which might also impact on participants’ outcomes or things you are trying to measure).

12.5.2 Assessing the effectiveness (‘the how’) of your project

The checklist below is provided as a simple way of assessing the effectiveness of the project (‘the how’) and to identify potential areas for improvement. 

You may have additional or different questions you want to ask depending on the nature of your project and your reason for evaluating. 

Table 16: Checklist for assessment of the effectiveness of a project

Key Question	Y/N	Examples of Follow Up (To address identified issues)
Are all stakeholders wholly engaged in the      project?	No	Arrange a meeting of all stakeholders to review current levels of involvement and see if this needs to change and in what ways. For example, are more resources needed? Would formalizing the project in a memorandum of understanding make a difference? If there is a genuine lack of engagement from one partner and this is unlikely to change, should the support of another organisation be sought?
Is there a shared vision and common goals?		Create a kind of glossary in joint work – in which the goals are defined, and a common interpretation is fixed.
Does each stakeholder have a clearly defined role and responsibilities?		Create a kind of organigram in which each party is represented with its roles & responsibilities – this should be accessible to all.
Are the expectations of each stakeholder fair and reasonable?
Does each stakeholder have a good understanding of requirements of other stakeholders?
Is there regular communication between stakeholders?
Have all parties received fair recognition for their efforts?
On balance, have the benefits of the collaboration justified the time and effort that have gone into it?

It can help to break down an overarching or key evaluation question into a small number of sub questions to guide the information gathering and analysis. The table below shows how one project Entrepreneurship training for unemployed graduates went.

 

Table 17: Checklist to assess an Entrepreneurship training for unemployed graduates

Name of Project: Entrepreneurship training for unemployed graduates.

Objective	Focus of the evaluation	Evaluation question	Questions included in evaluation tools (asked to interviewees/ respondents)
To help participants turn their ideas into operational and profitable businesses.	Focus on the impact of the activities or project (the ‘what’).	How, and to what extent, has the project helped graduates to turn their ideas into operational and profitable businesses?	» What knowledge and skills have been gained by participants through this project? » How have these skills helped participants in their quest to turn ideas into fully fledged businesses? » Are there other things that could be done to support participants in their quest? » What additional information is needed to make a decision about the future of the      project.
To identify how well the stakeholders are working together and if improvements could be made.	Focus on the effectiveness of the      projects (the ‘how’).	How effective are our current      projects?	» What are we doing well? » What are the things that are not working so well? » How can we improve these      projects?

 

12.5.3 Quantitative analysis techniques

Figure 11: Overview if quantitative analysis techniques

Source: Cottage Health, n.d.

 

12.5.4 How to present quantitative data

Table 18: Overview of data presentation formats

What do you want to present?	Which chart to use
Comparison of two or more categories	Bar chart, circle chart
Binary date (e.g. yes/no responses)	Pie chart, bar chart
Change over time	Line graph, area graph
Correlation between two variables	Scatter graph
Frequency	Bar chart
Percentages	Bar chart, pie chart
Dispersion (how spread out your date set is)	Box and whisker plot, bar chart

Source: The National Council for Voluntary Organisations, n.d.

 

12.5.5 Tools to use when using information

Having a clear structure makes your report easier to read. Before you write, plan your headings and subheadings such that your report is easily comprehensible and legible. You can find a sample structure of an evaluation report here below:

TABLE OF CONTENTS

ABBREVIATIONS

ACKNOWLEDGEMENTS

EXECUTIVE SUMMARY

INTRODUCTION

1. 1. Project background, source of financing, partners/ stakeholders, target group and participants, process of implementation, objectives, activities/tasks and results 

1.2 Evaluation concept – type, scope, purpose, criteria and questions

METHODOLOGY

2.1 Methods of collecting information, and samples 

2.2  Reliability and validity issues

RESULTS

3 They should be discussed according to the evaluation criteria and questions 

LESSONS LEARNT, CONCLUSIONS AND RECOMMENDATIONS

4.1 Lessons Learnt

4.2 Conclusions and recommendations

ANNEXES

Annex 1: Map of the intervention area

Annex 2: Research tools (e.g. questionnaires and scenarios used)

13.Conclusion

 

It is our submission therefore that once the entire cycle is traversed with specific attention to detail, the youth entrepreneurship support actions implementers will have an enhanced knowhow in the evaluation of their actions.

14. References

 

The following resources will provide you with further information or support that you may need.

• Bartosiewicz-Niziołek, M., Nałęcz, S. (2021) Towards better youth employment actions. Blended learning course on evaluation. Youth Impact project. Jerzy Regulski Foundation in Support of Local Democracy. https://moodle.youth-impact.eu/ 
• Bartosiewicz-Niziołek, M., Nałęcz, S., Penza-Gabler, Z., Pintera, E. (2021) Your leverage to better youth employment projects. Evaluation Toolkit. Youth Impact Project. https://www.youth-impact.eu/toolkit/ 
• Bednárová, E. (2021, May 6). Hodnotenie dopadu mládežníckych projektov (praktické príklady vybraných slovenských organizácií). Youth Impact. https://www.youth-impact.eu/wp-content/uploads/2021/05/Hodnotenie_dopadu_mladeznicke_projekty.pdf
• Bhatia, M. (2018, July 16) What is monitoring and evaluation? A guide to the basics. Humans of data https://humansofdata.atlan.com/2018/07/what-is-monitoring-and-evaluation/
• Bureau of Educational and Cultural Affairs (n.d.) A good start with S.M.A.R.T. (indicators). https://eca.state.gov/files/bureau/a_good_start_with_smart.pdf 
• Clinical and Translational Science Awards (CTSA) Consortium’s Community Engagement Key Function Committee (2011). Principles of community engagement – second edition. National Institute of Health, Publication No. 11-7782, pp. 175-176. https://www.atsdr.cdc.gov/communityengagement/pce_program_methods.html    
• Cottage Health (n.d.). Cottage Health Evaluation Toolkit: How to analyze and interpret data. Cottage Centre for Population Health https://www.cottagehealth.org/app/files/public/2246/Analyze_and_Interpret_Data_Cottage_Health_Evaluation_Toolkit.pdf 
• Department for International Development (2007). Humanitarian Funding Guidelines for NGOs, p 20. ISBN: 1 86192 915 3. https://www.ennonline.net/attachments/126/dfid-guidelines.pdf 
• Difference between inputs, activities, outputs, outcomes