PHASE TWO
Action
STEP 04
Collecting Evidence
In Step 04: Collecting Evidence, you will gather both quantitative and qualitative data to answer your process and outcome evaluation questions. This involves collecting stories and data to understand how your program has been implemented (process evaluation) and to assess the changes that youth participants are experiencing (outcome evaluation).
Both types of evaluation are important because documenting implementation is directly relevant to understanding outcomes. Without knowing exactly what program components have been implemented or to what degree they are implemented, you cannot make use of outcome evaluation results to understand what worked or didn’t work and why.
What are some engaging ways to collect data and other stories?
KEY TAKEAWAY
It's show and tell time! You will have evidence (data + stories) about your program, how youth experience it, and any changes that have resulted for youth.
01.
Choose reliable and valid measures to assess outcomes. Remember, an outcome is an element that you assess related to change in your program, so an outcome indicator examines how you will assess this change. These indicators can be both quantitative or qualitative, and provide simple and reliable means to measure or reflect changes connected to your program.
02.
Address ethical considerations. Everyone being engaged through data collection must give informed consent to their participation; they should be aware of what the evaluation is for, whether their responses will be kept anonymous and/or confidential, and what will happen if they choose to participate or choose not to participate. Participation in any evaluation is always voluntary; participants should know that they can withdraw their engagement at any time without being penalized.
03.
Collect relevant data. Methods could include surveys, interviews, focus groups, observation and field notes, and/or documents and records. Consider the strengths and weaknesses of each method. Your choices should be directly related to the question(s) you’re trying to answer and your selected indicators. Consider the following:
- Is the method reliable and valid?
- Is it feasible?
- Is it relevant?
- Is it appropriate (i.e. youth-friendly)?
Using a combination of methods can strengthen your evaluation! Visit YouthREX’s Youth Measures Inventory to explore outcome, process, and qualitative measures that are youth-friendly and open-access.
What is process evaluation?
Also known as implementation evaluation, a process evaluation is designed to determine whether a program is being delivered as intended. It documents what happens (if activities are conducted as planned and according to schedule), the frequency and intensity of the activities, and the extent to which the participants were reached.
Implementation evaluation requires close monitoring of implementation activities and processes. This type of information can be used to adjust activities throughout a program’s lifecycle.
What is an outcome evaluation?
An evaluation that assesses the extent to which a youth program affects:
a) its participants (i.e., the degree to which youth experience changes in their knowledge, skills, attitudes or behaviors); and
b) the environments of the program, community, or both.
Outcomes are therefore the post-participation effects of a youth program. An outcome evaluation assesses the extent to which such effects exist and whether they can be ascribed to the program. The outcome evaluation answers the question of whether the program produced valuable changes.
Several important design issues must be considered, including how to best determine the results and how to best contrast what happens as a result of the intervention with what happens without the program.
What are the benefits of an outcome evaluation? Why should we do one?
Outcome evaluations provide all stakeholders with concrete information about the extent to which the program has made a difference for youth participants.
What if we cannot do a random control trial? Does that rule out a “rigorous evaluation”?
Even if you can’t do random assignment, you can still do a rigorous evaluation. Quasi-experimental designs are also considered to be rigorous.
YouthREX’s approach to rigor is use multiple lines of evidence in assessing the extent to which a youth program affects its participants and the environments of the program, community, or both.