Leveraging Literature Review to Assess Program Impact
Written by Jessica Gibbons
As a nonprofit service organization, your mission likely prioritizes hands-on stakeholder engagement and resource dissemination. Understandably, this leaves little time for conducting in-depth, long-term-impact analyses.
So: How can you easily and effectively evaluate your organization’s impact without adding significantly to your workload? A simple literature review might be your answer.
What is a literature review?
A literature review (lit review) is a systematic approach to reviewing relevant publications from credible sources and synthesizing findings in a way that is useful to your work. When performing a lit review, you’ll want to seek out research (defined for our purposes as work that tests a theory of change that drives a social service program) that:
measured long-term outcomes, and
determined if specific activities actually cause a program’s intended impact(s).
For example, your proposed theory might be, “If we provide English language learning (ELL) services to recent immigrants, then their English skills will improve, which will, in turn:
reduce barriers in their everyday lives, and
lead to an increased sense of belonging and wellbeing.”
That said, the research you’d seek out might include articles about programs that offered high-quality ELL services that measured change (from program inception to one or more years post-program inception) in:
how participants navigated their everyday lives, and
their sense of belonging and well-being.
Why a literature review?
Nonprofit organizations generally have limited – if any – budget or capacity to conduct long-term follow-up evaluations. Nonprofit leaders: Consider a lit review. The research articles you’ll review are written by researchers who have the capacity, resources, and expertise to conduct such high-level, scientifically rigorous studies. What does this mean? Their findings are trustworthy. Try to glean insights from existing, applicable research instead of conducting your own long-term studies. Leave the research to the researchers! By reviewing their findings via lit review, you and your teams can continue focusing your capacities, resources, and expertise on high-quality program delivery and short-term outcome reflection.
Simple is savvy.
Focus.
Hone in on what’s useful to your programs. That might look like:
a collection of 3–to–10 high-quality sources that focus specifically on elements of your organization’s program(s), and
a brief, written summary of findings. The summary can be used as a touchstone to:
frame how program staff understand the work,
Inform the “why” behind the work,
act as the foundation of an evaluation effort for the program(s), and/or
support fundraising campaigns.
So long as you have a) clarity on what information you’re seeking, and b) the necessary capacity, a lit review can be completed in just a few weeks (based on our experience).
Frame the program for staff.
A brief literature review can be an invaluable tool in clarifying the overall design of the program and what specific aspects of the program are most important for driving the outcomes you hope to see. When paired with a logic model or theory of change, it can be used as a tool for onboarding program staff.
Find your foundation for program evaluation.
Program evaluation doesn’t need to include measuring the long-term outcomes for participants if the program lacks the necessary resources; instead, it can focus on the first half of the theory of change equation: If we provide high-quality services, we will see these immediate results. We know that long-term positive change demonstrated in the research will not be a reality for participants if implementation of your program activities is incomplete or departs significantly from your plan.
The first step in program evaluation is to monitor and answer pertinent questions like:
How well is the program being implemented?
Are participants fully engaged?
Next, the evaluation needs to capture data on immediate or short-term outcomes that, according to the research, are predictive of longer-term change. For example, if the lit review for the ELL program indicates that reaching a certain grade level of literacy in English reading and writing in the near term predicts long-term increases in wellbeing, then your evaluation would focus on measuring the grade level literacy of participants once they’ve completed the program. The findings from a lit review can inform how you define and measure implementation and immediate outcomes.
Fundraise in alignment with your evidence-based approach.
Lit reviews that demonstrate an evidence base for your program activities can be a powerful tool in fundraising, particularly if paired with specific program evaluation data. Funders and grant makers typically ask nonprofits to demonstrate and report on the long-term outcomes of their work; however, as discussed, time and funding opportunities for long-term research work is often scarce in nonprofit land. Being able to submit a list of citations and a brief summary (perhaps accompanied by a copy of your program’s logic model or theory of change) shows that a) you’ve done your homework, and b) your program is rooted in evidence.
Five Steps to Conduct Your Own Lit Review
1. Frame your questions.
Literature reviews start with framing questions. Based on your understanding of the program’s activities and intended long-term outcomes, write a set of questions to guide your search. You might consider:
one question per intended outcome or planned activity, or
a more general question that allows for more possibilities. (Note: This more open-ended approach will require some narrowing later in the process.)
For example, your framing question might be, “What long-term outcomes occur for adult participants who complete the ELL curriculum?” This question leaves you open to a variety of outcomes that researchers may have investigated. A narrower question might be, “What is the linkage between completion of ELL curriculum and sense of belonging among adult immigrants?”
2. Clarify your criteria.
Once you’ve developed one or more framing questions, you’ll need to write down a set of criteria that defines what’s relevant for your search. Specific criteria to address include:
How old is “too old” for a research paper?
What types of sources are credible?
peer-reviewed research
institutional research
What populations need to have participated in the research?
For example: If your program serves only immigrants learning English as a second language, then research on literacy among native English speakers is likely irrelevant.
What activities must the research include if it’s to provide a fair comparison to your program’s offered activities?
For example: If you offer a six-week, in-person course, then research about an asynchronous online course would be incongruous.
What outcomes are critical to capture in your lit review?
If there are particular outcomes that you report to funders already or routinely talk about in your materials, it will be helpful to include a few sources that look specifically at those outcomes. If you can’t find research that links your activities to a key outcome, then that’s a flag to slow down and look deeper at your theory of change.
3. Build your container.
We recommend a simple Google spreadsheet to house sources as you conduct your lit review (It’s what we do!).
4. Search!
Free, high-quality research is now widely accessible online. For lit review novices and experts alike, we recommend these tried-and-true resources:
Urban Institute: One of the foremost public policy and program research institutes, focusing on a wide range of topics that each contribute to overall individual and community thriving
MDRC: The gold standard for social policy evaluations. If you want to know “what works” in education, poverty reduction, or health, their randomized controlled trial (RCT) results are definitive.
Brookings Institution: Focuses on global and domestic policy. Use this for macro-level context on social trends, governance, and economic development.
Google Scholar: The easiest starting point.
JSTOR (Open Content): JSTOR has a massive collection of open-access journals. With a free personal account, you can access up to 100 articles per month.
DOAJ (Directory of Open Access Journals): A community-curated list of open-access journals. This ensures that you’re finding high-quality, peer-reviewed content that is 100% free to read.
IssueLab (by Candid): This is, perhaps, your most important tool. It’s a searchable repository of over 30,000 reports, evaluations, and case studies published by foundations and nonprofits.
The Campbell Collaboration: Think of this as the “Cochrane Library” for social science. They produce systematic reviews that synthesize all available research on a specific intervention (e.g., “Do after-school programs reduce crime?”).
5. Synthesize.
Once you’ve gathered a set of sources that meet your criteria, read them and summarize the relevant findings.
