Approaching Data like an Analyst: From Data to Findings
Written by Jessica Gibbons
In our last journal entry, we looked at the most versatile options for data collection. Now let’s take a look at how we think about and practice data analysis at Elevate.
Context is Key
You need to understand how the data was collected from the jump. This can help explain what might otherwise be mysterious patterns in the data. For example, if a lot of people did not answer a specific question, you need to know if there was skip logic built into the survey or interview, which will clarify how many of those blanks are because the person simply was not asked the question or if they were asked and truly did not answer.
Other factors like if it was an online or in person survey or interview, when and where folks were asked to participate, and what the level of effort was for data collection. For example, if multiple organizations are responsible for data collection and Org A has a whole team dedicated to this while others are squeezing it into an already arm-long to-do list of existing staff, that would probably explain why we get more folks served by Org A than Org B - not because the folks served by Org B didn’t want to participate.
Review and Tidy Your Dataset
Humans make mistakes and are often inconsistent. When humans complete surveys, they often will do things like check “Yes” to the question “Are you a military veteran?” but then write that they have never served in the military in reply to the follow up questions. When analyzing data, we first review the dataset to check for inconsistencies, answers that aren’t valid (such as 1000 as an age or a rating of 12 on a 10 point scale), and outliers.
Best practice is to always keep a copy of the original data, create a clean copy where you have corrected or taken out inconsistent or invalid responses, and document the decisions you made about how to handle these issues. This way, you can always go back to see what you changed and why.
Keep it Simple (and useful)
For quantitative data, we primarily rely on just a few simple equations: Count, Sum, Average, Median, and Percentages. You can do this in Excel, Google Sheets, or more sophisticated softwares like R and SPSS.
For qualitative data, we look for the “strong signals" that came up most often and the context in which they are discussed, and we look specifically for “weak signals” that perhaps only one or two participants brought up. Weaker signals can be early warning signals or signals about equity issues that need to be tended to. For smaller qualitative datasets, we often do this in Excel, and for larger datasets we use Atlas.ti.
Stay focused on answering your key questions. Remember that you don’t actually have to do analysis on every piece of data in your dataset, and you certainly don’t have to look at every combination of interesting data points. Return to your evaluation focus and the information decision makers will actually use. Clarify with your decision makers what data they need to see broken down by important characteristics, such as race, location, income, etc.
At Elevate, we work in teams, so any data analysis we complete will generally be done collaboratively or at a minimum reviewed by our team mates to check our work for errors. A single cell reference being incorrect in Excel can make a world of difference, and it can be hard to spot small errors when you have been looking at the same spreadsheet for 5 hours (or more) straight. We recommend you have a colleague review and spot-check your analysis. It is much easier to catch an error before the report has been sent to the funder.
Identify Findings
Once you have built out your analysis, it is time to review it all together and write straight forward statements to describe the results. We often start by writing statements to answer the key questions from our stakeholders, and then expand to describe other results or relationships within the data that may be relevant. An example might be “The majority of participants in Program A are single mothers, with an average age of 35, who stay engaged in the program for 3 to 4 months,” or “The greatest gains were among participants who were served at locations 1 and 3”. This exercise gives you clarity on what the results are and what is most critical to communicate in the next phase of the life cycle, reporting.
