I’m all about data. I’ve spent most of my career collecting it and trying to extract meaning from it. I’ve worked with numerous organizations on data projects, first as a child policy researcher and now as a data visualizer. And I’ve found that many organizations spend a lot of time, money, and space collecting and storing something that they rarely use and do not need: data. I have a few suggestions for organizations struggling with data: Give up on measuring impact, just measure progress, and format data so that you can easily distill its meaning and consider next steps.

Give up on measuring impact

Measuring impact is difficult and expensive. It’s difficult because you need a good counterfactual: what would happen if you changed nothing. The impact of an intervention or program is the difference between what happened when a program or intervention was implemented and what would have happened without it. Since, in the real world, you can’t observe the same group of beneficiaries with and without the intervention, you need a good proxy for the would-have-been condition. The best type of proxy, a randomized control trial (RCT), requires randomly selecting folks to receive and to not receive an intervention and comparing those who did and did not receive it over time to estimate the impact of the intervention.

Of course, withholding an intervention from potential beneficiaries can be a difficult and morally questionable pursuit, and tracking large groups of beneficiaries and non-beneficiaries over time is expensive. Non-randomly selected comparison groups are not nearly as good counterfactuals because they may differ significantly from the intervention group at the outset, in both known and unknown ways, so it’s difficult to determine whether the outcomes observed are due to the intervention or to preexisting differences. This costly and challenging process is further complicated by the need to start with a well-established intervention, one that has already worked out the kinks.

Given the many challenges of measuring impact, most organizations should not waste time and money on impact evaluations. Instead, they should consider interventions that already have a strong research base, ideally because they have been rigorously tested with RCTs. In a Stanford Social Innovation Review article, Mary Kay Gugerty and Dean Karlan suggest that, before beginning a new program, organizations ask: “What do other evaluations say about it? How applicable is the context under which those studies were done, and how similar is the intervention? Study the literature to see if there is anything that suggests your approach might be effective.”

Just measure progress

Less can be more when it comes to data. As Gugerty and Karlan note, “[t]he pressure to appear ‘data-driven’ often leads organizations to collect more data than anyone can be reasonably expected to use. In theory, more information seems better, but in reality, when organizations collect more data than they can possibly use, they struggle to identify the information that will actually help them make decisions.”

Let’s say you choose to implement a rigorously tested intervention. The type of data that will help you to make good decisions is data that shows your progress. If you can pinpoint where things are going astray, you are in a better position to make fixes, alter plans, and refine processes.

Many organizations make their plan using a logic model, aka theory of change, a flow chart with inputs and outputs. It basically says: If we do this, then we expect this to happen. The best logic models draw on past impact evaluations to determine what inputs are most likely to lead to what outputs. But even carefully designed logic models are often neglected or forgotten once a program is funded and under way—which is too bad, because a logic model tells you exactly what data is worth collecting. This includes data which tells you whether each component of the logic model happened as you expected, including whether:

• You implemented the program as planned;
• You targeted the right type and number of people;
• You effectively engaged those people; and
• Participants had good experiences in the program and gained knowledge or changed behavior as you expected.

That last bullet point is important. Of course, you want to see some sort of change over time. But that in itself is not impact. If you see positive change, it may mean that your program is helping people, but, again, you would need a good counterfactual to measure its actual impact.

The type of data needed to measure progress usually includes enrollment data, attendance data, feedback survey data, and sometimes testing data to measure, for example, knowledge gained or pounds lost in a healthy eating program.

The Jewish United Fund of Chicago’s (JUF) Early Childhood Collaborative worked deliberately to streamline its data collection from the start, which was a challenge, given their broad goals. The coalition of early childhood directors, teachers, researchers, and community members aims to strengthen the pipeline of teachers and leaders; advance a culture of excellence in the schools; and expand access to education by lowering financial and geographic barriers that families may face. They began by drafting and redrafting a logic model based on the experiences of families and educators and research on the elements of quality preschool. The logic model became a collective roadmap for what they should do and what data points they should track. They then developed a parent survey specifically designed to track progress on each component of the logic model.

Format the data so that you can easily distill its meaning and consider next steps

Having narrowed their data collection to specific components of the logic model, the collaborative then faced the challenge of making that data accessible to all coalition members, many of whom were either data-hesitant or data-averse. That was my entry point into the effort. I worked with JUF to plug their logic model into their parent survey data. We created a data dashboard anchored by the logic model flow chart. On the dashboard, the components of the model are colored in different shades of blue to show where schools were exceeding expectations, meeting expectations, or falling short of expectations. And the dashboard allows members of the collaborative to click on any component to see more detailed views of the data over time. Moreover, they can filter the data by school or family demographics to examine the data from different angles. These features allowed members to pinpoint areas of challenge and strength for certain subgroups of families or certain schools.

School staff reflect on the data in regular meetings with a coach, and this practice has led to change. Before the dashboard was in place, school administrators would look at survey results and fixate on particularly negative comments on a few surveys. “The dashboard allows them to see the big picture. It has increased their savviness about data,” notes Kate Warach, the collaborative’s project manager. Schools have made various changes based on the data so far from alterations in the school calendar to the way students are greeted each day. Both large and small adjustments like these help individual schools and the collaborative as a whole to more closely realize the vision laid out in their logic model.

The collaborative’s experience shows that organizations need not be buried in useless data. Instead they can wield it more efficiently and effectively by making clear plans informed by research, collecting a limited amount of data, and then visualizing it to move quickly from data to action.

Amelia Kohm is founder of and consultant at Data Viz for Nonprofits.