Identifying Measures to Evaluate Network Progress
How to use both quantitative and qualitative data to tell your network's story
By Esther Hammerschlag, MA, East Chewuch Consulting
Network development is an iterative and dynamic process that doesn’t typically follow a straight trajectory, meaning an evaluation plan which only tracks numbers won’t paint a complete picture of your network. I like to think of evaluation first as telling the story—a story that is then backed by numbers. Although people may argue the benefit of one type of data over another (quantitative v. qualitative), the reality is you need both to tell the full story of your network. While numbers provide measurements across time, contextual and anecdotal information bring your numbers to life. Finding this balance and artfully merging the two is where you can most successfully demonstrate your network’s value.
To accomplish this as a network evaluator, I like to administer surveys to members and/or stakeholders that include both closed-ended (e.g., multiple choice or scales) and open-ended questions. This allows me to:
- Develop quantitative data based on responses to closed-ended questions
- Analyze responses to open-ended questions for themes
- Extract relevant and meaningful quotes that personalize and contextualize the data and themes
- Develop meaningful dashboard tools that monitor network function over time
Collecting and analyzing qualitative data
As an example, I recently completed a one- year follow-up evaluation of a relatively new, informal network that was working to improve its infrastructure and capacity. As one way to demonstrate value, I included an open-ended survey question asking members to describe how they had personally benefitted from participating in the network. I analyzed their responses using the same themes identified in last year’s survey, to allow comparison from one year to the next.
By doing so, I found that an increasing number of members realized benefit from the network: almost half of members cited close ties to their own personal, professional, and organizational goals (nearly double than in 2016), and two-thirds indicated an increase in their own skills, knowledge, and awareness (an increase from 50% in 2016).
I then looked at responses to a multiple-choice question asking members how much they agreed or disagreed with the statement, “The goals of the coalition support the goals of my own organization.” More than double the percentage strongly agreed with the statement in 2017 than did in 2016. (Do you see the pattern emerging?)
Knowing organizations want to see numbers—and changes over time—I developed a graph showing progress in these indicators from 2016 to 2017. These indicators could continue to be measured in future years by asking the same questions, analyzing responses the same way each time. This graph shows how you can translate qualitative (descriptive) data into numerical displays of network value and progress over time.
Several members also provided valuable comments in their responses, especially as to how participation has helped them within their own organizations. One member said, “Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition,” and another stated, “It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.”
Turning data into a story
You can begin to see how this network’s data emerges into a story. Using this example, the following passage about member benefit could be used in newsletters, annual reports, or relevant funding applications (names were changed for the purposes of this article):
“Not only do an increasing number of our members feel the goals of the network support the goals of their own organization, but a growing number of our members indicate an increase in their own skills, knowledge and awareness. In fact, two-thirds of our members in 2017 cited an increase in their own skills, knowledge and awareness as a result of network participation, and more than double the number of members in 2017 as in 2016 strongly agreed that the goals of the network support the goals of their own organization. According to Joe Smith, CEO of Rural Community Hospital, ‘Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition.’ Lucy Brown, the Executive Director of Family Planning Clinics states, ‘It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.’”
You could develop similar vignettes for other elements of network value using the same approach. For example:
- Describe challenges faced by your network. How have they diminished over time? What characteristics of your network have been important to addressing these challenges?
- What are the greatest accomplishments of your network? How have they benefited the community? What characteristics of your membership have contributed to these accomplishments?
- Describe a specific project your network has undertaken. How has it benefited the network, network members, and the community?
Identifying the right data to collect
Before you develop a survey to collection evaluation data for your network, make sure to clearly define your goal. For example, do you want to quantify benefits to the community? Do you want to quantify challenges your network has faced, and demonstrate how you’ve overcome these challenges? Or do you want to evaluate strengths and challenges related to your network’s infrastructure and leadership?
Some of my favorite open-ended questions to ask when evaluating networks are:
- What are the greatest successes of the network?
- How has the network benefited the community the most?
- What characteristics of the network contribute to the successes and benefits you mentioned?
- How have you personally benefited from your participation in the network?
- What are the biggest challenges the network faces?
- What do you think has contributed to the challenges you mentioned?
- What are the greatest opportunities for the network?
Responses to these types of questions can be sorted into themes and then quantified, as in the example I showed you. To provide additional measures that corroborate these themes, some of my favorite closed-ended questions use scales. For example:
- Ask members to rate their perception of likelihood for network sustainability
- Ask members how much they agree or disagree with a variety of statements related to elements of leadership, member participation and engagement, and indicators of a shared agenda and priorities
- Ask members to rate the importance of various areas for development such as obtaining funding, addressing member turnover and growth, or obtaining organizational buy-in and support
Finally, think about developing dashboard tools as you write your surveys. How might responses to questions be turned into data points that can be tracked over time? Whether you want a big-picture snapshot of your network’s value, or you want to hone in on something more specific, using this approach will help you tell your story in a compelling way. Sharing a compelling network story is one of the best ways to maintain engagement and participation of your members and stakeholders, strengthen funding applications, and gain financial backing from your members by personalizing your request to make it more meaningful.
Esther Hammerschlag, MA, is the owner of East Chewuch Consulting and provides a variety of program planning and development services for health and social service programs, including strategic planning, community health assessment, and program evaluation. As a former network director, she has a special interest in working with health networks that are working to improve the health of rural communities. She brings over 17 years’ experience in the planning and implementation of a variety of health and social service programs from the perspectives of consultant as well as community facilitator, and holds a master’s degree in Rural Development from the University of Alaska Fairbanks with a focus on creating effective and meaningful collaboration to improve community health