Identifying Measures to Evaluate Network Progress

September 2017

How to use both quantitative and qualitative data to tell your network's story

By Esther Hammerschlag, MA, East Chewuch Consulting

Network development is an iterative and dynamic process that doesn’t typically follow a straight trajectory, meaning an evaluation plan which only tracks numbers won’t paint a complete picture of your network. I like to think of evaluation first as telling the story—a story that is then backed by numbers. Although people may argue the benefit of one type of data over another (quantitative v. qualitative), the reality is you need both to tell the full story of your network. While numbers provide measurements across time, contextual and anecdotal information bring your numbers to life. Finding this balance and artfully merging the two is where you can most successfully demonstrate your network’s value.
 
To accomplish this as a network evaluator, I like to administer surveys to members and/or stakeholders that include both closed-ended (e.g., multiple choice or scales) and open-ended questions. This allows me to:

  1. Develop quantitative data based on responses to closed-ended questions
  2. Analyze responses to open-ended questions for themes
  3. Extract relevant and meaningful quotes that personalize and contextualize the data and themes
  4. Develop meaningful dashboard tools that monitor network function over time

Collecting and analyzing qualitative data

As an example, I recently completed a one- year follow-up evaluation of a relatively new, informal network that was working to improve its infrastructure and capacity. As one way to demonstrate value, I included an open-ended survey question asking members to describe how they had personally benefitted from participating in the network. I analyzed their responses using the same themes identified in last year’s survey, to allow comparison from one year to the next.

By doing so, I found that an increasing number of members realized benefit from the network: almost half of members cited close ties to their own personal, professional, and organizational goals (nearly double than in 2016), and two-thirds indicated an increase in their own skills, knowledge, and awareness (an increase from 50% in 2016).
 
I then looked at responses to a multiple-choice question asking members how much they agreed or disagreed with the statement, “The goals of the coalition support the goals of my own organization.” More than double the percentage strongly agreed with the statement in 2017 than did in 2016. (Do you see the pattern emerging?)

Knowing organizations want to see numbers—and changes over time—I developed a graph showing progress in these indicators from 2016 to 2017. These indicators could continue to be measured in future years by asking the same questions, analyzing responses the same way each time. This graph shows how you can translate qualitative (descriptive) data into numerical displays of network value and progress over time.
 
Several members also provided valuable comments in their responses, especially as to how participation has helped them within their own organizations. One member said, “Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition,” and another stated, “It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.”

Turning data into a story
 
You can begin to see how this network’s data emerges into a story. Using this example, the following passage about member benefit could be used in newsletters, annual reports, or relevant funding applications (names were changed for the purposes of this article):

“Not only do an increasing number of our members feel the goals of the network support the goals of their own organization, but a growing number of our members indicate an increase in their own skills, knowledge and awareness. In fact, two-thirds of our members in 2017 cited an increase in their own skills, knowledge and awareness as a result of network participation, and more than double the number of members in 2017 as in 2016 strongly agreed that the goals of the network support the goals of their own organization. According to Joe Smith, CEO of Rural Community Hospital, ‘Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition.’ Lucy Brown, the Executive Director of Family Planning Clinics states, ‘It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.’”

You could develop similar vignettes for other elements of network value using the same approach. For example:

  • Describe challenges faced by your network. How have they diminished over time? What characteristics of your network have been important to addressing these challenges? 
  • What are the greatest accomplishments of your network? How have they benefited the community? What characteristics of your membership have contributed to these accomplishments?
  • Describe a specific project your network has undertaken. How has it benefited the network, network members, and the community?

Identifying the right data to collect
 
Before you develop a survey to collection evaluation data for your network, make sure to clearly define your goal. For example, do you want to quantify benefits to the community? Do you want to quantify challenges your network has faced, and demonstrate how you’ve overcome these challenges? Or do you want to evaluate strengths and challenges related to your network’s infrastructure and leadership?
 
Some of my favorite open-ended questions to ask when evaluating networks are:

  • What are the greatest successes of the network?
  • How has the network benefited the community the most?
  • What characteristics of the network contribute to the successes and benefits you mentioned?
  • How have you personally benefited from your participation in the network?
  • What are the biggest challenges the network faces?
  • What do you think has contributed to the challenges you mentioned?
  • What are the greatest opportunities for the network?

Responses to these types of questions can be sorted into themes and then quantified, as in the example I showed you. To provide additional measures that corroborate these themes, some of my favorite closed-ended questions use scales. For example:

  • Ask members to rate their perception of likelihood for network sustainability
  • Ask members how much they agree or disagree with a variety of statements related to elements of leadership, member participation and engagement, and indicators of a shared agenda and priorities
  • Ask members to rate the importance of various areas for development such as obtaining funding, addressing member turnover and growth, or obtaining organizational buy-in and support

Finally, think about developing dashboard tools as you write your surveys. How might responses to questions be turned into data points that can be tracked over time? Whether you want a big-picture snapshot of your network’s value, or you want to hone in on something more specific, using this approach will help you tell your story in a compelling way. Sharing a compelling network story is one of the best ways to maintain engagement and participation of your members and stakeholders, strengthen funding applications, and gain financial backing from your members by personalizing your request to make it more meaningful. 

​​​​​Esther Hammerschlag, MA, is the owner of East Chewuch Consulting and provides a variety of program planning and development services for health and social service programs, including strategic planning, community health assessment, and program evaluation. As a former network director, she has a special interest in working with health networks that are working to improve the health of rural communities. She brings over 17 years’ experience in the planning and implementation of a variety of health and social service programs from the perspectives of consultant as well as community facilitator, and holds a master’s degree in Rural Development from the University of Alaska Fairbanks with a focus on creating effective and meaningful collaboration to improve community health

Networking News

Using Network Analysis For Evaluation and Design

December 2017

The effectiveness of any collaborative group is highly dependent on the quality of connections that exist among participants. This article describes a sample network to show how network leaders can use Social Network Analysis to measure relationships, increase collective self-awareness, and maximize collaboration strategically.

Networking News

Identifying Measures to Evaluate Network Progress

September 2017

This article describes approaches to measuring results when you want to tell the story of a rural health network's development. It provides ideas for questions to include in annual surveys and offers suggestions for how to use data collected from members and stakeholders.

Networking News

Why Communication Planning is Different for Networks

August 2017

Because of the unique nature of networks, successful communication planning often means embracing approaches that diverge from traditional tactics. This article discusses how to build effective network communication strategies and shares insights on common challenges and solutions.

Networking News

Leadership Influence Without Authority

May 2017

Understanding the motivational priorities for different personality styles can help a leader be aware of how to frame information when influencing others. This article provides a number of steps that rural health network leaders can take as they work on strengthening their credibility and building influence.

Networking News

Aligning and Streamlining Your Planning Efforts for Long-Term Success

April 2017

In taking a systems approach, rural health networks are often encouraged to create working plans that document their commitment to, and strategies for, achieving a shared vision in a number of different focus areas. This article outlines the different purposes of several plans and how they can be connected to one another.

Networking News

From Silos to Bridges: Collective Action to Build Rural Health Care Capacity

January 2017

Rural health networks often serve as bridges among different organizations and groups throughout a community. This article describes what brings stakeholders to the table, why they stay engaged in network activities, and how they can address their collective challenges in rural health care together by building connections.

Networking News

How an Understanding of Lean Can Support Network Objectives

November 2016

Performance improvement systems and tools can support the problem-solving processes of rural health networks by offering straightforward methods for analysis and implementation. This article outlines some of the ways that network leaders can benefit from the Lean methodology.

Networking News

Sharing Data in Collaborations

March 2016

This article provides a step-by-step process for preparing rural health networks for HIT interoperability and health information exchange.