Editor’s Note: You can learn more about community measurement strategy by enrolling in our latest online course, The Community MBA. Learn more here.
A newly hatched baby chick is adorable. Who can resist the sight of a little fluff ball with a tiny beak? But growing the size of that chick by ten wouldn’t make it 10 times more adorable. In fact, some positive qualities of that baby bird would be lost with a supersized version. Imagine a kid trying to convince his parents to take home a gigantic pet.
There can be a similar dynamic with communities. Bigger is not always better. Sometimes a small group of core community members can be more effective than a larger group of more casually engaged people.
If growth isn’t always the best metric to measure community health, what other options are there?
This article will take a look at concerns with using growth as a measure of community health and will provide case studies for different metrics you may find useful, including retention, diversity, and maturity. This information can help you build a compelling case for why these other measures can be a strong indicator of community health.
The Pitfalls of Focusing on Community Growth
At the Mozilla Summit 2013, Mozilla’s Director of Special Projects Chris Hofmann old a story about one amazing contributor to the open-source community who filed over 3,000 Firefox bugs and edited over 5,000 support articles.
If you’re only measuring and optimizing for community growth, you could overlook these superstars. Adding one new person who contributes at that level won’t change the size of your community much, but it certainly would contribute more to your actual goals —and that’s what matters to the business at the end of the day.
If growth is so misleading, why are so many of us focusing on it to the exclusion of more important community health metrics? Two key reasons emerge.
1. Pressure to Make Apples-to-Apples Comparisons to “Traditional” Business Health
Many of us face pressure to use growth as our main measure of community success even if it doesn’t tell the full story. One reason is that growth in revenue and users are standard metrics used by organizations to show business health.
There is an expectation to fit community into that health framework, so executives can make an apples-to-apples comparison. Unfortunately, that’s simply impossible or ineffective in most organizations.
Unlike your company’s revenue and customers, you may not want your community to grow forever. Here’s the heart of the issue: The terms “users” and “customers” are often conflated with “community members,” although the relationships are very different.
That deeper relationship between the company and community member means that the cost and effort involved in coordinating a healthy community rises exponentially as more people are involved. The return on investment as the community grows often drops off steeply.
If your measure of community success is growth, your incentives may push you to scale to the point where you’re putting more effort into managing your community than you’re getting value.
There is a lot of writing on this subject, including Fred Brooks’ book The Mythical Man-Month, where he talks about the downsides of larger groups and states that adding more people to a delayed software project makes it launch even later.
2. Let’s Face It: Finding Other Metrics Is Hard Work
Another reason that growth is often used as a success metric is that other community health metrics can be very difficult to track. Size is often the easiest number to measure.
To take another example from Mozilla, there is a small marketing budget for Firefox and the organization relies on worth-of-mouth from the community to tell the world about what they’re doing. There is no way to count how many people have told their friends about Firefox or helped someone install it on their computer, so the growth in an Ambassador program can serve as a proxy for those activities.
Choosing that easier measure over harder-to-track metrics can mask serious problems. When Mozilla set a growth goal in 2014, the size of the community got larger, which made it seem like things were on the right track. Because other metrics took more effort to measure, it took a while until we could see a troublesome pattern emerging in the data.
Once we did, we discovered a startling retention problem. The chart below shows the number of people joining and leaving over a period of 20 weeks.
This chart shows that growth efforts were being undercut by other issues that needed to be addressed. If new community members are joining while long-term members are leaving, the overall skill level in the community decreases. It may be better to pivot to training people to rebuild those skills instead. If the new members are getting discouraged and are leaving soon after arriving, it may be better to pivot to programs such as mentorship and onboarding that help new arrivals get settled in.
If you’re looking at just your community’s growth, you are likely missing important information. Getting a broader understanding of what’s happening by bringing in other measures can help you learn more about your community health and tell a better story about the value it is creating.
So what are some of the other metrics you may want to consider?
Alternative Community Health Metrics: Retention, Diversity, Maturity
1. Measuring Community Retention
A retention rate measures the number of members who remain active in your community compared to the number of people who leave or become inactive. This is a good metric for established communities because much of the history, information, and know-how is held by long-time members. They need to remain involved long enough to pass that on to newer members.
Marketing and sales commonly use retention rates, so this is a number your organization will already understand.
A slightly different version of the retention metric is to measure the “age” of your community. According to Bitergia co-founder Jesus M. Gonzalez-Barahona, “This term refers to ‘length of time in the project’ and measures how long ago each current member joined it.” If you retain members, the average length of time that members have been in your community will increase. An increase in community age is a nice proxy for the skills, experience, and the capacity of a community.
In order to measure retention, you’ll need to define what an active community member means for your project. At Mozilla, we defined an active community member as someone who had interacted with one of our community tools (mailing list, bug tracker, etc.) within the last 180 days.
Once you’ve established a benchmark, you can then see how many people met that threshold during each period you want to measure, such as a week or month. Be aware that different communities will need different measures of activity. For example, a community of US-based students who are active during the summer may want to define activity over a semester instead of an entire year.
One potential downside of measuring retention is that you may fail to see the benefits of healthy turnover. When some existing members leave, they make space for new people to step up and allow new ideas to enter your community.
Some members may also be detrimental to your community’s functioning, and it would be a positive if they moved on and found a new community to join. Since the ideal rate of retention is not to have 100% of people stay forever, this metric can be tricky to interpret. Increasing retention is good, but increasing it at the expense of the community’s productivity and health is not.
Always step back and look at the bigger picture with retention.
2. Measuring Community Diversity
A diversity measurement provides information about the level of different perspectives in your community around factors such as gender, age, ethnicity, or nationality. For example, the Wikipedia Statistics dashboard tracks where editors come from. Participation from multiple regions is seen as a sign of a healthy community effort.
Similarly, the Dreamwidth community recognizes the value of diversity and calls this out as their greatest strength.
A diversity measure is particularly useful for communities that are focused on innovative tasks. Researcher Ann Barcomb explains that “Diversity should lead to a more creative approach to problems and less groupthink.” This makes sense—having a wide range of perspectives on an idea should give you a wide range of potential solutions.
The classic book on open source, The Cathedral and The Bazaar, states this as Linus’ Law: “With enough eyeballs, all bugs are shallow,” You may not see the answer to a problem, but other people looking at the problem with their own skill sets can see a solution that you can’t.
One very relevant measure of diversity is the distribution of leadership roles. If an organization is trying to create a healthy community but they’re holding on to all leadership opportunities for employees, that will hurt that community. A diverse leadership structure would have employees of that organization as well as volunteers or people employed by other organizations.
Chris Buerger makes a very good point about the need for your community’s governance to include different perspectives in order to be healthy: “Good governance generally is a result of a broad set of inputs, not a single organization or person driving a specific agenda.”
The study “The Effects of Diversity on Group Productivity and Member Withdrawal in Online Volunteer Groups” points out a potential downside of diversity as a metric. In their research, they find that “increased diversity in experience with Wikipedia increases group productivity and decreases member withdrawal—up to a point.”
Beyond that point, group productivity remains high, but members are more likely to withdraw. As with retention, you need to be careful how you interpret a diversity measure. The goal isn’t necessarily to increase diversity as much as possible without any foundational reason for doing so.
3. Measuring Community Maturity
A maturity metrics provides information about the state of a community. Are you just getting started, or are you established and have all of the pieces you need to be successful? This is a good health metric for new communities.
Maturity measurements demonstrate progress in the early days, when you are ramping up your strategy and things are just starting to come together. A maturity score also highlights gaps and will help you put together a roadmap for how to evolve your community.
This is the case for the Digital Manufacturing Commons (DMC) community, an open-source software project that launched over the summer and has started to receive a lot of interest. Jim Barkley, the project lead, found that a maturity score revealed whether or not he was providing a positive experience to new community members. Jim explains how he uses a maturity score along with a growth metric to make sure the two measures are in sync:
If a new project ranks low on the maturity model, but the community grows rapidly, there is a real danger that the growth works against your project. If a project is unable to grow its practices to support the rapidly growing community, it will have a negative impact on the reputation and quality of the project.
Jim produced his maturity score by using the book Producing Open Source Software as an example of what a fully mature software community looks like. Then he identified what pieces existed for the DMC and what pieces weren’t in place yet. For instance, the DMC has technical infrastructure such as wikis, version control and issue tracking, but there is no contributor agreement in place yet. This work identified areas for improvement to make sure new people are supported when they start to join the community.
You can create a maturity metrics for your company by identifying another community like yours that has reached maturity to use as a model. What similar items does your community have in place, and what items aren’t available? The Community Roundtable also provides a foundational maturity model to measure your community against.
This metric can also be made as sophisticated as you need. Instead of having a simple item count, Jim weighted items, so he could put emphasis on pieces that are more important to him. He also normalized the score on a scale of 1 to 100 so it could be put into a framework the business more easily understands. He could then assign a maturity level using an A, B, C, D or F grade.
One potential downside of using maturity is that the relationship to other community metrics may change over time, and it may be difficult to interpret results. For example, for new communities, growth may increase as maturity increases in a linear fashion. However, over time as the community matures, growth may plateau. It’s important to be aware of the story that this relationship tells.
Conclusion
No one metric can tell you everything you need to know. But if you tell a story that weaves together the different metrics you’re monitoring, a clear narrative will take shape.
Dynamics within a group of people are complex, and one measure cannot capture all of that complexity. Consider using multiple measures to get a better picture of your community’s health. This will allow you to tell a complex story that is a fitting reflection of the complexity of the relationships being built.
Growth, retention, diversity and maturity are also not the only measures you can choose. Depending on your community’s goals, you may also measure interest, conversion, satisfaction, capacity, or development. To get a better understanding of best practices with community metrics, please share your metrics with the CMX Community and join the discussion around community health.
