Editor’s Note: You can learn more about community measurement strategy by enrolling in  our CMX Fundamentals of Community Strategy online course. Learn more here.

A newly hatched baby chick is adorable. Who can resist the sight of a little fluff ball with a tiny beak? Growing the size of that chick by 10 times wouldn’t make it 10 times more adorable though. In fact, some positive qualities of that baby bird would be lost with a supersized version. Imagine the scene in a store with a kid trying to convince his parents to take home a gigantic pet.

cartoon

There can be a similar dynamic with communities — bigger is not always better. Sometimes a small group of core community members can be more effective than a larger group of more casually engaged people.

If growth isn’t always the best metric to use, what other options are there?

This article will take a look at concerns with using growth as a measure and will provide case studies of different metrics you may find useful — retention, diversity, and maturity. This information can help you build a compelling case for why these other measures may be helpful in your own work.

Pitfalls of Focusing on Community Growth

At the Mozilla Summit 2013, Mozilla’s Director of Special Projects Chris Hofmann tells a story about one amazing contributor to the open source community who filed over 3,000 Firefox bugs and edited over 5,000 support articles.

If you’re only measuring and optimizing for community member growth, you could overlook these superstars. Adding one new person who contributes at that level won’t change the size of your community much, but it certainly would contribute more to your actual goals — and that’s what matters to the business at the end of the day.

Chris Hoffman at Mozilla Summit 2013

Chris Hoffman at Mozilla Summit 2013

If growth is so misleading, why are so many of us focusing on it to the exclusion of more important community health metrics? Two key reasons emerge.

1. Pressure to Make Apples-to-Apples Comparisons to “Traditional” Business Health

Many of us face pressure to use growth as our main measure of success even if it doesn’t tell the full story of our community’s success. One reason is that growth in revenue and users are standard metrics used by organizations to show business health.

There is an expectation to fit community into that framework, so executives can make an apples-to-apples comparison across all functions. Unfortunately, that’s simply impossible or ineffective in most organizations.

Unlike your company’s revenue and customers, you may not want your community to grow forever. Here’s the heart of the issue: The terms ‘users’ and ‘customers’ are often conflated with ‘community members’ in many organizations, although the relationships are very different.

That deeper relationship between the company and community member means that the cost and effort involved in coordinating a healthy community rises exponentially as more people are involved. The return on investment as the community grows, then, often drops off steeply.

If your measure of community success is growth, your incentives may be pushing you to scale to the point where you’re putting more effort into managing your community than you’re getting value back out.

There is a lot of writing on this subject, most notably Fred Brooks’ book The Mythical Man-Month, where he talks about the downsides of larger groups and states that adding more people to a delayed software project makes it launch even later.

2. Let’s Face It: Finding Other Metrics Is Hard Work

Another reason that growth is often used as a success metric is that other community metrics can be very difficult to track. Size is often the easiest number to measure.

To take another example from Mozilla, there is a small marketing budget for Firefox and the organization relies on good worth of mouth from the community to tell the world about what they’re doing. There is no way to count how many people have told their friends about Firefox or helped someone install it on their computer, so the growth in an Ambassador program can serve as a proxy for those activities.

Choosing that easier measure over harder-to-track metrics can mask serious problems. When Mozilla set a growth goal in 2014, the size of the community got larger, which made it seem like things were on the right track. Because other metrics took more effort to measure, it took a while until we could see a troublesome pattern emerging in the data.

Once we did, we discovered a startling retention problem — the chart below shows the number of people joining and leaving over a period of 20 weeks.

mozilla

This chart shows that growth efforts were being undercut by other issues that needed to be addressed. If new community members are joining while long-term members are leaving, the overall skill level in the community decreases. It may be better to pivot to training people to rebuild those skills instead. If the new members are getting discouraged and are leaving soon after arriving, it may be better to pivot to programs, such as mentorship and onboarding, that help new arrivals get settled in and get past initial hurdles.

If you are looking at just your community’s growth, you are likely missing important information. Getting a broader understanding of what’s happening by bringing in other measures can help you learn more about your community and tell a better story about the value it is creating.

So what are some of the other metrics you may want to consider?

Alternative Metrics: Retention, Diversity, Maturity

A. Measuring Community Retention

A retention rate measures the number of members who remain active in your community compared to the number of people who leave or become inactive. This is a good metric in particular for established communities because much of the history, information, and know-how is held by long-time members. They need to remain involved long enough to pass that on to newer members.

Marketing and sales commonly use retention rates, so this is a number your organization will already understand.

A slightly different version of the retention metric is to measure the “age” of your community. According to Bitergia co-founder Jesus M. Gonzalez-Barahona, “This term refers to ‘length of time in the project’ and measures how long ago each current member joined it.” If you retain members, the average length of time that members have been in your community will increase. An increase in community age is a nice proxy for the skills, experience, and the capacity of a community.

In order to measure retention, you will need to define what an active community member means for your project. At Mozilla, we defined an active community member as someone who had interacted with one of our community tools (mailing list, bug tracker, etc.) within the last 180 days.

Once you’ve established a benchmark, you can then see how many people met that threshold or not during each period you want to measure, such as in a week or month. Be aware that different communities will need to have different measures of activity. For example, a community mostly made up of U.S.-based students who are active during the summer may want to define activity over an entire year or semester instead of 180 days.

One potential downside of measuring retention is that, by focusing on optimizing it to the exclusion of other metrics, you may fail to see the benefits of healthy turnover. When some existing members leave, they make space for new people to step up and allow new ideas to enter your community.

Some members may also be detrimental to your community’s functioning, and it would be a plus if they moved on and found a new community to join. Since the ideal rate of retention is not to have 100% of people stay forever, this metric can be tricky to interpret. Increasing retention is good, but increasing it at the expense of the community’s productivity and health is not.

Always step back and look at the bigger picture with retention.

B. Measuring Community Diversity

A diversity measurement provides information about the level of different perspectives in your community around different factors, such as gender, age, or nationality. For example, the Wikipedia Statistics dashboard tracks which regions editors come from. Participation from multiple regions is seen as a sign of a healthy community effort.

Similarly, the Dreamwidth community recognizes the value of diversity and calls this out as their greatest strength.

A diversity measure is particularly useful for communities that are focused on innovative tasks. Ann Barcomb explains that “diversity should lead to a more creative approach to problems and less groupthink”. This makes sense — having a wide range of perspectives on an idea should give you a wide range of potential solutions.

The classic book on open source, The Cathedral and The Bazaar, states this as Linus’ Law: “With enough eyeballs, all bugs are shallow”. You may not see the answer to a problem, but other people looking at the problem with their own skill sets can see a solution that you can’t.

One very relevant measure of diversity is the distribution of leadership roles. If an organization is trying to create a healthy community but they’re holding on to all leadership opportunities for employees, then that will hurt that community. A diverse leadership structure would have employees of that organization as well as volunteers or people employed by other organizations.

So, are all of the decision-makers in your community employees of your organization or do you see diversity in employment status with some people with responsibility being volunteers or employees of other organizations? Chris Buerger makes a very good point about the need for your community’s governance to include different perspectives in order to be healthy: “Good governance generally is a result of a broad set of inputs, not a single organization or person driving a specific agenda.”

The study “The Effects of Diversity on Group Productivity and Member Withdrawal in Online Volunteer Groups” points out a potential downside of diversity as a metric. In their research, they find that “increased diversity in experience with Wikipedia increases group productivity and decreases member withdrawal – up to a point.

Beyond that point, group productivity remains high, but members are more likely to withdraw.” As with the retention metric, you need to be careful how you interpret a diversity measure. Increasing this is helpful up to a point. The goal isn’t necessarily to increase diversity as much as possible without any foundational reason for doing so.

C. Measuring Community Maturity

A maturity measure provides information about the state of a community — are you just getting started or are you established and have all of the pieces you need to be successful? This is a good metric for new communities.

Maturity measurements demonstrate progress in the early days, when you are ramping up your strategy and things are just starting to come together. A maturity score also highlights gaps and will help you put together a roadmap for how to evolve your community.

This is the case for the Digital Manufacturing Commons (DMC) community, an open-source software project that launched over the summer and has started to receive a lot of interest. Jim Barkley, the project lead, has found a maturity score revealed whether or not he was providing a positive experience to new community members. Jim explains how he uses a maturity score along with a growth metric to make sure the two measures are in sync:

“If a new project ranks low on the maturity model, but the community grows rapidly, there is a real danger that the growth works against your project. If a project is unable to grow its practices to support the rapidly growing community, it will have a negative impact on the reputation and quality of the project.”

Jim produced his maturity score by using the book Producing Open Source Software as an example of what a fully mature software community looks like. Then he identified what pieces existed for the DMC and what pieces weren’t in place yet. For instance, the DMC has technical infrastructure such as wikis, version control and issue tracking but there is no contributor agreement in place yet. This work identified areas for improvement to make sure new people are supported when they start to join the community.

How do you create a maturity metric for your company?

You can do this by identifying another community like yours that has reached maturity to use as a model — what similar items does your community have in place and what items aren’t available? The Community Roundtable also provides a foundational maturity model to measure your community against.

This metric can also be made as sophisticated as you need. Instead of having a simple item count, Jim weighted items, so he could put emphasis on pieces that are more important to him. He also normalized the score on a scale of 1 to 100 so it could be put into a framework the business more easily understands — he could assign a maturity level using an A, B, C, D or F grade.

One potential downside of using maturity is that the relationship to other community metrics may change over time, and it may be difficult to interpret results. For example, for new communities, growth may increase as maturity increases in a linear fashion. However, over time, as the community matures, growth may plateau. It’s important to be aware of the story that this relationship tells.

No one metric can tell you everything you need to know, but if you tell a story that weaves together the different metrics that you’re monitoring, a clear narrative will take shape.

Conclusion

Dynamics within a group of people are complex, and one measure cannot capture all of that complexity. Consider using multiple measures to get a better picture of what’s happening in their community. This will allow you to tell a complex story that is a fitting reflection of the complexity of the relationships being built.

Growth, retention, diversity and maturity are also not the only measures you can choose. Depending on your community’s goals, you may also measure interest, conversion, satisfaction, capacity, or development. To get a better understanding of best practices with community metrics, please share your metrics in the CMX Hub discussion around identifying key metrics for community health.

David Boswell

David Boswell

David Boswell is a community expert who has over 15 years experience with non-profit and open source communities. He is currently the Director of Community at ON.Lab and has worked previously with Mozilla, NASA, Sun Microsystems, Nokia and more on their community efforts.