Community Management

7 Steps to Create Effective Surveys to Collect Community Feedback

Donal O'Conghaile

April 2, 2024

Do you know what your members want? Does your community give you its opinion, or have you ever asked? A community survey is an excellent way to gather feedback en masse to inform the business decisions that affect your community.

When planning a complete redesign of Smiley360.com, a community of consumers who experience brands for free and share their opinions, getting the input of members was essential. After all, our community lives on our site and our site lives because of our community, so members should have a say in any major changes we make to it.

We knew we needed to rebuild the site from the ground up, but how could we do that without upsetting or disregarding the 400,000 members who had joined us over four years?

A Community Built On Feedback

We launched Smiley360 in December 2009 to meet a big need for brands looking for ways to get people talking about them in an open and honest way. People join Smiley360 for fun, to experience new products, services and content, and to share their experience with friends—and brands want the feedback Smiley members can provide.

As the community grew steadily, it became apparent that we had outgrown the original site. We decided to rebuild our application, but not before asking members what they wanted (therefore preventing community backlash down the line). To start this process the right way, we launched a Community Feedback survey for members to have their say in the future of Smiley360.

This could work for any community builder at any point in their journey: pre-launch, before a big change, even just an ongoing survey in order to let your members know that you’re listening.

Thousands of our most vocal and opinionated members took this survey. A thriving community of members who love to talk… a community manager’s dream. But collecting this feedback is no easy feat.

Here we share our process for how to create a community survey to collect useful, unbiased, and actionable feedback from members to help inform major decisions for developers, clients, or your own internal team.

7 Steps to Create a Community Feedback Survey

Now we’ll go into each step in detail so you can collect the data you need and act right away for the benefit of your community.

1. Set Your Objectives

Before you begin writing questions for your community survey, ask yourself and your team what your objectives are:

What do we want to learn?
How is this survey data going to help us?
What do our members want from us?
How do members feel about the community? What do they love and what would they change?
How can we provide a better experience for the entire community?

Make assumptions about the answers you expect to see, and write survey questions that will either confirm or reject your assumptions. For the re-launch of Smiley360.com, we focused on what features our users spent the most and least time on, what new features they wanted us to introduce next and how they felt about the design and layout, text content, and ease of use of the website.

In your community, you might run a survey to define or redefine your community guidelines, validate an idea you’re thinking of launching, or simply measure satisfaction and utility of the community.

2. Write Your Survey Questions

There are a number of things to keep in mind to ensure that your community survey is easy for your members to complete, and gives you the most useful results.

Watch Out For Bias

Unless you have a survey professional at your disposal who knows what she’s doing, be extra careful when writing your questions so that survey questions do not lead members to answer in any particular way.

You might be tempted to write:

Instead, keep your options balanced, with a neutral opinion in the middle:

It’s important to choose your words carefully so that you do not influence the member. Consider this example of how leading questions can influence the testimony from an eyewitness of a car accident:

How fast was the car going when it smashed into the other car?
How fast was the car going when it hit the other car?
How fast was the car going when it made contact with the other car?

Which of these questions have leading language, and which one is bias-free?

Choose the Right Question Types

There is more than one way to ask a question. There’s also a right way and a wrong way. Consider this example:

You might want to know where your members spend the most time on your site, but the problem is that this question is anything but specific. It does not account for people who visit only the FAQ pages but not the blog, or vice versa. Furthermore, analyzing the responses from this question would not give you any real insights. Instead, rephrase this question as:

We asked a question just like this to learn where our members spent the most time. The results? We were able to cut the least-used parts of our website and focus on the areas our members value the most.

It’s A Numbers Game

When asking questions around numbers, you must make sure that everyone can fit into only one box. For example, consider this question:

What about someone who spent $20 or $40 exactly? That person doesn’t know which box to pick, but they have to choose one. This will muddle your data, and you would be surprised at how often this mistake is made. Instead, make sure that a person can fit into only one range, or better yet, ask for exact numbers—this way you can calculate means, modes and other statistically relevant information.

Beware Of Open-Ended Questions

Maybe you really need to know your community’s ice cream flavor preferences, but a question like this will make analyzing the responses a nightmare. Why? You’re probably not managing a community of grammar-obsessed English teachers, and your members will most likely have a hundred-and-one different ways of spelling Chocolate Fudge Brownie (Choc Fudge Brownie, Chocolate Browny, Fudgy Brownie, that one with the brownie bits inside…)

The lesson? Never use an open-ended field when you can get specific. In the basic ice cream example above, give users a dropdown menu of all delicious Ben & Jerry’s flavors. The results will give you statistically accurate preferences—meaning you can treat your members to their favorite flavors.

In some cases, you will have to use an open-ended-question type. In all of our surveys, we like to end with something like:

It is always the final question, so if any thoughts popped into respondents’ minds throughout the survey, now is their chance to share.

Don’t Skip Over Skip Logic

Skip logic means designing a custom path through a survey that changes based on a participant’s responses. If a portion of your survey is about being a parent, the important question to ask is Do you have children? If the participant answers no, do not send him through 10 questions about children. Send him to the next portion of the survey that is not about having kids. That is skip logic—and you should use it at every available opportunity.

When writing a survey, your goal is to keep it as short as possible for members. If you’re asking question after question that is not relevant to the member, don’t ask it. You will waste your participant’s time, and risk not receiving any responses at all.

Now I hear you asking, what is the optimal length of a survey? The answer: as long as it needs to be—and no longer. If you need to ask three questions to accomplish your goal, ask them. If it needs to be 50, ask them. Be weary of too many non-required questions. If the question is not required, can you just save it for a follow-up interview? Just be up front about the length of time your survey takes, so the member knows what they are getting into.

Want to know the top 10 metrics you should be using to track and measure your community’s business value? Download our free report here.

3. Test, Test, Test!

Hold up! I know you want to blast this survey out and start seeing feedback right away, but don’t skip the two rounds of testing necessary to help finalize your survey. First, test internally with your team to make sure everything is in line with the goals you set in Step 1. Here’s where you also ask your team to look out for typos or awkwardly phrased questions—we are blind to our own mistakes!

The second round of testing involves piloting your survey with a small number of members. Have 50,000 community members from whom you want to collect feedback? Start with a pilot group of 500. This way, you can make sure that participants are going through the survey in the way you have intended.

4. Send It To Members

You have a number of options to deploy your survey:

Set up the survey using a service such as SurveyMonkey, SurveyGizmo, Zoomerang or Google Forms and share the link through all your communication channels, especially email and social media.
If you want to target a subset of your community, send it out to a smaller list via a personalized email.
If resources allow, conduct your survey over the phone to focus on a small group of your most active members.
Mail out a paper copy of your survey and include a pre-paid return envelope. It’s true—some people still use snail mail!

We deployed our Community Feedback survey on our own Smiley360 platform. We already send out an average of eight surveys each month for clients, so it made the most sense to stick to where our members are accustomed to giving feedback. Close to 10,000 members weighed in with their opinion, which was more than enough to deliver useful results.

5. Collect Responses

You sent out your survey, and responses are starting to roll in. Depending on the engagement rate within your community, you might need to give members a little push to get a large enough volume of responses for statistically relevant data.

You can consider providing an incentive for your members to take action. A giveaway is a surefire way to motivate people to give up 5 to 10 minutes of their time to answer your questions. Offer up some gift cards or a prize more specific to the interests of your community.

If your survey’s deadline is approaching and you’d like to get a bump in your response rate, send out a gentle reminder email. Just be sure to filter out those who have already taken the survey as well as those who have expressed that they are not interested.

Keep in mind that the members of your community have different levels of passion and engagement. Some will give you feedback at any and every possible chance because they care deeply about the community and its future. Others might not have any opinion on your community’s next steps. It’s important to meet both of these types where they are at, as even more disengaged members may have ideas for getting more engaged.

6. Analyze the Responses

Here’s the fun part—studying your survey data and turning that information into actionable steps for your developers, clients, or your own internal team. For a thorough and accurate analysis, follow a five-step process:

Clean up your data by removing any invalid responses. Depending on the survey software you have used, your results may include participants who did not complete the survey to the very end. These people might have begun taking your survey and realized midway through that it was not relevant to them—you don’t want their responses. Also, look out for any bogus responses from people who just want to win your prize. You can spot them by watching out for ‘straight-lining’ (respondents who have chosen the same option for every question) or typed gibberish in the text boxes.
Summarize your data in a spreadsheet. Putting your results into a readable form lets you and your team study the results. Software services such as SurveyMonkey can do this for you automatically.

If applicable, break up your data into categories. You can segment your respondents into categories such as gender, age, satisfaction or activity rate and see how the results vary within each group.
Visualize the results for your developers, clients or anyone else who will be looking at your results. Give them bar charts, pie charts and graphs to help interpret the data.
Read your open-ended responses. Analyzing this data is not easy, because you have no other choice but to read through all the responses. If you notice an issue coming up repeatedly, it is probably a feature worthy of serious consideration. You can use word clouds here to show what terms came up the most, but keep in mind that word clouds might not be scientifically accurate. Open-ended responses give the opportunity to add qualitative information to the quantitative data received.

7. Act on the Results

With feedback, comes action:

Study the results and interpret what’s meaningful and what’s not.
Prioritize what needs to be done now and what can wait.
Having trouble figuring out what to work on first? Go back and ask your community.
Allocate resources accordingly.

Some issues may come to light that you hadn’t even realized. Ask yourself, is this something that just a few particularly loud users are raising? Or is this something that affects the community at large? Be careful to filter through these ideas and suggestions and focus on the most important ones that align with your community, business, and client goals.

Our Survey Says…

In our case, feedback from community members validated our roadmap for how to re-launch our application. If you’re a community manager interacting with your members every day, you have a good idea of what your members want—collecting data from an in-depth survey will give you additional learnings to drive action.

Donal O'Conghaile

April 2, 2024