Breaking Down Udemy’s Model for Tracking the ROI of Community

No matter where your responsibilities lie in your organization, you become what you measure. And yet many of us sitting in community roles are stuck on what, how, and when to measure community success.

Luckily, while defining community ROI can feel overwhelming, leaders before us have paved the way to success so we can learn from their triumphs.

Eliza Davidson, now Manager of Instructor Communications at Udemy, has been building community at Udemy since 2012. In that time, she has worked tirelessly to measure community success and share her wins. In 2014, for instance, she showed that Udemy instructors inside the community were 4 times as likely to publish a course on the platform than non-members.

From growth numbers alone, Udemy knows that instructor community has been key to success. Last year alone, the group almost doubled to almost 35,000 members. But, as any community professional can tell you, community growth is useless without context.

So, in 2015, Eliza set out to get even more granular about building a community ROI model that matched with company priorities. She helped create a community dashboard, streamline a product feedback cycle, and set up a reporting and goal-setting structure to keep their team aligned with others and moving toward more business impact.

Above all, Udemy is now able to point to what community activities drive engagement on the Udemy platform, what community programs bring together the most engaged instructors, and they can swiftly decide where to focus their team’s attention. That means their team is able to secure the resources they need to focus on the right work.

If you’re looking to create a measurement strategy for community in your organization, Udemy’s Eliza Davidson has you covered. She’s shown how to track how community work plays into the bigger picture of her company’s work, and how to get the resources and support she needs to make the Udemy instructor community even stronger.

1. Start measuring something, somewhere.

It’s okay to start measuring in a rudimentary way, with a general hypothesis and rough goal in mind. As long as you have an idea of what business value you’re driving, you just need to start somewhere and allow your measurement strategy to mature over time.

When Eliza started at Udemy in 2012, the goals for instructor community were simpler. There was actually just one distinct goal to achieve: get community members to publish more online courses on the Udemy platform.

“We were just focused on getting more courses up,” she explains. This simplicity is a gift to the creative mind: you’re given one pinpointed barrier to break through and forced to focus all your creativity around that one goal. If you’re feeling overwhelmed with community measurement, narrow your focus down to just one goal to start then optimize toward that.

Now things have moved to the next level. “In the past year, we’ve narrowed in our focus, which is dovetailing with the company itself maturing,” says Eliza. The community team is now focused on determining how community engagement impacts overall course quality, which is a more complex goal to measure towards. The groundwork laid during this first phase allowed them to get up to speed quickly.

2. Collaborate to get the data you need.

So how does the Udemy community team measure their impact on this more complex business priority?

They don’t do it alone. They work with their data engineering team to get all the data they need.

While many community managers don’t know how to run SQL queries, they have the power to be a community professionals’ best friend. In many companies, a data engineering team or BI analyst holds the keys to the SQL castle.

“This year, we’ve worked with [them] to work out a framework to track minutely the engagement that’s going on in our groups. We can see daily how new people are doing, for example.”

In order to set up a measurement strategy, you have to know what your data sources will be. Will the data come from your BI team? Your community platform dashboard? A third-party analytics tool?

3. Set your community engagement metrics.

With access to all this data, Eliza carved out daily and monthly metrics that the team uses to ensure their community is healthy. When they run their community engagement experiments, they can then track how they move the needle on the health of the community, and how that health impacts the business’s priorities.

Daily Metrics:

• How many posts have been added in the community?
• How many comments have been created in reaction to the community’s posts?
• How many comments come from new community members (an indicator of engagement of their “fresh talent”)?

Monthly Metrics:

Monthly, the team measures two simple metrics:

• On an aggregate level, how many people joined the community?
• How many participated?

Armed with this basic information, they’re then able to show how this community health impacts business health.

Caveat: Some things may remain manual.

While Eliza works hard to ensure that her team is able to get aggregate data on community health, there are some measurements she has yet to codify. For instance, she is now working out how to move the community-product feedback process into a more automated cycle.

“For every new product release, we keep track of which instructors are passionate about which products. We keep ongoing lists so when we release an update to our announcement tool, we have a group to talk to on the phone.” Currently, there is no automated way to do this, and Eliza isn’t even sure it would be a good thing if there were.

“A lot of it stems from the balance between online and offline interaction. Every month, someone on the team – either I or Lindsey Bonner [Udemy’s Instructor Community Senior Associate] – will reach out to top contributors. We send a personal email or make a phone call. These people are spending a lot of time providing a lot of value for us, and we want to give them attention.”

If you’re struggling with manual processes, and wishing you had more time to do one-to-one giving, know you’re not alone: “This is something I wish we did a ton more of. We are still figuring out the balance. There’s continual asking of ‘How do we do this more?’”

4. Give context: Create reports around community engagement metrics and business objectives.

Reporting

Once the team has collected the community health metrics, Eliza reports to Udemy executives once per month on how the community is building business value. This is where community health and business objectives intersect.

“On a monthly basis, we share how engagement in the group is going. Is it increasing? Are there more people participating? One thing we spend time on is focusing on the tenor of the conversation. It’s the biggest window we have into how the instructors feel.”

She also now reports more to other teams laterally in order to get a community perspective into multiple departments. “We’ll work with design. We will send instructors to the designers to talk to them about how they feel about the look and feel of new releases. Or we’ll work with marketing. We’ll let marketing know how instructors are responding to promotions.”

All in all, it’s a collaborative effort and Eliza’s role grew from outward-facing to more inward-facing as the community grew, evangelizing for the members in every department.

Business Objectives

“As part of the instructor team, we have yearly goals. Our monthly goals feed into yearly goals. And if we didn’t have the day-to-day information, it would be hard to put an actual number on engagement and establish those monthly and yearly goals. So it’s all important.”

“On a quarterly basis, our goals also center around sentiment. How happy are our members? We track that through surveys we send via email. We also track Facebook group sentiment using a data framework internally.”

“Because of all that work, we were able to set concrete goals and track specific things with specific instructors. We’re able to track new instructors posting. With 20,000 members, we’d miss welcoming new people before, and with 35,000, you can only imagine. Now we have a system around it, and it allows us to focus our energy in the correct places.”

5. Evangelize, evangelize, evangelize (internally).

When you’re doing all this work, it’s important to lift your head up on a regular basis and share wins internally. At Udemy, their team is growing so fast that they’ve struggled to let new hires know about the community’s impact.

“Our company has gotten a lot bigger. We almost doubled this year. To broadcast the voice of instructors, you need a loud voice. So my role is very much bringing that voice of the instructor community to every aspect of the company. As things have gotten bigger, there are more people to talk to and there is more evangelizing to do.”

A Note: As You Grow, Remember Who Got You There

“I’m continually impressed by how helpful and knowledgeable our members are,” says Eliza. “They know every single thing about our product. That’s valuable.”

“The community team has been the face of the organization to the instructors. In the future, we are going to become more of the gatekeepers to the organization to the community, so that they have a direct impact on how we work.”

“We will listen to what people are saying and be able to say, ‘Here is how we are going to solve your problem.’ We want to be able to say, ‘We’re here. We’re listening to what you’re saying.’”

Over the course of the next year, the team will continue to refine the way in which they derive business value from community, and they’ll do it across the company, from PR to design to HR to marketing. As Eliza moves into a communications-focused role and the Instructor Experience team matures, so too will their measurement strategy.