Survey answers and behavior data: using both

Ask members what they want and they will tell you. Look at what they actually use and you will sometimes find a different story. Members say they want more live workshops, then do not attend. They say the content library is overwhelming, then use it every week. They rate a course highly and never finish it.

This does not mean members are unreliable. It means surveys and behavior data measure different things. Surveys capture intentions, feelings and reasons. Behavior data captures actions. Relying on either one alone leads to mistakes. Used together, they give you a far more accurate picture of what your members value and why.

What behavior data is good at

Behavior data is anything your membership platform or analytics records about what members do: logins, lessons completed, downloads, forum posts, event attendance, cancellations, upgrades.

• It is honest about actions. Nobody over-reports their attendance in your attendance records.
• It covers everyone, not just the members who answer surveys.
• It shows change over time, often before members themselves would notice or mention it.

But behavior data goes quiet on the most important question: why. It shows that members stopped attending live sessions, not whether the time slot stopped suiting them, the topics lost relevance or the recordings are simply good enough. It also cannot tell you about things members want that do not exist yet.

What surveys are good at

Surveys, interviews and feedback forms capture what members think and feel.

• They reveal reasons, motivations and frustrations.
• They surface unmet needs and ideas for new content.
• They show how members describe their problems in their own words, which is gold for your marketing.

Surveys also mislead in predictable ways. People answer as the member they would like to be: the one with time for weekly workshops. They are poor at predicting their own future behavior. The members who answer are often your most engaged, so quieter members are underrepresented. And the wording of a question can steer the answer. The practical side of writing good surveys is covered in surveying your members without annoying them.

Ask questions that connect to behavior

The easiest way to make surveys more reliable is to ask about specific, recent behavior rather than hypothetical preferences. Compare these pairs:

Weaker: Would you like more live workshops?

Stronger: In the past month, which of these did you use? (live workshops, workshop replays, project videos, the forum, none of these)

Weaker: How satisfied are you with the membership?

Stronger: What is the last thing you made or finished using something from the membership?

Weaker: What content should we add?

Stronger: What were you trying to do the last time you could not find what you needed here?

Specific questions produce answers you can check against your data, and they produce more concrete ideas.

Put the two side by side

The real power comes from linking survey answers to the behavior of the people who gave them. If your survey is not anonymous, or members choose to include their email, you can group responses by what those members actually do. Tell members plainly if you will do this.

Consider a hypothetical online woodworking school. A survey of 120 members produces a clear message: 80 of them say they want more advanced projects. The owner is ready to film a series of complex furniture builds.

Then she checks the behavior data. The existing advanced projects have the lowest completion in the library; members start them and stop partway through. Grouping the 80 answers by behavior, she finds that only 30 of those asking for advanced content have completed even one intermediate project. The other 50 want advanced work but have not finished the steps that lead to it.

A few short follow-up conversations reveal the missing piece. Members want to learn advanced techniques, but the existing advanced projects take twenty hours and need tools most of them do not own. What they really want is advanced skills in shorter projects with ordinary tools. Neither the survey nor the data alone would have shown that.

When they disagree

• For decisions about what members will use, lean on behavior. If members say they want something but do not use the version you already have, find out why before making more.
• For decisions about why, lean on what members tell you, then check it. Reasons for cancelling come from asking, but you can confirm them by looking at what those members did before they left.
• For new ideas, test small. If members ask for something new, run a small version, such as a single workshop or a pilot series, and watch what people do before investing heavily.
• Treat the gap itself as information. A big difference between what members say and what they do often points to an obstacle: time, difficulty, format or findability.

Next steps

1. List two or three open questions about your members, such as why live attendance is falling or what content to make next.
2. For each, write down what behavior data could show and what only members can tell you.
3. Rewrite your next survey to ask about specific, recent behavior.
4. Where members are willing, connect their answers to their behavior and compare groups.
5. When the two disagree, hold a handful of short conversations to find the missing reason.
6. Tell members what you learned and what you will change, following the advice in acting on member feedback and closing the loop.