Member surveys with AI: writing questions and reading the answers

Surveys are one of the few ways to hear from members who never post, never email and never come to live sessions. Yet many owners avoid them, or send one and never quite get around to reading the answers. Writing good questions is harder than it looks, and a few hundred free-text replies can feel impossible to make sense of.

AI helps at both ends. It can critique your questions before you send them, catching the small flaws that produce misleading answers, and it can sort open answers into themes in minutes. What it cannot do is decide what matters. That part, and the checking in between, stays with you.

Start with the decision, not the questions

The most common survey mistake is asking things you are merely curious about. Every question should connect to a decision you are willing to make. Before writing anything, finish this sentence: “After this survey, I will decide whether to…”

A hypothetical community choir association called Many Voices wanted to decide whether to move its monthly workshops from weekday evenings to weekend mornings and whether to add recorded sessions. Those two decisions shaped every question it asked. Anything unrelated was cut.

Draft questions with AI, then ask it to critique them

AI is good at producing a first draft, but it is even more useful as a critic. Ask an assistant such as ChatGPT, Claude, Gemini or Microsoft Copilot to review questions you have written, or ones it has drafted:

I am surveying members of [describe your membership] to help me decide [describe the decision]. Here are my draft questions: [paste questions]. Review each one and flag any that are leading, ask about two things at once, use jargon members may not know, or have answer options that overlap or leave people out. Suggest a clearer version of each flagged question. Then tell me which questions do not help with my decision and could be cut. Keep the survey under [8] questions.

A few terms worth knowing. A leading question nudges people toward an answer, such as “How much do you enjoy our new workshops?” A double-barreled question asks two things at once, such as “Was the session useful and well organized?” Members can only give one answer, so you cannot tell which part they meant. A good AI critique catches both and offers neutral rewrites such as “How useful was the last workshop you attended?” with a clear scale.

Include at least one open question, such as “What is one thing that would make your membership more valuable to you?” These produce the most useful insights, and they are where AI helps most later.

Before sending, ask two or three members to take the survey and tell you if anything confused them. For more on timing and length, see surveying your members without annoying them.

Remove identifying details before analysis

When the answers come in, export them and remove anything that identifies a person: names, email addresses, member numbers and any details inside free-text answers that could point to someone, such as “as the only dentist in our local chapter.” Replace each respondent with a code if you need to keep answers together.

Use a paid business plan for this work and check the data and training settings so your uploads are not used to train the model. Even anonymized feedback is your members' words and deserves care. Protecting member privacy when you use AI tools goes into more depth.

Theme the open answers with AI

With a clean file, ask the AI to read the open answers and group them:

Here are anonymized answers to the question “[paste the question]” from members of [describe your membership]: [paste answers or upload the file]. Group the answers into themes. For each theme, give a short name, a one-sentence description, the number of answers that fit it, and two representative quotes copied exactly from the answers. Put answers that fit no theme in an Other group. Do not invent quotes, and do not merge themes that are clearly different.

A good result looks like a short table: “Recordings for missed sessions, 41 answers,” followed by two real quotes. Many Voices discovered that weekend mornings were less important to members than simply having recordings, which changed its plans entirely.

For multiple-choice questions, the numbers are easier to calculate in a spreadsheet than to trust from an AI summary. If you need formulas or charts, AI for spreadsheets shows how to get them without sharing raw data.

Check the themes before you act

AI can miscount, merge distinct ideas or quietly drop the unusual answers that often matter most. So check its work:

• Search your file for two or three quotes and confirm they are real and exact.
• Read at least twenty answers yourself and see whether the themes match your impression.
• Look through the Other group; this is where surprising ideas hide.
• Compare themes across groups, such as new versus long-standing members, if your data allows.

Treat the themes as a guide to where to look, not a final verdict. A theme with many answers is worth serious attention, but a single thoughtful answer can still point to a real problem.

Close the loop with members

Members who take a survey want to know it made a difference. Write a short summary of what you heard and what you will do about it. AI can draft this from your notes, but edit it so it sounds like you and makes only promises you will keep. Many Voices shared a three-point update and started recording workshops the following month, and its next survey received more replies.

Your survey checklist

1. Write down the decision the survey will inform.
2. Draft up to eight questions, including at least one open question.
3. Use the critique prompt to catch leading and double-barreled questions.
4. Test the survey with two or three members.
5. Remove identifying details from the answers before any AI analysis.
6. Theme the open answers with AI and check quotes and counts yourself.
7. Share what you learned and what you will change.