The purpose of evaluation
A key question for anyone managing or facilitating an online community is how to make it sustainable. Sustainable communities need to maintain (and grow) an active and engaged membership. Structured online events or activities can play a very important role in engaging community members and ensuring their regular and active involvement. However, these events must be effective – badly planned and/or facilitated events can turn members off and lead to the failure of the community.
The facilitator’s overall impression of whether an event is effective or not is useful. But without a more rigorous evaluation, we can miss underlying issues which have the potential to damage members’ ongoing engagement in the community. So some kind of evaluation process is vital to its ongoing success and sustainability.
This article focuses on evaluating community events: not to check that they meet a minimum standard, but as a way of engaging in a process of ‘continuous improvement’.
The evaluation process
 |
Many educational and management approaches employ cyclical models involving reflection or evaluation – eg Kolb’s experiential learning cycle and the action research cycle. The PDSA Cycle shown here represents the stages of Deming’s approach to quality improvement in business, but can be adapted to provide a model useful in relation to online community events and activities:
|
Carrying out the evaluation
Most of the evaluation will take place as part of the STUDY phase. But during the event (the DO phase) the facilitator should keep notes on what’s going well and what isn’t. If it’s not too big a group, it can also be really helpful to keep notes on the level of engagement of each member – not as a form of assessment, but as data that may be useful later.
During the STUDY phase:
Decide how you will gather feedback
Without its members, the community does not exist. So feedback from participants in an event is an essential component of evaluation:
- Online tools such as surveymonkey or polldaddy are quick to complete, so community members are usually happy to complete a survey. But make the survey brief, and tell them how long it’ll take.
- Individual interviews: you may get much more meaningful information about the event’s effectiveness if you personally approach participants. This may be a phone call or skype, or using an asynchronous method such as email. Ideally, a neutral 3rd party will gather the data, since participants may be unwilling to open up to the facilitator. You may be able to set up a reciprocal arrangement with another facilitator to gather data from each other’s participants.
- Collaborative feedback: you could set up a wiki that participants can use to record their feedback. Or you could set up a forum or a synchronous discussion space that participants use to discuss the event. Ideally this would be anonymous, so you may appoint one member to gather the raw data and provide you with a summary. Again, you may be able to arrange with another facilitator to manage this and gather feedback.
Develop evaluation questions and tools
- Communicate clearly what you are evaluating – point the participants back to the activity if possible so they can reflect on it. Be specific – make sure the questions clearly identify aspects of the activity that you want feedback on.
- Focus on how well the activity met their needs, not just on how much they enjoyed the process. If possible, ask for feedback at the higher levels of Kirkpatrick’s model – eg has the activity had a positive result, has it made a difference, have they been able to apply what they learned during the activity?
- Focus separately on the design of the activity and its facilitation: they are distinct, and the effectiveness of each is essential.
- Include questions that are open and qualitative so you can find out why things happened the way they did.
Gather feedback from community members.
- Tell participants how you will make use of the feedback: if you focus on improvement and they believe you are sincerely interested in making things better next time, they are more likely to engage in the evaluation process.
- Gather feedback not just from those that took part in the event, but also from those who chose not to. Asking those who didn’t take part why they didn’t can tell you a lot about the design of the event and the way it was communicated to community members!
Reflect and evaluate
Feedback from participants is just one form of information on which to base the evaluation. Your own reflection is another essential component.
- You may want to reflect on the activity before you gather data from participants – that way your own thoughts won’t be overly influenced by feedback. But once you have gathered the data, that’s a chance to reflect on and learn from the feedback from participants.
- In your reflection, avoid placing blame on participants. Assume you can do better and learn from mistakes. If people go wrong, ask yourself how could I have communicated more clearly?
- Use your own recollections and notes from during the activity to triangulate – compare them with what participants have told you about the activity, what happened and why.
- Focus on improvement – even where you think the activity was effective, try to identify specific things you could do to make it better next time.
- Share your evaluation with the participants – it’s a community, right? Sharing the evaluation means it becomes part of the larger community collaboration and conversation, and can enhance member commitment. But… if there are comments or conclusions that you feel are private or could offend members, leave them out of the published version.
Image: A Midnight Modern Conversation by William Hogarth.
Bibliography
Davies, C (nd). Kolb Learning Cycle Tutorial. Downloaded 30 May 2010 from http://www.ldu.leeds.ac.uk/ldu/sddu_multimedia/kolb/static_version.php
Unknown author, Business Performance Ltd (nd). Why Measure Training Effectiveness? Downloaded 30 May 2010 from http://www.businessperform.com/workplace-training/evaluating_training_effectiven.html
Unknown author, Carpenter group (nd). The Deming Cycle. Downloaded 30 May 2010 from http://www.quality-improvement-matters.com/deming-cycle.html
Unknown author, Warwick University (2008). Action Research. Downloaded 30 May 2010 from http://www2.warwick.ac.uk/services/ldc/resource/evaluation/tools/action/
pleft.wordpress.com 
