Reflective practice: the need for open questions

Use of open questions in (e)learning

Open questions are more likely to lead to in-depth discussion in face-to-face or online learning contexts than closed questions. A simplistic example:

• “Is A better than B?” (closed) invites just a yes or no answer.
• “What makes A better than B?” (open) invites the learner to justify and explore the rationale behind their decision.

This is not to say that closed questions should never be used, just that their use should be limited – in the above example, the two questions could provide a useful sequence for initiating discussion.

Use of open questions in reflection

Similarly, when encouraging teachers to reflect on their own practice, we need to encourage them to consider and respond to open questions. A reliance on closed questions tends to discourage in-depth reflection. Why? Because in-depth reflection requires consideration of causes and significance of events, and open questions are required to really explore these.

In a recent project, I developed an app which allows the user to reflect on, create and store responses to reflective prompts (questions) relating to e-learning practice. The questions are drawn from one section of the eLearning Guidelines (eLG).

During the development process, I decided to make small adjustments to the wording of many of the questions to make them more open questions. For example:

• The original question “Do learners have the opportunity to self-assess their readiness for eLearning?” seemed to invite a simple yes/no answer, so it was reworded as “What opportunities do learners have to self-assess their readiness for eLearning?”

• Likewise, “Is it clear at the start of the course what support teaching staff will offer learners?” was reworded as “How is it made clear at the start of the course what support teaching staff will offer learners?”

I feel these open versions of the questions will encourage teachers to review the evidence and reflect on effectiveness more deeply.

Whether or not you are currently using the eLearning Guidelines, you might find the app a useful tool for reflecting on your e-learning practice. The Android app is free and is available to download from the apps page.

Connectivism: why I’m a skeptic

Is connectivism a theory? I guess. But when considering its usefulness to my own teaching and learning, I have reservations.

Here are 3 reasons I still have doubts about the value of connectivism as a theoretical construct:

1. Yes, at a micro level the neurological processes of thinking and learning involve connections within networks. And yes, at a macro level as individuals we are connected to a variety of networks for sharing information. These are useful and informative parallels but there is no evidence that one is more than an analogy for the other. Because network connections are required at the micro level does not mean that they are necessarily a pre-requisite of learning at the macro level. There is a temptation to use one as an analogy of the other, but this seems likely to be an over-simplification.
2. Connectivism is overly focused on learning as managing information: ‘… connectivism is the thesis that knowledge is distributed across a network of connections, and therefore that learning consists of the ability to construct and traverse those networks.’ (‘Connectivism’ and Connective Knowledge) If we visualise this at the macro level, the successful learner would appear to be little more than an effective navigator of information networks. For learning in a purely theoretical context, that might be fine. But as a professional developer, I’m more concerned with developing capability than knowledge.
3. Connectivism does not adequately build on the theoretical constructs I have found useful in my own teaching and learning. It’s not that every theory has to explain every event, but connectivism seems to have inadequate room for concepts such as reflective practice or higher levels of thinking inherent in models such as Bloom’s taxonomy.

Image: Tyramide filled neurons from the cingulate cortex of mouse brain by Neurollero

Evaluating online community activities

The purpose of evaluation

A key question for anyone managing or facilitating an online community is how to make it sustainable. Sustainable communities need to maintain (and grow) an active and engaged membership. Structured online events or activities can play a very important role in engaging community members and ensuring their regular and active involvement. However, these events must be effective – badly planned and/or facilitated events can turn members off and lead to the failure of the community.

The facilitator’s overall impression of whether an event is effective or not is useful. But without a more rigorous evaluation, we can miss underlying issues which have the potential to damage members’ ongoing engagement in the community. So some kind of evaluation process is vital to its ongoing success and sustainability.

This article focuses on evaluating community events: not to check that they meet a minimum standard, but as a way of engaging in a process of ‘continuous improvement’.

The evaluation process

Many educational and management approaches employ cyclical models involving reflection or evaluation – eg Kolb’s experiential learning cycle and the action research cycle. The PDSA Cycle shown here represents the stages of Deming’s approach to quality improvement in business, but can be adapted to provide a model useful in relation to online community events and activities: PLAN: plan the community event DO: facilitate the community event STUDY: evaluate the event ACT: feed the evaluation results back into further community development

Carrying out the evaluation

Most of the evaluation will take place as part of the STUDY phase. But during the event (the DO phase) the facilitator should keep notes on what’s going well and what isn’t. If it’s not too big a group, it can also be really helpful to keep notes on the level of engagement of each member – not as a form of assessment, but as data that may be useful later.

During the STUDY phase:

Decide how you will gather feedback

Without its members, the community does not exist. So feedback from participants in an event is an essential component of evaluation:

• Online tools such as surveymonkey or polldaddy are quick to complete, so community members are usually happy to complete a survey. But make the survey brief, and tell them how long it’ll take.
• Individual interviews: you may get much more meaningful information about the event’s effectiveness if you personally approach participants. This may be a phone call or skype, or using an asynchronous method such as email. Ideally, a neutral 3rd party will gather the data, since participants may be unwilling to open up to the facilitator. You may be able to set up a reciprocal arrangement with another facilitator to gather data from each other’s participants.
• Collaborative feedback: you could set up a wiki that participants can use to record their feedback. Or you could set up a forum or a synchronous discussion space that participants use to discuss the event. Ideally this would be anonymous, so you may appoint one member to gather the raw data and provide you with a summary. Again, you may be able to arrange with another facilitator to manage this and gather feedback.

Develop evaluation questions and tools

• Communicate clearly what you are evaluating – point the participants back to the activity if possible so they can reflect on it. Be specific – make sure the questions clearly identify aspects of the activity that you want feedback on.
• Focus on how well the activity met their needs, not just on how much they enjoyed the process. If possible, ask for feedback at the higher levels of Kirkpatrick’s model – eg has the activity had a positive result, has it made a difference, have they been able to apply what they learned during the activity?
• Focus separately on the design of the activity and its facilitation: they are distinct, and the effectiveness of each is essential.
• Include questions that are open and qualitative so you can find out why things happened the way they did.

Gather feedback from community members.

• Tell participants how you will make use of the feedback: if you focus on improvement and they believe you are sincerely interested in making things better next time, they are more likely to engage in the evaluation process.
• Gather feedback not just from those that took part in the event, but also from those who chose not to. Asking those who didn’t take part why they didn’t can tell you a lot about the design of the event and the way it was communicated to community members!

Reflect and evaluate
Feedback from participants is just one form of information on which to base the evaluation. Your own reflection is another essential component.

• You may want to reflect on the activity before you gather data from participants – that way your own thoughts won’t be overly influenced by feedback. But once you have gathered the data, that’s a chance to reflect on and learn from the feedback from participants.
• In your reflection, avoid placing blame on participants. Assume you can do better and learn from mistakes. If people go wrong, ask yourself how could I have communicated more clearly?
• Use your own recollections and notes from during the activity to triangulate – compare them with what participants have told you about the activity, what happened and why.
• Focus on improvement – even where you think the activity was effective, try to identify specific things you could do to make it better next time.
• Share your evaluation with the participants – it’s a community, right? Sharing the evaluation means it becomes part of the larger community collaboration and conversation, and can enhance member commitment. But… if there are comments or conclusions that you feel are private or could offend members, leave them out of the published version.

Image: A Midnight Modern Conversation by William Hogarth.

Bibliography

Davies, C (nd). Kolb Learning Cycle Tutorial. Downloaded 30 May 2010 from http://www.ldu.leeds.ac.uk/ldu/sddu_multimedia/kolb/static_version.php

Unknown author, Business Performance Ltd (nd). Why Measure Training Effectiveness? Downloaded 30 May 2010 from http://www.businessperform.com/workplace-training/evaluating_training_effectiven.html

Unknown author, Carpenter group (nd). The Deming Cycle. Downloaded 30 May 2010 from http://www.quality-improvement-matters.com/deming-cycle.html

Unknown author, Warwick University (2008). Action Research. Downloaded 30 May 2010 from http://www2.warwick.ac.uk/services/ldc/resource/evaluation/tools/action/

Wikis: more than just collaboration

Definitions of wikis, especially in education, often state that wikis are ‘collaborative’. Most wiki software does support collaboration, but not all applications of wikis need to be collaborative. In fact, collaborative features can be detrimental if we want to publish our own writing and not have it changed or deleted… the reflective thinker may not want to be disturbed!

For example, I maintain a Mediawiki site for my own articles and other resources that I don’t want changed. It used to be an open wiki, but dealing with the spam became too time-consuming. So now the wiki is only open to be read by visitors, not for writing. I no longer feel the need to apologise for this – I don’t let others browse the documents on my hard disk, but I do let others browse (but not edit) the writing on my wiki. And there are plenty of other channels for collaboration out there.

Some wiki purists might say that using a wiki solely for your own writing, without allowing for input of others, is against the wiki philosophy. But I’d argue we should be able to use the tools in ways that best meet our needs, and the best tools provide flexibility in how we use them. And most of us have the need to write in different read-write modes: sometimes it’s private, sometimes it’s public, and sometimes it’s collaborative. The best wiki tools should let us easily write and manage documents in a range of read-write modes.

Ideally, Mediawiki would allow me to easily manage the read-write mode of any article and its associated discussion page. Unfortunately, it’s not that straightforward – permissions are set in the Mediaiwiki configuration file and the documentation warns against relying on the plugins available for finer-grained managing of permissions. So it hasn’t been feasible for me to effectively manage the read-write mode of individual pages on my main wiki.

Recently I’ve been trying out PMWiki, which makes much better allowance for controlling access to the wiki site and to individual pages within it. Any page can have a password for reading and a password for writing, and these can be set relatively simply. That means you have fine-grained control over the read-write mode of any specific page.

That’s just what I need – and I believe that’s what learners need too. Not all learning happens collaboratively: successful learners need to be reflective as well as collaborative. Effective Web 2.0 tools provide for personal reflection as well as more social approaches to learning.

Photo by Matan: Copy of Rodin’s ‘Thinker’ at Columbia University