The real population boom – the over 60s: great new killer facts and graphics from Age International

February 4, 2015

Are developing countries heading for another debt crisis? And if so, what is anyone doing about it?

February 4, 2015

How do you keep 100 students awake on a Friday afternoon? Fast feedback and iterative adaptation seem to work

February 4, 2015
empty image
empty image

I wrote this post for the LSE’s International Development Department blog
sleeping students

There’s a character in a Moliere play who is surprised and delighted to learn that he has been speaking prose all his life without knowing it. I thought of him a couple of weeks into my new role as a part-time Professor in Practice in LSE’s International Development Department, when I realized I had been using ‘iterative adaptation’ to work out how best to keep 100+ Masters students awake and engaged for two hours last thing on a Friday afternoon.

The module is called ‘Research Themes in International Development’, a pretty vague topic which appears to be designed to allow lecturers to bang on about their research interests. I kicked off with a discussion on the nature and dilemmas of international NGOs, as I’m just writing a paper on that, then moved on to introduce some of the big themes of a forthcoming book on ‘How Change Happens’.

As an NGO type, I am committed to all things participatory, so ended lecture one getting the students to vote for their preferred guest speakers (no I’m not publishing the results). In order to find out how the lectures were going, I also introduced a weekly feedback form on the LSE intranet (thanks to LSE’s Lucy Pickles for sorting that out), and asked students to fill it in at the end of the session. The only incentive I could think of was to promise a satirical video (example below) if they stayed long enough to fill it in before rushing out the door – it seemed to work. The students were asked to rank presentation and content separately on a scale from ‘awful’ to ‘brilliant’, and then offer suggestions for improvements.

It’s anonymous, and not rigorous of course (self-selecting sample, disgruntled students likely to drop out in subsequent weeks etc), but it has been incredibly useful, especially the open-ended box for suggestions, which have been crammed full with useful content. The first week’s comments broadly asked for more participation, so week two included lots of breakout group discussions. The feedback then said, ‘we like the discussion, but all the unpacking after the groups where you ask what people were talking about eats up time, and anyway, we couldn’t hear half of it’, and asked for more rigour, so week three had more references to the literature, and 3 short discussion groups with minimal feedback – it felt odd, but seemed to work.

At this point, the penny dropped – I was putting into practice some of the messages of my week two lecture on how to work in complex systems, namely fast feedback loops that enable you to experiment, fail, tweak and try again in a repeat cycle until something reasonably successful emerges through trial and error. One example of failing faster – I tried out LSE’s online polling system, but found it was too slow (getting everyone to go online on their mobiles and then vote on a series of multiple choice questions) but also not as energising as getting people to vote analogue style (i.e. raising their hands). The important thing is getting weekly feedback and responding to it, rather than waiting til the end of term (by which time it will be too late).

LSE content weeks 1-3The form is not the only feedback system of course. As any teacher knows, talking to a roomful of people inevitably involves pretty intense realtime feedback too – you feel the energy rise and fall, see people glazing over or getting interested etc. What’s interesting is being able to triangulate between what I thought was happening in the room/students’ heads, and what they subsequently said. Broad agreement, but the feedback suggested their engagement was reassuringly consistent (see bar chart on content), whereas my perceptions seem to amplify it all into big peaks and troughs – what I thought was a disastrous second half of lecture two appears to have just been a bit below par for a small number of students.

The feedback also helps crystallize half-formed thoughts of your own. For example, several complained about the disruption of students leaving in the middle of the lecture, something I also had found rather unnerving. So I suggested that if people did need to leave early (it’s last thing on Friday after all), they should do so during the group discussions – much better.

What’s been striking is the look of mild alarm in the eyes of some of my LSE faculty colleagues, who warned against too much populist kowtowing to student demands. That’s certainly not how it’s felt so far. Here’s a typical comment ‘I think that this lecture on the role of the state tried to take on too much. This is an area that we have discussed extensively. I think it would have been more useful to focus on a particular aspect, perhaps failed and conflict-affected states since you argue that those are the future of aid’. Not a plea for more funny videos (though there have been a few of those), but a reminder to check for duplication with other modules, and a useful guide to improving next year’s lectures.

What is also emerging, again in a pleasingly unintended way, is a sense that we are designing this course together and the students seem to appreciate that  (I refuse to use the awful word co-create. Doh.) Matt Andrews calls this process ‘Problem Driven Iterative Adaptation’ – I would love to hear from other lecturers on more ways to use fast feedback to sharpen up teaching practices.

And now of course it’s over to the students themselves to say what’s really been going on…..

P.S. Nice distinction made by LSE’s Jean-Paul Faguet in a discussion of this post. End of term evaluation is best for getting feedback on content, as by then students have a picture of the whole course, and the bright sparks can feed back on what was missing. Realtime feedback is most use for adapting the format/presentation as you go along.

24 comments

  1. Absolutely concur about the importance of PDIA.

    At the LSE we also have teaching surveys at the end of each term, where students rate each teacher across fifteen indicators and comment on areas for improvement. Each teacher will then receive this quantitative and qualitative data for each course they teach upon. The report also shows how one fares relative to other lecturers on the quantitative indicators.

    One limitation of this feedback mechanism, however, is that it is after ten weeks of teaching. Whereas anyone whose read Matt Andrews’ work knows that PDIA requires feedback early on.

    So I do two things:

    (1) Ask for anonymous feedback scrawled on a scrap of paper in week 3: students write what they like, don’t like and what could be improved.

    (2) Listen to immediate, natural feedback. This is possible if you facilitate participatory lectures. If hands are raised, questions posed, objections made then you know immediately what students think. Personally, I never talk for more than ten minutes. I pose questions mid-lecture to students, identifying them by name – by using the register. This IS intimidating at first: getting people to think on the spot. But, in my experience, once I have mandated participation, it quickly becomes normalised, catalysing voluntary participation.

    Decentring authority encourages students to become more critical during the lectures and then you get to know what they think of the arguments. If they have misunderstood, points can be clarified; if they are interested in approaching the topic from another perspective, you can engage with that there and then.

    Win-win. PDIA in action.

    1. You have a register?! Interesting Alice, I also found that students were initially reluctant to participate, then once they got into the groove, wouldn’t shut up…… (By week two I was getting feedback saying less participation, more lecture please!)

  2. Yes, all courses have registers of enrolled students.

    Wow, have never heard requests for less participation. Interesting. For me, the challenge is how best to widen participation so it’s not just the ordinarily vocal ones. How best to create a space in which quieter people feel confident to engage.

  3. Duncan, interesting to reflect on the factors that make it work- the students have an incentive to tell you what they really think as it is something they are paying to do and are already showing commitment by turning up on a friday pm.

    You have a structured framework for engaging with them as a tutor and a shared understanding of what your respective roles are. They are feeding back directly to the decision maker who can make the changes fairly directly and it is quickly obvious if you have taken them seriously or not thus encouraging further feedback. I think it demonstrates its the quality of the relationship you have that matters as much as the mechanism. I wonder how much more difficult this is in programming with many more intermediaries, a lack of a shared understanding of roles and what each party wants and many more problems in how to get getting the necessary feedback.

  4. The most eye-opening lecturing experience I have had was teaching a “flip lecture” – I was asked to try it out when teaching a one-hour introduction to economics for UCL’s Arts-Sciences undergraduate degree. The core idea is to flip the use of the time: students should get familiar with the material in advance so the hour together can be used for something more creative and interactive.

    Several weeks in advance of the lecture, I recorded 3 x 10 minute videos of myself presenting the material in powerpoint (software like Camtasia is great for this). The students were required to watch these and pose questions to me in advance – and then to favourite each others’ questions. Then, 24 hours before the face-to-face session (“the lecture”), I received a set of around 100 questions that all the students had posed, and these were ranked by their popularity (which was really interesting in itself). I clustered their questions so I had 5-6 themes and spent the live hour we had together answering and discussing those – inviting students by name to pose again the question that they had submitted on line.

    It was an outstanding experience in three respects:
    1) I have never seen a room of such engaged students, eager to know the answers to their well-thought through questions.
    2) the questions they were asking me to answer were far more profound and interesting than any questions I recall coming up when I was sitting in introductory lectures as a student
    3) their questions pushed me to come up with new insights and say things that I had never said or thought through before. That’s co-creation, whether or not you like the word!
    So I am really impressed by flip lecturing, and recommend it – though, yes, it requires more preparation. Here’s a blog on it by the course director at UCL, Carl Gombrich, who asked me to try it out:
    http://www.carlgombrich.org/flipping-lectures-reflections-on-a-term-of-learning/

    1. Sounds amazing Kate, will definitely see if I can use this in some way, although given pressures on LSE students, not sure if this level of prep will work there.

      1. Actually the real additional prep is on the part of the lecturer – all the pre-recorded videos. For students, it’s 30 mins watching vids in advance plus some question-thinking time – that’s equivalent to just one less reading in advance of the lecture. So for them I think it is excellent substitution of time use. It’s essentially more work for the teacher – which is one of the real reasons why it is not done more widely. But I reckon it is a trend that will grow and grow.

  5. As a regular trainer and facilitator, I use a very low tech version of this… at the end of each day I ask participants to take two post its and draw a smiley face on one and the things that they found useful/interesting/thought-provoking and then on the second draw a frowny face and write what they would like to hear more about/what needs to change/anything they didn’t like/any questions. It is really helpful for refining the following day and I always include a short feedback on how I have adapted/responded when I start to show people the benefit/purpose of the exercise. I think this element is important as people like to see the what the benefit is of providing feedback or they lose interest. This is important for broader application of PDIA as many communities provide feedback but nothing changes as the development model is not sufficiently flexible to support a quick response.

  6. I really like the sound of the “flip lecture” described by Kate. It seems to me that this approach should be the norm in education now, rather than the exception.

    1. Owen, I agree, once you see it in action, it becomes obvious that this should be the norm. And *you* are already really well set up to do a flip lecture: students could watch your (excellent) Complexity and Development video, submit questions to you in advance, then you turn up and off you go. I’ll bet that would be a great one hour’s discussion. In fact I suggest anyone involved in a development degree course who is reading this should jump in right now invite Owen to give his first ever flip lecture….

      1. well that would be me, as Owen is guest lecturer in my slot in a couple of weeks – Owen, want to give it a go? If so I can tell them this Friday, and get them to send in Qs following week

        1. Ok, well Owen was willing to give it a go, but in the end we couldn’t be sure that enough of the students could commit to watching quite a long video (45m) in advance, so it didn’t work out this term – will definitely include it in the design next time tho.

  7. Great post. I also employ these kinds of tools in my teaching. I do a few things: first, I teach using multiple methods (as Duncan describes) so that I am actively experimenting with ways of engagement ; second, I look for informal feedback from groups of 5 to 10 students after each class, targeting them based on their engagement in class (a purposeful sample including the engaged and the disengaged) , third, I use formal feedback mechanisms like the response cards etc ( though I find the comments here often clash, with different students indicating a preference for different things and me not knowing why they made their comments…). I also do mid term reviews and end of term reviews, which are not so useful for pdia. Fourth, I actively adapt methods to comments, as much as possible or I explain why an unpopular method is my chosen method and therefore will bit be adapted (there are teaching methods I adopt that are there for specific pedagogical aims…that sometimes don’t work but can’t be changed mis stream) . I love flipped classrooms and experiential sessions but find a good proportion of most of my classes prefers traditional class structure. Here are my two parting points : I think teaching is really hard and you need to get feedback and adapt, but you also need to recognize that sometimes your methods etc won’t be popular but may still be valid and the feedback you get may be more about popularity…so you need to be careful…adapt to deliver a better product and facilitate more learning, not just to keep students in the room (function over form)… last thing to say: I don’t always do this. If I’ve been learning and adapting the same course for five years I so less adaptation now… which is also a hallmark if effective pdia, I think (the purposeful muddling isn’t meant to last forever)

  8. Woah. Lots of PDIA. But Matt, how do you get a purposeful sample of disengaged and engaged students to stay after class to provide feedback? Do you just ask them by name to stay for a bit? Aren’t they/ you rushing to the next timetabled class? Moreover, do the disengaged ones really publicly share why they are disengaged? Also, there are many other reasons for quietness, it may not signal disengagement.

    I also want to second Matt’s point about not being entirely led by student feedback, at least not immediately. Students often find the first lecture, seminar or class with me quite ‘intimidating’ but subsequently express great support for my methods. If I solicited feedback early on and changed my style to be more popular then this would have impeded their learning. I guess the point is to give students a couple of weeks to review and reflect before making changes.

  9. I usually find the students informally after class, at lunch or I email them and see if we can grab a short chat. I find there are many reasons for engagement and disengagement and all learning is useful. I find my students very willing and confident to tell me where I’m not doing well (maybe it’s a Harvard thing but I don’t find my students are hindered by talking to me…but I could be wrong…I like talking to them so I can ask why why why and really understand their point
    ..aomething I find hard from written comments).

  10. Kudos on bringing rapid feedback to bear. It’s a great idea. Wish I’d thought of it!

    I personally love the flipped approach. It’s not the norm yet, except in the more experimental schools/programs in the US, which is where I’ve taught.

    It works really well when you balance the expert info/engagement that students like from adjunct faculty/practitioner experts/guest lecturers while optimizing in-class time for meaningful interaction and experiential learning that deepens the learning experience and outcomes for everyone.

    Two notes:

    1. It doesn’t have to be you as the lecturer recording all the pre-class material. There’s a lot of pre-recorded materials available that you can draw on and supplement as needed.

    2. The key to success is the design of the interactive sessions with students. There’s lots of ways to do this depending on your purpose and content.

  11. A brilliant blog, Duncan and some excellent added comments. I am always looking for ways to make my International Development sessions more participatory and will give the flip lecture idea a try. It would be very helpful to learn of other approaches that are used. How, for example, would colleagues deal with topics such as development theory in a participatory way?

    1. Kevin,

      There are likely many ways to build more interactive methods into classes on development theory. Start by reflecting on your the objectives and goals of the course. Consider what is it important for students to learn, in terms of the concepts, theories etc as well as in terms of practical skills. Why is this important, and how might content delivery and in-class engagement/interaction be optimized to best realize your objectives and goals? Then get creative…

      Also, I’ve found, that it is important to stimulate student’s curiosity and interest. Provocative and/or wicked questions are good for this, as is posing scenarios – i.e. put the student in a hypothetical (but realistic) scenario, where they have to develop and advocate for a particular strategy live in class. This shifts the student mindset from being delivered content/spooned, to one of wanting to research and explore in order to be able to do something that they might aspire to do in the real world…

      Hope that helps. I’m happy to chat through this briefly if it would help.

      Rashmir.

  12. Hi Duncan, I am one of your students and having been to all of your lectures so far, I have really appreciated the way you’ve tailored the style and format to our feedback. Friends and I have commented how refreshing (and endearing) it is to have a lecturer who is so thoroughly committed to innovative teaching and not simply assuming their content is compelling enough to keep people interested. Plus being a part of the instant feedback loop is extremely entertaining! Keep it up, we really appreciate it. Also don’t be disheartened that numbers are dwindling, it’s just because people are freaking out about consultancy projects and dissertations and the class isn’t assessed, plus a lot of people don’t have any other class to come in for on Fridays. Everyone who does show up is really into it.

    1. Great post Tasneem, and the distinction between being democratic on process, but more directive on content, sounds about right – although the economics students currently in revolt against their pre-2008 curriculum might disagree!

  13. Thank you for the quick response Duncan.
    Well indeed, I imagine if the course content then was a monoculture of readings/ideas (which often happens in economics), then protest away!
    We look forward to reading more of your blog!

Leave a comment

Translate »