Is Disruption a good thing? Let’s ask Southern Civil Society leaders for a change.

May 13, 2016

Links I Liked

May 13, 2016

How to read and comment on a draft paper – your suggestions please

May 13, 2016
empty image
empty image

Today’s vlog (I’ll be coming back to you in a few weeks to ask whether these are worth doing)

I spend a lot of time commenting on draft research and policy papers, both for Oxfam and beyond. So I put down some ideas Calvin on bad writingon how I approach it, got some great input from Oxfam Research Team colleagues, and now we want to ask you to chip. Then we’ll publish a revised version in our series of research guidelines. Although this is based on NGO and aid agency reports, I suspect a lot of this goes for academic papers too.

How to approach a draft: Please try and put yourself in the shoes of the target audience and think what would interest or inspire them. Don’t go into internal lobbyist mode, combing through the document looking for/shoehorning in references to your particular hobbyhorse!

I read the paper as I would speed read the final article: the exec sum first, then the conclusion, then the top and tail of each chapter. I try to focus my comments on those sections because they are (by far) the most important in terms of impact.

But you do need to read the whole thing. Is the paper internally consistent? Have any nuggets (killer facts, case studies, telling graphics, genuinely surprising or new findings) failed to make it to the exec sum – a common crime in NGO papers?

As a reader, you are also likely to be better placed than the author to point out places where some extra narrative would help. I call it ‘hand holding’ – suggesting linking text between paras to improve the flow, or explaining why a certain piece of analysis is significant and worth a reader’s time  – often underplayed by authors who are so obsessed with their subject, that they can’t imagine why anyone would find it boring!

Is the good stuff at the top? Let’s assume a large percentage of readers don’t read all the way through to the end. You need to make sure the best, most powerful ideas and arguments are up front, preferably in an executive summary, but if not, at the top of the paper. And then repeat them regularly throughout the text.

Tom Waits on bad writingRemember what the paper is for and don’t try and expand its remit: If possible, take a look at the original terms of reference for the paper. They should set out the audience and purpose. That can help you avoid giving the classic unhelpful comment ‘it’s too long, needs cuts, and here’s another 10 issues you need to cover’! Depending on whether it’s at the wonky end of the spectrum (research paper) or the more popular end (policy paper), there will be a different balance of the need for rigour and accessibility, but both matter in all cases.

Think about what is not there. This is difficult but can be really useful – it’s easy to critique what’s in front of you, but it is often more helpful to stand back and identify what is missing – in terms of arguments, approaches, or sources. As well as stepping back, try to look sideways – the author is likely to be a specialist, up to their neck in the detail of a particular subject. Is there anything they could usefully import (whether as content or analogy) from different issues or disciplines?

Style and Language matter: A personal bugbear – a lot of NGO papers are really badly written. Full of impenetrable jargon, deadened by the passive tense, and/or shrill in tone ‘the IMF must do X, Y, Z’. That turns off potential readers and greatly reducec a paper’s impact. Here’s some good advice on writing for impact. Dealing with bad writing is tricky, especially if the writer does not have English as a first language, but in my experience, people are usually grateful for specific suggestions and edits. If rewriting the whole paper is not feasible, concentrate on making the executive summary accessible.

How clear are the ‘so whats’? A common weakness is papers that are strong on diagnosis and exposition of the problem, but weak on what to do about it (someone once caricatured such work as ‘bad sxxt, facty, facty’ papers). Does the paper have specific, well argued suggestions for how to improve things or are the recommendations bland and generic? (see How to write the recommendations to a report on almost anything).

Feeding Back

Be kind: Once you’ve scribbled all over the paper, it’s time to feed back to the author. Take a deep breath. However brilliant/damning your critique, start by reminding yourself that there is a human being on the other end of this email. They have tried their best, even if the result needs work. You need to help them, not cast them into despair. Maybe talk to them face to face rather than just send an email?

Cue the infamous sxxt sandwich. Start off saying what you like about the paper, then what could be strengthened, but

Seems to be a word missing......

Seems to be a word missing……

finish off by stressing what is worthwhile. This needn’t be phoney – there is usually something to applaud in any piece or work. And even if people know what you are doing (a civil servant once told me ‘everything above the ‘but’ is bxxxxcks’), it still eases the pain and makes it easier to absorb and respond to criticism.

Be specific: Don’t just say ‘there’s some good stuff in here’. Give examples: a particular sentence that is nicely written, a particularly powerful paragraph, a quote judiciously used.  Knowing what works (and hence what to retain) can be helpful as well as good for (battered) authorial morale. Even more so when you suggest improvements. To a harassed author on a deadline, comments like ‘needs more on gender’, or ‘you should read Amartya Sen’ are more likely to drive them to self harm than write a better paper. Specific text changes only, if possible.

Receiving Feedback

Being on the receiving end of feedback can be bruising, especially when people don’t follow this kind of advice (I’ve got a few scars). If you feel your hackles rising, don’t get defensive, don’t hit ‘reply’. Probably worth sleeping on it before reacting. Remember that this is free consultancy, after all, and you get to choose which advice to respond to. If there are multiple commentators, so much the better – more advice, and easier to pick and choose what to take on board.

And yes, all this takes time, so it’s worth first making sure that the author really wants your comments, and getting them in ahead of deadline.

OK, over to you – what have we missed?

8 comments

  1. Nice tips.

    I would also suggest structuring feedback thematically. Rather than submit a long-winded list (a chronology of my musings), I try to identify three key important topics/ issues. (This is clearer and more manageable for the recipient). And then – within each of these issues – I try to provide specific, concrete, constructive suggestions as to how they might be addressed.

    Kindness matters, yes. One way of being kind – I think – is to phrase comments as questions. Instead of presuming that I know the answer and the author has idiotically not considered alternative perspectives/ readings/ methods/ issues, I might ask ‘I wonder what you think about X…?’. Or ‘do you think…?’. ‘I’d be interested in your thoughts on…?’. Same goes for conference and panels, right?

      1. One more thing.

        Mindset.

        Some people resent reviewing, as an unremunerated imposition on their time, distracting them from ‘their’ projects. That kind of mindset likely fosters a curt, blunt, potentially rude, or maybe rambling series of comments. Certainly unhelpful.

        I think it’s really important to recognise that reviewing another person’s work is contributes to the collective project. We’re not just working for ourselves, but trying to do something collectively. And it’s fun, intellectually, to consider and engage with someone else’s ideas, then contemplate ways in which they might be improved.

  2. Thank you for sharing your ideas on this, I especially support the executive summary, new stuff up front and the know your reader recommendations! From my side, some very basic thoughts on the use of evidence and ‘so-whats’ in papers from civil society (I’d offer different tips on academic papers), which sometimes reflect excitement about solutions without adequately building the case for what needs to change. Getting the evidence right seems a given, even if it’s not easy. We can always argue that we need more research (a very tired if true rec at the end of most papers…) but what specific evidence do we have for the things we find problematic and want changed? Do we have any evidence that something has been tried and (even better) works? Nothing should be in a summary or part of a set of asks if it has not been raised and examined in more detail in the paper itself — even if it’s addressed in the long bit very few will read. Finally, when we do offer solutions (the ‘so-whats’), the list of things we want done should indicate who we think needs to do them, so we know where to look for action/change/progress. Looking forward to others’ thoughts…

  3. These are very helpful, Duncan. I have really begun to appreciate the critical importance of spending a serious amount of time and thought on the Executive Summary and you’re so right that we often miss the critical messages up top.

    I would only add something about the ethics and negotiation around feedback and editing. I’ve been involved in some big organizational research projects where there were problems on both ends: leaning too hard or stepping too lightly. Editors who basically rewrote other’s papers without full negotiation and feedback. Or editors who failed to edit and the quality ended up being extremely poor and inconsistent. Finding the balance is tricky but so important if we believe that building research capacity is part of all of this. So critical to spend some time up front talking about what editing and feedback means and ultimately, a clear process for respecting authorship, determining quality and making decisions where there are differences.

  4. Interesting blog.

    I’ve written published books, produced screenplays, speeches given at Davos, Congress, The UN Security Council, Op-Eds, long form journal articles, and taught writing at NYU. But before all that, starting at age 24, I helped coordinate UN reports for the UN Security Council — trying to pass on comments from high level Secretariat officials who wanted everything written in UN-ese to the drafters: eight fiercely independent and grammatically uncoordinated field reps. I learned one thing. Never ask various parties for text. Only for facts. Put one author in charge. The boss who signs off on it. The relationship we are then talking about is how such a boss forms his personal scribe to draft for him better and better text over time. And there I think there is no better feedback than on-the-page edits. New productivity collaboration software can be great. But the best rapport a manager can build with a staff mirrors one I got paid a lot more for as a guostwriter after leaving the UN. The manager should not have to worry too much about how to give feedback. The writer needs to be the adaptable one and gets him or her what they want. What they want may then change of course as drafts are refined. Good writers can adapt. Those who cannot adapt should ideally be in other jobs than writing. Cheers, M

  5. I have given this a lot of thought and have years of experience on both ends of the equation, in academia, as a consultant and as a development policy influencer. The issue of mindset is important for all parties and those doing the reviewing certainly need to be kind just as those receiving the feedback need to develop a relatively thick skin. The ‘asking questions’ method is OK in its place but if something is wrong that needs to be pointed out in very clear terms. Also, you need to keep the ongoing correspondence to a minimum for the sake of everyone’s sanity. Asking questions implies that you are seeking to open a dialogue and whilst this may be useful or appropriate in some cases, in a lot of situations what is needed is quite clear guidance with a view to finalisation of a document.

    Here are some rules/guidelines that I think may be helpful:

    1. Review the paper that has been written not the one you think should have been written/that you would have written

    2. Provide handholding on the things that get lost sight of such as the structure, style and tone including identifying terms of art (I usually insert a comment/prompt for the author to verify that a particular term will be meaningful to the intended audience) and where you are able to suggest linkages to other approaches or disciplines you should do so provided you are keeping Rule 1 in mind

    3. Do not provide handholding for basics as this reinforces bad/inexperienced behaviour. So, I will tell you that I think a reference is required to substantiate an assertion, but I am not going to provide you with the reference because I am a reviewer not a research assistant

    4. Which is a pre-rule really – make sure you are clear as to what you are doing, if it is a review you should not be expected to copy edit but you should flag if a full copy edit is required

    5. Don’t be ‘that guy/girl’ whose review can be distilled to ‘I don’t really know what I want, but this isn’t it’ – this is particularly infuriating for consultants who have been contracted to provide a set number of days to complete a project and who (almost certainly) have already put in more time than the contract has allowed for

    6. Provide your feedback in a timely manner, if you can’t do so say so when you are asked to do the review so that they can pass it to someone else

    7. Check that everyone’s expectations are more or less the same. This is more important when reviewing something that someone has done ‘from the heart’ and I usually say ‘do you want me to tell you how wonderful it is that you have written a book or do you want my opinion, if it is the latter I will read it and provide it to you in detail, if it is the former then I don’t need to read it’

    8. Don’t say things that make you look like a jerk. I was recently asked (post proofs stage) to ‘add in more references’ – no concern as to quality or relevance, just ‘more’.

  6. I’ve had a few chastening experiences that have expanded my thoughts on this issue:
    1. Power matters – in particular your position relative to the person you are feeding back to. If you are an ‘upper’ to their ‘lower’, you need to be even more aware of how they are likely to receive your comments
    2. This blog is actually a form of (very) public feedback on a bunch of books, papers and conversations, but I haven’t always followed these guidelines. In particular, a couple of times a year I overstep my self-imposed boundaries of snarkiness and regret it. Yes, there’s nothing like a good rant for getting something off your chest, but it can do damage at the other end – someone recently told me, only half jokingly ‘I thought my academic career might be over when I read your post (I didn’t like their paper much)’. That’s horrible.
    3. Even worse is the gender issues lurking here – both feedback and blog battles can easily become testosterone-fuelled and very male. Need to think about that before I slag someone off in public.
    And yet, and yet. If I blog the full sandwich ‘there were a lot of really interesting issues raised at the seminar, and the food was really good’, half the readers will abandon ship before they ever get to the substance – there’s a reason why snark or at least stark, gets more traffic
    Off to my foetal huddle to consider………

Leave a comment

Translate »