How to read and comment on a draft paper – your suggestions please
Today’s vlog (I’ll be coming back to you in a few weeks to ask whether these are worth doing)
I spend a lot of time commenting on draft research and policy papers, both for Oxfam and beyond. So I put down some ideas on how I approach it, got some great input from Oxfam Research Team colleagues, and now we want to ask you to chip. Then we’ll publish a revised version in our series of research guidelines. Although this is based on NGO and aid agency reports, I suspect a lot of this goes for academic papers too.
How to approach a draft: Please try and put yourself in the shoes of the target audience and think what would interest or inspire them. Don’t go into internal lobbyist mode, combing through the document looking for/shoehorning in references to your particular hobbyhorse!
I read the paper as I would speed read the final article: the exec sum first, then the conclusion, then the top and tail of each chapter. I try to focus my comments on those sections because they are (by far) the most important in terms of impact.
But you do need to read the whole thing. Is the paper internally consistent? Have any nuggets (killer facts, case studies, telling graphics, genuinely surprising or new findings) failed to make it to the exec sum – a common crime in NGO papers?
As a reader, you are also likely to be better placed than the author to point out places where some extra narrative would help. I call it ‘hand holding’ – suggesting linking text between paras to improve the flow, or explaining why a certain piece of analysis is significant and worth a reader’s time – often underplayed by authors who are so obsessed with their subject, that they can’t imagine why anyone would find it boring!
Is the good stuff at the top? Let’s assume a large percentage of readers don’t read all the way through to the end. You need to make sure the best, most powerful ideas and arguments are up front, preferably in an executive summary, but if not, at the top of the paper. And then repeat them regularly throughout the text.
Remember what the paper is for and don’t try and expand its remit: If possible, take a look at the original terms of reference for the paper. They should set out the audience and purpose. That can help you avoid giving the classic unhelpful comment ‘it’s too long, needs cuts, and here’s another 10 issues you need to cover’! Depending on whether it’s at the wonky end of the spectrum (research paper) or the more popular end (policy paper), there will be a different balance of the need for rigour and accessibility, but both matter in all cases.
Think about what is not there. This is difficult but can be really useful – it’s easy to critique what’s in front of you, but it is often more helpful to stand back and identify what is missing – in terms of arguments, approaches, or sources. As well as stepping back, try to look sideways – the author is likely to be a specialist, up to their neck in the detail of a particular subject. Is there anything they could usefully import (whether as content or analogy) from different issues or disciplines?
Style and Language matter: A personal bugbear – a lot of NGO papers are really badly written. Full of impenetrable jargon, deadened by the passive tense, and/or shrill in tone ‘the IMF must do X, Y, Z’. That turns off potential readers and greatly reducec a paper’s impact. Here’s some good advice on writing for impact. Dealing with bad writing is tricky, especially if the writer does not have English as a first language, but in my experience, people are usually grateful for specific suggestions and edits. If rewriting the whole paper is not feasible, concentrate on making the executive summary accessible.
How clear are the ‘so whats’? A common weakness is papers that are strong on diagnosis and exposition of the problem, but weak on what to do about it (someone once caricatured such work as ‘bad sxxt, facty, facty’ papers). Does the paper have specific, well argued suggestions for how to improve things or are the recommendations bland and generic? (see How to write the recommendations to a report on almost anything).
Be kind: Once you’ve scribbled all over the paper, it’s time to feed back to the author. Take a deep breath. However brilliant/damning your critique, start by reminding yourself that there is a human being on the other end of this email. They have tried their best, even if the result needs work. You need to help them, not cast them into despair. Maybe talk to them face to face rather than just send an email?
Cue the infamous sxxt sandwich. Start off saying what you like about the paper, then what could be strengthened, but
finish off by stressing what is worthwhile. This needn’t be phoney – there is usually something to applaud in any piece or work. And even if people know what you are doing (a civil servant once told me ‘everything above the ‘but’ is bxxxxcks’), it still eases the pain and makes it easier to absorb and respond to criticism.
Be specific: Don’t just say ‘there’s some good stuff in here’. Give examples: a particular sentence that is nicely written, a particularly powerful paragraph, a quote judiciously used. Knowing what works (and hence what to retain) can be helpful as well as good for (battered) authorial morale. Even more so when you suggest improvements. To a harassed author on a deadline, comments like ‘needs more on gender’, or ‘you should read Amartya Sen’ are more likely to drive them to self harm than write a better paper. Specific text changes only, if possible.
Being on the receiving end of feedback can be bruising, especially when people don’t follow this kind of advice (I’ve got a few scars). If you feel your hackles rising, don’t get defensive, don’t hit ‘reply’. Probably worth sleeping on it before reacting. Remember that this is free consultancy, after all, and you get to choose which advice to respond to. If there are multiple commentators, so much the better – more advice, and easier to pick and choose what to take on board.
And yes, all this takes time, so it’s worth first making sure that the author really wants your comments, and getting them in ahead of deadline.
OK, over to you – what have we missed?