How businesses can save the world (when their shareholders aren’t breathing down their neck)

April 10, 2015

Links I Liked

April 10, 2015

Do Aid and Development need their own TripAdvisor feedback system?

April 10, 2015
empty image
empty image

I’ve been thinking about TripAdvisor recently, as a model of fast, crowdsourced feedback which highlights rubbish hotels and

We're watching you-hoo

We’re watching you-hoo

restaurants, and creates pressure for them to shape up.

There’s plenty of rubbish performance in the aid and development sector, but our feedback loops are mainly limited to conversations in corridors and the occasional email. So what would be your top candidates for a developmental TripAdvisor?

Let’s start with aid organisations themselves. My colleague James Whitehead asks: “Is there space for ‘Rate My Aid’? In a humanitarian crisis the affected populations are often over-surveyed yet have very little voice in the services they receive. Feedback from communities could be cross-referenced against data on back donors to create a leaderboard of best UN agencies, best INGOs, best local NGOs, and worst… If I was a donor, I might look to the leaderboard before I looked to the project report – to what beneficiaries say about the service rather than what the service deliverer says about it.”

Aid organisations could rate each other too. Imagine a system where grant recipients could feed back anonymously on the behaviour of their donors; do they listen to you? Make unfeasible reporting demands? And vice versa – did money go missing? How big were the over-runs? Which ones really delivered exciting programmes?

feedback powerAnother topic could be the corporations who are looking for NGO/official aid agency partners – some of this is a genuine interest in development, some is PR spin. It would be great if you could just click on the feedback for any given company and find out who is serious, and which ones are just Uber/UN Women PR disasters in the making.

But why stop at organisations? What about individuals? Here’s where it gets a little trickier.

What about rude aid workers? When humanitarian agencies and their partners are filling in a gap in state provision, a government could easily insist that all aid workers wear a unique identifying number like police officers. Combined with clear crowdsourcing you could easily see which agencies treated people with dignity and which staff might need further training. The days of humanitarian agencies acting as unaccountable non state actors could be numbered…

Consultants: they charge the earth, often for substandard work. How do we find out in advance which ones are good, and which ones just endlessly cutting and pasting the conclusions from their last report?

Journalists: they phone you up, suck your brains dry in search of new ideas for their next piece. Which of them actually write

Not any longer?

Not any longer?

anything as a result, or credit you or your organization when they do so?

Peer Reviewers for research proposals: Half of them don’t deliver, or use the opportunity for cathartic venting about their hobby horses, rather than actually reviewing the proposal.

Civil servants: which ones really listen and interact, which go through the motions?

Campaigners and Advocates: Do Government or World Bank staff get to comment on how they feel about the work of different aid agency lobbyists?

Some design issues:

There is likely to be a huge difference between feedback from targets and from peers – any way to differentiate?

Everyone will try and game the system (‘sure I’ll take that meeting, but you have to give me a good review’). I wonder what TripAdviser does to avoid that?

The dangers of blacklists: at the very least you would need an appeal procedure against unfair reviews. Would that make it unfeasibly expensive and bureaucratic?

I sent these thoughts to a couple of IT accountability gurus: Martin Tisne at Omidyar asked some good questions:

‘Is it 2-way or 1-way? Who is the ‘client’ for this type of platform? With TripAdvisor, the clients are the paying customers of the hotels – i.e. it’s not 2-way, the hotel does not get to exercise much choice in terms of the customers it takes on or not, no accountability__saad_murtadhablack listing etc. Uber and the like purport to have a 2-way system with customers rating cab drivers and cab drivers rating customers but I suspect that on the whole it’s the cab drivers who are really being assessed rather than the other way around. So what are the experiences out there we can learn from that are genuinely two-way and what have they learned about [dis-]incentives to publish and how to deal with conflicts of interest?’

While Tom Steinberg added some sage advice, based on his experience of setting up lots of such feedback sites in the UK:

‘The art to succeeding here is to start by choosing just one kind of review of one very carefully chosen kind of service – one that has been chosen to minimise the perverse incentives and so on. Try to get that very narrow service working and then move on from there.’

What do you think?

An edited version of this piece appears today on the Guardian Development Professionals blog.



  1. I’ve been thinking about this idea for years!! Particularly for donors. There seems no way for grantees to express any concerns or try to get improvements in processes, focus etc from donors. I think a system to provide feedback would be great. Of course, it’d probably become just as gamed and useless as Tripadvisor after a few years, but even a few years would be great!

  2. Very interesting idea. Greater transparency and accessibility to performance information is certainly needed and long overdue.

    As a former donor, I urge that any such experiment gets granular enough to be meaningful and useful. For example no 2 program officers are the same, and there are a variety of internal and external factors that allow program officers to be effective. General perspectives on larger donor organizations won’t be useful unless it can get beyond the macro. The same is true for large implementing orgs (from NGOs, to the large consultancies…).

  3. Clearly an important issue in development cooperation. In part impacted by the tension / question who the “client” is (the recipient or the donor?) and who wants to hear what message, plus the assymetry regarding information (not only do development agencies & NGO’s implement, but they play key role in information transmission in both directions…)

    Interesting attempts to get feedback are “The Listening Project” (CDA) and the MyWorld Survey, but a more structured platform working “project by project” (or at least “agency/NGO by agency/NGO”) could definitly be of added value. (The UK’s survey of multilaterals could have been interesting place to test-run such a system). Of course, some effort would be needed in roll-out and communication and ensuring the system works (what type of access is needed in remote areas, language issues, how to “understand” qualitative comments, etc…) Moreover, it wuold also be important to ensure all involved are confident and open to the (possible) input and criticism and that it doesn’t just become a tool to complain.

    As suggested, limited trial, e.g. a platform for CSO partners of multilaterals and large donors to provide input might be a good place to make a “pragmatic” start.

  4. Hi Duncan

    I was excited to see the topic of your blog this morning as we are currently hard at work developing AidReview to be just this: an independent, free and easily accessible website on which users can search for and post reviews, comments, or suggestions related to any international development and aid activity. Any actors involved with or affected by aid efforts will be able to submit feedback on any project, and the reviewed party will be notified and have the opportunity to respond.

    You might remember that we sent you the initial AidReview concept note and asked for your views on this a little while back?

    We believe that although global efforts to improve the accountability and transparency of aid have grown significantly over recent years, the aid industry still spends huge amounts of money doing things without sufficient knowledge or feedback on what works. More feedback loops are needed, especially those that are genuinely independent, open and unconstrained. AidReview will increase voice and accountability at all levels of the aid architecture, leading to more effective programming and better development outcomes.

    We’re currently working with a small team of volunteers to design the functionality of AidReview. We’ve recently conducted our user requirements survey, and are now starting the process of benchmarking – approaching sites such as TripAdvisor and speaking to experts to identify just the sorts of issues that you raise in your blog – and to find solutions for AidReview.
    We’re also looking for seed funding to enable the development of AidReview into a proven concept that is ready for launch globally – in case you know of anyone that might be able to help with this!

    If anyone’s interested in knowing more – or would like to get involved – please see or contact for further information.

    Best wishes
    Josie Stewart and Amy Harrison
    AidReview co-founders

  5. I am sure you know about Keystone Accountability Surveys. These independent and confidential survey, administered to the southern partners of 20 plus INGOs through Keystone Accountability, allowed southern partners to provide completely anonymous feedback on the quality of their relationship with their respective INGO partners. The Keystone Accountability Survey is innovative and unique because, unlike other surveys, it allows NGOs to benchmark their performance against that of their peers. This benchmarking process helps INGOs learn from each other, and many or the orgs have shared the results with their partners. Some are considering replicating the process to allow beneficiaries to assess their partners as well. To date, I think close to 30 INGOS have joined in the survey process, including Save the Children UK, CARE USA, Trocaire, International Rescue Committee and Oxfam Canada (UK? Intl?)

  6. Thank you for this blogpost! Being involved in research on the issue of accountability between INGOs and their target group, I found this really interesting. Some short thoughts, which are strongly linked to the remarks of Martin Tisne.

    The logic behind Tripadvisor seems to be founded in an economic relationship between (potential) clients and companies. This relationship is ruled by a logic of exchange and mutual gain (‘I will give you X, if you perform or yield Y’). Feedback in this relationship is a client’s way to exert influence over the services he receives from providers. The power behind this feedback, what makes feedbackgiving worthwhile for the client, is market-based; the clients’s feedback works because it can lead to the (potential) exit of himself or other clients; it can make other clients look for other hotels. But, the question is if this customer-feedback model fully applies to the developmental sector. Does the development sector have clients? If so who are they? Is it the beneficiaries? And do these beneficiaries have the power to exit? What power do they have to influence the work of NGOs? Or are the donors the clients of NGOs? Should we think of development work, which undoubtedly entails aspects of transaction, as a relationship between clients and customers, or does it also entail aspects of cooperation?

    Relatedly, one could wander if thinking about feedback does not require a further distinction between the different types of development activities NGOs are involved in. Feedback in humanitarian aid and feedback about policy influencing seems to refer to different things.



  7. Although the concept is attractive, rather like TripAdvisor itself, this could be a bit of a gimmick and of little real use because of difficulties of knowing whether to trust the opinion of a disgruntled, or conversely, an over-enthusiastic commenter. The problem for some aid donors is that, as you have pointed out, who the ‘client’ is can be difficult to define. Final end-users of aid (the real beneficiaries) can be too remote in the process of delivering good aid to be aware of who provided that aid or what could have been improved – although I do accept some may be. For vertical-funding donors, the feedback would likely be from donor-funding recipients who pass the aid on beneficiaries themselves. A chain of accountability feedback is needed – which good organisations should already have in place. What is needed is for those organisations to be transparent about the feedback they already receive and, importantly, to be transparent about what they has done about that feedback. This is much more realistic in my view.

  8. Thanks for the post Duncan. Quite interesting to track the shifts in the private sector, beyond customer satisfaction ratings. E.g. some multinational oil and mining companies follow practical and consistent steps to acquire a ‘social license to operate’ by demonstrating their commitments to corporate social responsibility in negotiations with the national government and with local communities around their operational sites. Part of this practice includes setting up grievance mechanisms and hiring dedicated community relations staff. Conversely, most NGOs continue to be rated and ranked based on what they write (and promise!) in concept notes and proposals and subsequently in their reports submitted to the donors. At CDA, we continue to study feedback loops and consistently find that their effectiveness is tied to institutional capacity to absorb, make sense of and act on the feedback (and incentives to do so!). Keystone Accountability survey of partners and the Ground Truth initiative piloting practical ways to gather recipient feedback about humanitarian service providers are important steps which could contribute to a shift in our sector too.

    But I do worry about this turning into a beauty contest with perverse incentives to improve the façade in order to get a higher ranking. And indeed, as the previous poster mentioned, who is the customer /client we are trying to impress with the rankings? Donors or aid/service recipients? We also know that the power structures and dynamics between NGOs and local communities cannot be ignored in any feedback process. An effective feedback loop in protracted humanitarian contexts or long-term development programs requires a consistent response to feedback and quite often, a sustained dialogue about what type of corrective action is needed, who needs to be engaged in that action, roles and responsibilities, and expected improvements. DFID aspires to “adaptive programing.” But this requires a collaborative problem-solving process that comes with a particular disposition, set of skills, and political will to advocate for mid-course corrections or possibly, significant strategy level changes. Otherwise feedback can easily become background noise and be treated as whinging and whining… too often ignored. A lot of aspiration about feedback loops… but it gets stuck in institutional bottlenecks.

  9. All Oxfam affiliates recently participated in the Keystone Accountability survey (mentioned in one of the comments above). The feedback report from our partners will be posted on our website ( within the next few weeks.

    This survey asks partners about what they value (or not) in their relationship with Oxfam. It can also be used to ask community members about what they value in their relationship with partners. As the survey is done through an independent organization (, it offers a guarantee of anonymity in the replies. It is much more in depth than a quick review that could be generated from something like a trip advisor platform, so I think it avoids some of the more problematic aspects of individual replies. Further, it offers an orientation to learning, by establishing dialogue with our partners, to strengthen our good practice and overcome areas of weakness. One of the benefits of the survey process is that, because other organizations have carried out a similar process, we also have the option to discuss this with our peers, looking at lessons and ways to improve.

    As non-traditional platforms are being established to capture quick feedback, I hope they consider mechanisms to ensure solid input and opportunities for lessons moving forward.

  10. Really interesting post Duncan. Thanks for your thoughts, and for sharing Martin and Tom’s reflections too. It sounds like it’s got potential and I’ll watch AidReview’s progress with interest. However, I also have reservations about how truly representative a website feedback platform could be when you think about how severely limited many people’s access to the internet/electricity/technology still is, esp. for certain groups more than others e.g. women/girls in particular. There are also potential issues around literacy and language – most aid agencies HQs still only work in a handful of languages for example. I imagine that it’s easy enough to crowdsource relatively affluent people’s holiday feedback, and I can see this having legs in disasters where the bulk of the population was already well-educated and tech savvy but not nec in other contexts.

    On a slightly different subject, I’ve been pondering the value in aid/dev employers collecting longitudinal feedback on employees from past projects and partnerships they were involved in, to see what happened some way down the line as part of their annual performance reviews – potentially via previous employers if nec. Could be awkward and costly, but potentially pretty revealing…

  11. Interesting idea. Some sort of system of feedback for development consultants could be a relatively straightforward starting point. There is a real need to cut the waste of development funds used to pay for overpriced and sometimes shockingly poor work, and to ensure that consultants that provide genuinely useful information and analysis get the recognition they deserve.

  12. Isabella Jean (comments) points to Keystone Accountability’s Ground Truth programme as an example of trying to create a robust feedback system that puts affected people in the driver’s seat of humanitarian aid – or at least in the co-pilot’s seat. We’ve worked with many agencies, including OXFAM, over the last three years and these are some of the key lessons we have learned:

    • Enthusiasm for listening and acting on feedback from affected people grows over time as aid agencies become accustomed to the idea of rapid-cycle learning and affected people come to understand the benefits of candid engagement.
    • Feedback – and the dialogue it triggers – acts as both a driver of continuous reflection among program implementers, prompting them to ask tough questions about their performance, and continuous improvement.
    • Embracing the approach requires determined leadership in the humanitarian community and external incentives (essentially from donors) to create a culture that rewards listening, learning and responding to affected people.
    • Survey fatigue among affected people can be avoided by ensuring respondents recognize it is worth their while to engage – because they see the results of doing so as programs take their views into account.
    • Getting agencies to use the data is the big challenge. In addition to incentives, this requires a continuous stream of feedback because one-off or occasional surveys are easy to ignore. Ongoing feedback, on the other hand, provides continuously updated benchmarks against which to manage.

  13. I am grateful to Duncan and those who have responded to his blog post with their comments. Thank you – you are honing in on what we at Keystone call Constituent Voice, which in our view is the most radical and important trend in development today. I would like to make three points.

    First, the technical problems to ensuring that the views of those who are meant to benefit from development interventions are adequately heard and acted upon have been or will be solved in the next little while. There is – as the comments evidence – an explosion of innovators in this space. New communications technologies are an enabler. For those who want to be part of the innovation process I recommend two networks. Feedback Labs, with hubs in Washington DC, New York and London, is the emerging Academy of Constituent Voice practitioners – a place where diverse practices can be celebrated, examined, tested for quality, honed and spread. The Fund for Shared Insight and Making All Voices Count are the first of what I hope will become many collective funds established by funders to support the wave of Constituent Voice innovators and activists. Even more importantly, these funders are actively engaging with feedback data and discovering how it can improve their performance and results.

    Second, the real story here is not about the technical issues or technology. It is about what the folks at Ashoka call framework change. More than organizational or even system change, framework change involves the basic ways we think about something, our norms and culture. The tipping point will come when enough organizations understand that making the voices of those who are meant to benefit truly matter is not only the right thing to do, but also the smart thing to do. The examples of how feedback improves results are multiplying, and from wildly different fields ranging from therapeutic counselling (e.g., Feedback-Informed Therapy) to prison reform (User Voice). The upcoming June issue of Alliance Magazine will contain many more examples as part of its focus theme, “Feedback as Transformation”.

    Finally, Francis Ford Coppola was wrong, sadly, when he said that directing films was the last dictatorship in an increasingly democratic world. There are all too many other dictatorship dominions around. But time is now running out on the dictatorship of development. The people can and will be sovereign. The days when implementers and funders of human and societal development programs can say, without shame, that they do not know what people think about their work are running out. The days when development practitioners not only listen, but act on what they hear, are arriving. Soon not having credible Constituent Voice metrics will be seen as akin to not having accurate financial accounts. The time to end this dictatorship is now.

    David Bonbright
    Co-founder and Chief Executive, Keystone Accountability

  14. Thanks David. What l find interesting about Duncan’s Trip Advisor style proposal is how it opens up the possibility of ‘disintermediated’ feedback by affected communities which doesn’t rely on aid agency input. Do you have any idea if any of the groups involved in the Feedback Labs have managed to do this in an inclusive way?

  15. Thanks for your question Lucy. To understand it better I would like to understand it better first. What is it that you find interesting about “disintermediation” feedback? Am I right that when you say “inclusive” you mean feedback systems that adequately reflect the views of those who are most meant to benefit from the development intervention, to use an imperfect term, the neediest?

    As I reflect on your question I find myself asking other questions. Is any feedback system without mediation? What is mediation in this context? What is the relationship between purpose and the type of feedback system one wants? But I would like to understand you better before I tackle your question. Cheers!

  16. Hi David, thanks for wanting to better understand my question, and apologies that it wasn’t clear first time around. I’ve followed some of the excellent work that Keystone has done with aid agencies in supporting them to collect constituent feedback, and also the way in which Keystone collects information on behalf of some aid agencies too, and am very supportive of your approach. A.Barry mentioned the “Listening Project” (CDA) and the MyWorld Survey which I’ll take a look at, and I’ll also watch the AidReview project development with interest…

    Your point about whether any feedback system is ever truly mediation-free is a very important one, and I have a hunch that the answer to that question is ‘no’ as the way questions are phrased or the medium through which feedback is collected will always mediate people’s answers to some extent.

    What interests me about Duncan’s question is the idea of providing a way for affected populations to provide feedback without 3rd party intermediation e.g. they wouldn’t have to wait to be asked for feedback by an aid agency, and that this feedback wouldn’t need to be given to an aid agency/consultancy company which was paid for by the aid agency. Instead, a Trip Advisor style feedback system would allow people to provide feedback whenever they felt like it via an independent body, and the results would be immediately available which would allow affected populations (and aid agencies/donors) to do their own analysis and draw their own conclusions.

    I’d shared some of the challenges that I saw with this system in my earlier post, so wondered if you’d come across any examples of this type of internet-based feedback system via the FeedbackLabs which yes adequately reflect the views of those who are most meant to benefit from the development intervention?


  17. Hello Lucy,
    Just to clarify on the approach that we took with the Listening Project at CDA – we conducted listening exercises in 20 aid recipient countries and listened to 6,000 people on the receiving end of international assistance. The listening team members were aid workers/NGO/donor staff (130 organizations participated across 20 countries) and we solicited people’s opinions and perspectives about the cumulative impact of aid efforts (all forms of assistance)over time. The feedback was in many ways both specific but also meta-level -feedback to the aid sector about systemic issues. We lay out the most significant themes that emerged from the analysis of this cumulative voice in our book “Time to Listen: Hearing People on the Receiving End of International Aid”. But the Listening Project was by no means a continuous feedback loop since it was a one-off exercise. More recently, we’ve been working with aid agencies to understand what makes feedback loops effective, looking at enabling factors from the point of view of operational and institutional landscape.

    On your other point regarding unmediated feedback systems — distinguishing between supply side initiated feedback (solicited by aid agencies) vs demand side initiated (unsolicited complaints and feedback raised by local civil society or service recipients) is important. There are contexts where there is space for such participation and initiative and where people claim this space. Certain other contexts have seen such as high level of restricted participation, dependency on aid agencies and deference to them (multiple power structures at play as I mentioned in the earlier post) that any feedback process has to be structured with a keen understanding of socio-cultural and political dimensions of access, transparency and the factors that support an enabling environment e.g. the right to information, freedom of expression, non-retaliation, etc. This is where comparisons to Trip Advisor become less tenable..

  18. Lucy — Thank you for your generous comments about our work. And for elaborating on your question. I think the best way I can answer it is to cite Fredrik Galtung’s useful 3-part feedback system typology. For his blog post on this, see:

    But here is the essence:

    Type A: Principal-initiated and managed feedback system, reporting directly for example to a Presidential Delivery Unit[3], the Prime Minister’s Office, a Governor or Mayor. Type A feedback systems enable Principals, to get real-time feedback on problem hot-spots, the effectiveness of their departments and to initiate appropriate remedial actions.

    · Type B: Manager-initiated and controlled feedback systems, for example for a specific health service, school district, or a major infrastructure development project.

    · Type C: User-initiated and owned feedback systems actively engage citizens in sharing responsibility for resolving the problems that affect them. In the commercial world two well-known examples of user-initiated and owned feedback systems are and; and services as and couldn’t exist without user-generated feedback.

    Type C is more complicated and the capacity to implement this in many countries is weak. But it’s the key to a robust approach.,, and are three powerful examples of this approach that focus on public services.

    If I understand you correctly, you are asking about Type C, and there you have Fredrik’s three leading examples. Many of the citizen scorecard efforts of the last ten years are also Type C. One of the big challenges for Type C is financial sustainability.

    Fredrik makes a very wise observation in that piece that I often repeat: No one type is ideal. We need all three. He introduces the idea from the tech world of redundancy. We want redundancy in our overall feedback system design.

  19. Dear Duncan, Thanks for your great blog!

    We believe very much in your analysis and the vision that technology can empower the voice of “beneficiaries”.

    We have created, a technology startup, because of the same frustration as you: the aid sector is plagued with ineffective interventions, bureaucracies, wrongly designed programs and poorly performing consultants. If we were in a normal market, many UN agencies and NGOS would have disappeared long ago.

    At Bluesquare we are working hard on desintermediating aid through result based financing: we believe that technology can facilitate money flowing directly from donors to health centers, schools or households with limited interventions and unnecessary middle mans. Today, our data system supports result based financing in 14 countries, and finances 7500 health facilities, 200 schools and 20 tribunals. For more information on how this works :

    Part of our work with governments, the World Bank and is to use technology to enhance social accountability in health, education and governance sector. We want people to express their opinion about the services that they receive from hospitals, schools or police stations. Currently, we collect systematic feedback through mobile phones and display the “satisfaction” score on public web pages (see for example the Bwera primary school in DRC has 85% satisfaction from parents and pupils, or 83% for a health center

    But we believe that we could go much further: display comments on the provider page, link it with radio programs,… Would anyone be interested to co-create on this? We develop similar systems in health in Benin, Haiti, Burundi, CAR, Burkina, Nigeria, Cameroun, Senegal, Malawi. If yes, contact me :

  20. Wholeheartedly agree that this is an idea worth exploring. This could be an especially useful idea for consultants (among the other ideas you listed), who often seem to jump from gig to gig and get jobs more easily as their CV grows, even though they might not have done a particularly great job at any of those individual jobs. Looking at these people’s bios, you see a lot of “advised XX, consulted XX, was a a keynote speaker at XX”, but there is no way to know if that person did well in each of those functions, or if they left all of their clients feeling disappointed. It would be tricky to do this because of the small numbers of people involved and personal feelings (especially because criticism of an individual is always more likely to hit hard than criticism of an organization), but if it could be done tastefully, if could be a huge help.

  21. Just two quickies….
    1. A development trip advisor could be useful if people do not expect too much from it: as a simple high level indicator of satisfaction or otherwise from affected people (you would need more nuanced demographic categories to make a little more sense of it). Its main value would be a spur to further investigations and dialogue etc.
    2. However, feedback systems can be developed that enable much more specific feedback from specific groups on specifics performance dimensions – without becoming burdensome or expensive – as long as the feedback data feeds into a cycle of listening, engaging, learning and improving with respondent groups.This is where the exciting work lies.
    3. Perhaps a third… I am often uncomfortable by the articial dichotomy that gets created in accountability debates – posing it as an either/or or zero sum. what we are aiming at surely is a greater balance – mutual accountability for shared values and results.

  22. This website crowdsources, rates and ranks US NGOs, based on hundreds of thousands of pieces of feedback

    According to Johannes Kiess, the best feedback comes from volunteers in the NGOs concerned (the interns’ revenge!)and the ratings are starting to have a significant impact on fund raising.

  23. The same idea of a tripadvisor for aid agencies has come up in the consultations for the World Humanitarian Summit. Building on the great comments above, I posted this blog exploring the idea further:

    It looks like the contexts we work in are too different for a general tripadvisor. But there’s real scope for focused feedback mechanisms, like Tom Steinberg suggests.

Leave a comment

Translate »