Measuring Social Impact of NGOs: Problems and Suggestions

It is not uncommon to wonder where all the monies, donations, funds and aids have gone, especially in the case of Afghanistan. Why, after a decade of humanitarian interventions, is the country still in physical and economic shambles? In The Crisis Caravan: What’s Wrong With Humanitarian Aid, Linda Polman dedicates one chapter to the situation in Afghanistan. How crudely, she titles that chapter “Afghaniscam”. She writes:

The tragedy of the admirable Red Cross rules is that they are unenforceable. “In this kind of war, calling on, or expecting the parties to respect humanitarian principles is like calling on a gang of armed muggers to fight by the rules of boxing; it’s not just laughable, it’s irrelevant,” said researchers from the University of Utrecht and Cordaid in Afghanistan. (p.9)

Apt metaphor. She makes mention that there are two schools of thought that pervade humanitarian debates:

(i) the Florence Nightingale school believes that aid fails when warring parties (mis)use it to their advantage, i.e. your efforts are being exploited by the military to exacerbate the violence and so you become complicit with the war machinery; and
(ii) the Henri Dunant school believes that help must be rendered nonetheless. It is a human duty, idealistically speaking, so some relief is better than none.

In a TEDxSanJoaquin talk by an Afghan student majoring in communications, Mustafa Babak recounted in his presentation, Success Story of Failure:

There are three factors that dominate the discussions and arguments when it comes to Afghanistan’s development. The very first one is counter-insurgency. Now, foreign military presence in Afghanistan has brought some sort of stability in the country. But I believe counter-insurgency has done more bad than good. Counter-insurgency is continuing to harm hearts and minds. Counter-insurgency is overshadowing development. Counter-insurgency is firing the enemy [?] it continues to create. Counter-insurgency is turning innocents into insurgents. Why am I making such a blunt statement? For every undertaking, for every mission or project, there are indicators for both successes and failures. Afghanistan is no exception. There are very clear indicators for successes and failures. But what is interesting, though, is that foreign military and the Afghan people seem to disagree on these indicators. For foreign military, or counter-insugency efforts, the indicator for success is how many insurgents killed, how many insurgents detained, or how many kilometres of Afghan soil have been cleared out of the hands of insurgents? For Afghan people, however, it’s not the number of insurgents killed. For Afghan people, it’s how well they are more secure year after year. Are they living in a better security, compared to 2002 and 2003? But that’s not the reality. Afghanistan is getting more insecure and more insecure.” (08:26 — 10:32 in video)

So here we are, not only with different philosophies and methods in collecting and performing humanitarian and development work, but we are also confronted with differing indicators of success.

In Evaluating International Humanitarian Action, Adrian Wood, Raymond Apthorpe and John Borton describe some of the challenges faced by practitioners, with case studies from Somalia, the Horn of Africa, Cambodia, Rwanda, West Africa, Papua New Guinea, to Kosovo. In the Foreword, as well as the chapters that follow, the following questions are raised (pp. xx, 1):

  • Are victims being reached?
  • Does assistance and protection make a difference?
  • What are the impacts?
  • Is learning occurring?
  • Is performance improving?
  • How is the evaluation actually done?
  • Are the programmes properly set up to address the task?
  • Are the right questions asked?
  • Is the quality adequate?
  • Is there duplication?
  • Are evaluation findings being used?
  • Who are the evaluators and how are they recruited?
  • What methods do they use?
  • How do they arrive at their conclusions and recommendations?
  • How do they cope when the agencies being evaluated press for their report to be less critical?
  • How effective are they at identifying what went wrong, what went right and what lessons should be learned from the experience?

In a separate research on microfinance interventions in Pakistan, for example, Asad K. Ghalib has created the Social Impact Measurement Index (SIMI), a conceptual and mathematical formula to examine the social impact of development interventions. He highlighted four variables that form the basis of the Social Impact Assessment (SIA) model, reproduced below:

Asad Ghalib, MEASURING THE IMPACT OF MICROFINANCE INTERVENTION: A CONCEPTUAL FRAMEWORK OF SOCIAL IMPACT ASSESSMENT (2009)

Ghalib states:

The four variables discussed above endeavour to encompass as many aspects of social impact as practically possible, given the fact that social and human behaviour add to the intricacies of measuring social impact. What needs to be appreciated at this stage is that each variable is not a stand-alone entity, and it should not be assumed that at any given instance, an individual is affected by only one indicator at a time. The process of social change is long and complicated and its outcome depends heavily on other social factors as well. Literacy for instance, raises awareness about HIV/AIDS prevention and educates people in how to take preventive measures, so it affects health-related issues, which, in turn, affects keeps them at work, and enables them to provide for their families, thus keeping them away from poverty, and thereby improving livelihoods. The whole process therefore works in coordination, as system. (p.14, emphasis mine.)

In my earlier post on impact measurements, I have suggested creating an ongoing evaluation template instead of adopting a template that had already been implemented. Rather than prescribing a single fool-proof method for social impact assessment, Ghalib insisted on nine principles that should govern our practice (borrowed from the Inter-organizational Committee of the US Department of Commerce in 1994), a useful framework for applied arts practitioners working in developing societies to also consider:

  1. Involve the Diverse Public
  2. Analyse Impact Equity
  3. Focus the Assessment
  4. Identify Methods and Assumptions, and Define Significance of Assessment
  5. Provide Feedback on Social Impacts to Project Planners
  6. Use SIA practitioners for Conducting an Assessment
  7. Establish a Monitoring and Mitigation Programme
  8. Identify Data Sources: (i) Primary; (ii) Secondary; (iii) Published Scientific Literature
  9. Plan for Gaps in Data

These are implementable principles.

Ultimately, any project, I believe, should consider separate objectives, outcomes and sampling in the various phases:

(i) Planning
– WHAT: realistic objectives and specific outcomes?
– WHO: target/beneficiary group?
– WHEN: duration?
– WHY: assumptions?
– SO WHAT: continuity and sustainability?

(ii) Implementation
– HOW: methods? processes?

(iii) Evaluation
– WHAT/ WHY: type of assessments? indicators?
– HOW/ WHEN: methods in data collection?
– WHO: community? stakeholders?
– SO WHAT: continuity and sustainability?

(iv) Report
– WHERE/ WHEN/ HOW: publishable platforms, e.g. social media, traditional media, print?
– WHO: funding bodies and stakeholders? feedback to community?
– WHY: benefits?

So, I take these principles with me as I begin my work with NGOs, and perhaps design custom-made templates for each different programme that I need to evaluate, just like how I would design individual lesson plans for each class that I teach.

That should work for now, I hope.

Advertisements