Evaluating Academic Projects through Research Excellence Framework

Many of the issues around evaluation, measurement, and assessment revolve around output, its wider reach and its sustainability, notions stemming largely, I believe, from Social Sciences. In the UK, the Research Excellence Framework (REF) — previously known as Research Assessment Exercise (RAE) — is a system comprising an expert panel that evaluates the eligibility of a project for funding. In other words, the quality of research undertaken by Higher Education Institutes (HEIs) is monitored through this process of expert review.

Three criteria are used to evaluate proposals. They are:
A) Outputs (65%)
B) Impact (20%)
C) Environment (15%)

A. OUTPUTS

According to the REF, outputs are evaluated based on (i) originality; (ii) significance; and (iii) rigour (see definitions below). It is also understood that:

[the] outputs will span a range of writings, edited
publications and research-led creative practices, as
well as artefacts and curatorial outputs. The sub-panel
expects to evaluate research that encompasses
analytical, applied, ethnographical, historical,
pedagogical, practice-led, scientific, technological and
theoretical approaches to the widest domains of
dance, drama, music, performing and screen arts, and
covers the broadest understanding of the subject
disciplines within any cultural, geographical or
historical context. (see paper in Panel D, Part 2D)

According to the paper, specific examples of outputs may include:

• books (authored or edited)
• chapters in books
• journal articles
• working papers
• published conference papers
• electronic resources and publications
• exhibition or museum catalogues
• translations; scholarly editions
• creative writing and compositions
• curatorship and conservation
• databases
• grammars
• dictionaries
• digital and broadcast media
• performances and other types of live presentation
• artefacts
• designs and exhibitions
• films, videos and other types of media presentation
• software design and development
• advisory report
• the creation of archival or specialist collections to support the research infrastructure.

So how does the panel define these three indicators for outputs?

(i) Originality is defined as:

a creative/intellectual advance that makes an important and innovative contribution to understanding and knowledge. This may include substantive empirical findings, new arguments, interpretations or insights, imaginative scope, assembling of information in an innovative way, development of new theoretical frameworks and conceptual models, innovative methodologies and/or new forms of expression.

(ii) Significance is defined as:

the enhancement or deserved enhancement of knowledge, thinking, understanding and/or practice.

(iii) Rigour is defined as:

intellectual coherence, methodological precision and analytical power; accuracy and depth of scholarship; awareness of and appropriate engagement with other relevant work.

To attain a four-star rating (the highest on the rating scale), not only must the output achieve the “highest standards of excellence”, it must also be recognised internationally, acquiring the status of “world-leading”. Evidence must show that the research is:

• a primary or essential point of reference
• of profound influence
• instrumental in developing new thinking, practices, paradigms, policies or audiences
• a major expansion of the range and the depth of research and its application
• outstandingly novel, innovative and/or creative.

B. IMPACTS

Furthermore, impacts are assessed based on (i) reach; and (ii) significance. Here, the REF defines reach as “the extent and/or diversity of the organisations, communities and/or individuals who have benefited from the impact”, while significance refers to “the degree to which the impact enriched, influenced, informed or changed the policies, practices, understanding or awareness of organisations, communities or individuals.”

How different this indicator for “significance” varies with the earlier criterion within output is not explicitly made known, but the panel will assess both reach and significance cohesively, rather than singly or separately.

Music, drama and performing arts, for example, fall under Panel D, Unit of Assessment No. 35 (see link). So, the panel will assess the extent the arts have been conducive to “achieving impacts of ‘reach and significance'” (see link).

The indicative range of impacts is illustrated as follows:

Suggestions on specific examples of impact include the following (see paper for more):

• Generating new ways of thinking that influence creative practice.
• Creating, inspiring and supporting new forms of artistic, literary, linguistic, social, economic, religious, and other expression.
• Contributing to innovation and entrepreneurial activity through the design and delivery of new products or services.
• Contributing to economic prosperity via the creative sector including publishing, music, theatre, museums and galleries, film and television, fashion, tourism, and computer games.
• Informing or influencing practice or policy as a result of research on the nature and extent of religious, sexual, ethnic or linguistic discrimination.
• Research into the languages and cultures of minority linguistic, ethnic, religious, immigrant, cultures and communities used by government, NGOs, charities or private sector to understand and respond to their needs.
• Helping professionals and organisations adapt to changing cultural values.
• Contributing to continuing personal and professional development.
• Preserving, conserving, and presenting cultural heritage.
• Developing stimuli to tourism and contributing to the quality of the tourist experience.
• Influencing the design and delivery of curriculum and syllabi in schools, other HEIs or other educational institutions where the impact extends significantly beyond the submitting HEI, for example through the widespread use of text books, primary sources or an IT resource in education.
• Contributing to processes of commemoration, memorialisation and reconciliation.
• Contributing to a wider public understanding of basic standards of wellbeing and human rights conceptions.
• Informing or influencing the development of expert systems in areas such as medicine, human resources, accounting, and financial services.
• Influencing the methods, ideas or ethics of any profession.
• Providing expert advice to governments, NGOs, charities and the private sector in the UK and internationally, and thereby influencing policy and/or practice.
• Engaging with and mediating between NGOs and charities in the UK and internationally to influence their activities, for example in relation to health, education and the environment.
• Contributing to widening public access to and participation in the political process.

The paper further highlights examples of evidence of impacts. They are:

(i) Quantitative Indicators (e.g. sales figures, funding from public or charitable bodies, tourism data, audience figures, visitor numbers, employment data)

(ii) Critiques or Citations in Users’ Documents (e.g. reviews outside academic literature, blogs, catalogue notes, teaching materials, uptake of research by public or commercial bodies)

(iii) Public Engagement (e.g. number and profile of people engaged and types of audience, media coverage, descriptions of social/ cultural significance, downloads, ongoing engagement with a group)

(iv) Policy Engagements (e.g. evidence of influence on a debate in public policy, formal partnership agreements or research collaboration with major institutions, NGOs and public bodies, changes to professional standards and behaviour)

(v) Independent Testimony (e.g. acknowledgements in annual reports or other publications, testimony of experts or users who can attest to the reach and/or significance of impact, 3rd-party evidence of changed policies, practices, processes, strategies)

(vi) Formal Evaluations (e.g. professional evaluations of exhibitions, performances or other outputs, formal peer reviews, studies on the social return on investment)

Overall, a four-star rating means “outstanding impacts”, rather than “very considerable impacts”.

C. ENVIRONMENT

Lastly, environment is assessed based on (i) vitality; and (ii) sustainability. Similar to outputs, a four-star rating for environment means vitality and sustainability must be “world leading”, “international”, and having “excellent quality” (see link). In the paper, five sections have been identified:

(a) Overview
(b) Research Strategy
(c) People
– (i) Staffing Strategy and Staff Development
– (ii) Research Students
(d) Income, Infrastructure, and Facilities
(e) Collaboration and Contribution to Discipline or Research Base

In definitional terms, vitality refers to “the extent to which the research environment supports a research culture characterised by intellectual vigour, innovation and positive contribution within the discipline(s) and profession”, whereas sustainability refers to “the extent to which the research environment ensures the future health and wellbeing of the unit and the discipline(s).” In other words, environment here refers essentially to the promotion, maintenance and advancement of a rigorous research culture within the organisation, rather than the community (research subject) in which the project is based.

This means that our research projects should be directly or indirectly influenced by the REF if we want to consider the “reach” of our research. However, if we follow blindly and rigidly  to these suggestions just to obtain funding and, finally, to acquire the privileged four-star rating, there is a possibility that arts projects run the risk of losing organic spontaneity. This is because knowing the outputs, purposefully charting post-research milestones and ensuring sustainability are OUTSIDE the research process. As researchers on the ground, shouldn’t we be more concerned with ground issues, responding to changes as we, or the communities we work with, deem fit? I have previously argued (see video) that when practitioners and researchers are concerned with social impact and measurements, the drama process or the drama event becomes the EFFECT and the wider impact as the CAUSE, simply because “impact measurement” determines the direction of the work. This is a misplaced equation.

Misplaced “Cause-Effect” Directionality in Impact Measurement

Why?

A personal anecdote might illustrate this point. I have been a teacher since 2001. Today, a former student of mine who was visiting London for a day met up with me for lunch at Covent Gardens. It has been ten years since we last met. At 24 years of age now, Rachel is working with the civil service. While we reminisced the past, I cherish the fact that she maintained contact with me after many years of silence and distance. She then passed me a gift, a vacuumed-pack box of barbecued pork from Singapore. It was a priceless moment!

Often, teachers may not get to see the fruit of one’s labour. We don’t often reap what we sow. It is impossible to witness Rachel’s “growth” if she had not maintained contact after these years. In the same way, a teacher would not know the “impact” one has on his or her students, unless the student returns one day to thank the teacher. The inability to capture and record the impact of one’s teaching does not negate the effects it has on the student. But the “effects” were not my primary motivations for a meaningful engagement in the classroom; I didn’t become the teacher that I was with the “end” in mind. Rather, it was the daily interaction steeped in a pedagogical and humane philosophy that allowed me to adapt to real world changes in the classroom, to be present, and (hopefully) to be inspirational.

In the same way, the success of a prison rehabilitation programme is too tough to be ascertained. While institutions may rely on the reduced rates of recidivism as proof of rehabilitative success, many factors are at work; not just one. To boast that a single rehabilitation programme — say a drama workshop in the prison — has reduced the rates of re-offending is unjustifiable. Religion, social networks, family support, employment, financial and housing access, educational opportunities, personal resilience, etc are some of the other post-incarceration factors that are often overlooked in these “impact measurements”.

Therefore, I am putting forward a renewed equation:

New “Cause-Effect” Directionality in Impact Measurements

Instead of asking “What can we do to change our drama processes so that we can fully capture and measure impact?”, we should ask: “What are the community’s needs that are changing our artistic practices?” Following from that, then “What changes do we, then, need to make to measurement templates?”

As a result of this renewed question, the measurement template must be adaptable enough to record ephemeral and spontaneous moments of improvisations. Because of this, drama processes should be the CAUSE (not EFFECT), i.e. the drama processes  change how we should measure impact. I had previously argued for an omni-directional concept of CAUSE-EFFECT, but I shall not explore this here.

The next challenge now is not only the need to speak in the language of funding bodies and review panels, but to open up opportunities for dialogues on the efficacies of frameworks that shape, AND be shaped, by research projects. I think this is one of the fundamental problems within impact measurements.

*** Note: The postgraduate symposium in the arts was held on 26 May 2012 at the University of Manchester, dealing with some of the salient issues and debates as highlighted. Video presentations by each presenter, including mine, can be found here.

Advertisements