Policy research design in policy, data & research light environments

Submitted by Eoin Young on Τετ, 02/09/2015
Resources & Tools

 

In two of our ongoing projects (the Policy Bridging Initiative with the Regional Research Promotion Programme and a Policy Fellowship with the Soros Foundation Kazakhstan), we will run workshops on policy research design in February. In order to frame our discussion for workshop participants and also to start a broader discussion of effective approaches to policy research in challenging environments, I decided to put down what we have learned from quite extensive experience of mentoring policy researchers and think tankers for over a decade in what are commonly (and now more and more misleadingly!) called transition countries1 .

From the outset, I must say that the policy research design approach presented here is a very particular view, driven by a very particular set of circumstances. The characteristics that frame and drive the approach are: 

  • A small amount of previous policy or social science research to build on;
  • Limited adoption of performance based planning within government administration and so availability of such data;
  • Limited access to data & limited quality of data that you can access;
  • Limited openness of government to working with researchers or policy research evidence in decision-making;
  • Relatively small research budgets and short timelines to complete research projects.

And still the ambition of serious policy researchers has been to “say something sensible about a complex, relatively poorly controlled and generally ‘messy’ situation”2 . Robson’s definition of real world research is particularly apt in these circumstances and indeed limits the choices available. A disclaimer at the beginning: there are many different kinds of research done in the region and I am not implying this is how you should do policy research. More I am trying to draw out a picture of how the research process often plays out due to the circumstances in the region and within that framework, how to optimise the outcomes. The following 7 points build on each other to elaborate and somewhat justify the approach, as well as offering guidance along the way:

 

1. Answering the questions being asked by clients

Social science research is often said to be common sense in a tuxedo! Besides the tuxedo part, my interpretation of this phrase refers to the idea that researchers should basically be deploying the right kinds of approaches to answer the questions being asked, e.g. if the question is one about a specific demographic and their response to something, then guess what – we should probably try to collect as much insight from the target group in some kind of survey. In policy research, we have clients who are commissioning and setting the research agenda, and due to the fact that policy problems in the region tend to be under studied, the questions that clients ask tend to be more like: ‘How can I better understand why policy x is not working?’, rather than something specific like ‘What are the outcomes of policy x over the last 2 years?’. And even though clients are asking very broad questions that require in-depth analysis, the funding available for such studies tends to be relatively small. This combination of factors (as well as limited data available and previous research to build on) has lead to a situation where many studies are necessarily smaller case-based research in which we are trying to dig deep into representative examples, tracing the process from initial policy goals to implementation in an attempt to evaluate current policy approaches and identify what is not working.

 

2. Employing a Flexible Design

The inherent messiness of the situation and lack of established thinking in the corpus of policy research in the region often simply rule out fixed designs from the beginning3 . A more fruitful approach is often the adoption of a flexible or hypothesis generating (rather than testing) where we start with out own informed hunches, but allow what we find in the research to modify or even change our thinking. In my workshops, I often ask what do researchers normally feel after they collect their data? And the answer is…………confused! Or in other words they don’t find what they expect or it is much more complicated that they first expected. These experiences would indeed support the adoption of a flexible design.

 

3. Adopting an evidence maximising approach

Although this is often the situation, a flexible design in our case does not equal a qualitative only approach. We normally adopt a mixed methods approach, as we are simply trying to make the study as empirical as possible (in a data light environment) and so, work with as much evidence as is available, whether qualitative and quantitative. In my experience, this often results in studies where we frame in the quantitative data available and dig deeper into a case or cases in a more qualitative manner. The constant challenges between the quantitative and qualitative in academic disciplines is not helpful in this discussion – we need to be a lot more pragmatic and empirical.


4. Employing a source maximising approach

Although there may be little rigorous research available, even in transition contexts, there is a lot of commentary and evaluation of policy issues out there from international organisations, NGOs, political parties, government bodies and the media. While of course staying mindful of the reliability of sources, we should try to draw on all sources available to build up a picture of what has been done and said to date to get the fullest picture of the narrative around the policy and what questions people have already tried to ask and answer. This will help us immensely to focus and target the research on what really is the current missing policy information we need to focus on in the research. It will also help us to make the focus of the research a lot more policy relevant, i.e. asking questions that others in the policy community are also asking.

 

5. Looking deep in the context, not wide in the phenomena

Like all policy research the focus is on looking at cases of problems, processes and institutions and trying to suggest recommendations for that case, rather than using a case to understand theoretical phenomena better. This means taking a multidisciplinary approach and given that you want to map out a complex issue with little other research evidence available, it normally means looking deep into a limited amount of cases, e.g. 2/3 municipalities, a ministry, 5 schools or a service delivery process in 2 or 3 towns.

 

6. Leaving the capital city and getting the real data

When taking on a policy research project, it is generally not enough to talk to the usual suspects in the capital and repeat the commonly held problem description and what is thought to be the best practice solution. We should be trying to investigate public policy from the original idea to its delivery in an evidence-driven fashion. Tracing the process through political decisions to administrative implementation to citizen experience will bring new thinking and new evidence and the kind of complexity that is truly needed to develop smart policy change that fits the context. 

 

7. Rigorously planning to defend the research from day 1

Adopting more flexible designs is often taken for a less scientific or more fuzzy approach, which may indeed be the case. However, in a policy research arena, your own opinion on the suitability of the research method is generally not enough, as you are trying to convince the broader policy community to act on your recommendations. In my experience of advocacy for policy research, the starting point for research-based advocacy is that other experts question all elements of the methods. Only after the validity of the methods and evidence generated have been accepted will the discussion move onto the policy implications that the study has suggested. Therefore, we must prepare to defend the validity, reliability and transferability of the research approach from day one of the planning. 

 

In the approach we present, within a process tracing4 , flexible design approach, while we indeed allow for evolving thinking, it still needs to be rigorously planned. The key element is to identify what data you will collect and from where, before you talk about data collection methods. As such, we get researchers to come up with a working hypothesis, identify variables and supported indices to measure the identified variables and justifiable cases from which to collect the data5 . Even if in the initial stages of the research needs to be exploratory or scoping out the situation, we need to clear when the scoping stage is over and then the real planning begins. In fact, I have often seen policy research projects from the region that get stuck in this scoping stage, which just seem to bloat out and out and never really develop a basis for focused recommendations.

In this post, I have attempted to elaborate the main lines of thinking that drive our thinking on putting together a sensible research design in an environment that really limits your research choices. I am sure there are other views on the points made here and views on possible approaches that fit the challenges in transition (and developing) countries. We would be delighted to hear from you and broaden a discussion on the practical challenges and smart responses.