«

»

Research and evaluation in psychotherapy and counselling

Print Friendly, PDF & Email

Return to Journal Articles


Robert King[1]Professor, School of Psychology and Counselling,Queensland University of Technology.

Why research?

There are two reasons why organisations and agencies that provide psychotherapy or counselling, and professional associations for psychotherapists and counsellors should be actively engaged in research. The first is that research generates information that can usefully inform practice. In this respect, research can be understood as a form of quality assurance or quality improvement activity. The second is that engagement in research signals a commitment to the values and principles of evidence based practice, thereby conferring legitimacy. As a general rule, the weaker the research base of the practice area, the greater the imperative for research. Associations that support and promote practices that have little or no evidence-base have an obligation to contribute to the development of an evidence-base. Organisations providing services have an obligation to the public to ensure the services they provide are effective.

Resistance to research

Notwithstanding these imperatives, there are a range of rationales that individuals and organisations give for not undertaking research. These include:

  • Rejection of the research paradigm. This argument is often couched in terms of research objectifying people, being incompatible with humanistic or spiritual approaches to the person, or being fundamentally unethical;
  • Belief that the processes or outcomes specific to psychotherapy cannot be measured; and
  • Lack of expertise or resources.

None of these rationales stand up to scrutiny. The first is based on ignorance about research. There are many approaches and traditions in research. Research does not necessarily rely on objective measures and there will almost certainly be a research approach that has a good fit with the values and ethics of any given professional association. Engaging in a dialogue with researchers who do use objective measures will usually reveal that they are very aware of the limitations of those measures. The second is typically based on a false belief that measures are limited to indicators of symptoms and functioning. With a little imagination and some hard work it is usually possible to find, or if necessary devise, measures that are sensitive to a wide range of other processes or outcomes, whether they be mindfulness, spiritual awareness or creativity. The third may be a realistic barrier if considered within the resources of the service provider or professional organization but is one that can be overcome through collaboration. This will be discussed in greater detail later.

Fear of research

If the reasons for resisting research have no foundation, why do they persist? In my experience many practitioners are uneasy about research and evaluation. At a personal level, it is the scrutiny – the sense of someone looking over the shoulder. However irrational it may be, most therapists harbour a suspicion they are not quite good enough and that any kind of systematic enquiry will show this. Furthermore, many therapists are quite invested in their particular therapy approach. Even if they suffer from personal self-doubt, they hold on to the value of the therapy itself and hope that this will get them through. Evaluation of the therapy is a no-win scenario. If the therapy is found to be effective it only confirms what they already believe and at best enables them to hold their head a bit higher when in the company of therapists from ‘evidence-based’ traditions (who they usually steer clear of anyway). If the therapy is found to be ineffective, something they hold dear has been damaged.

Fear of research is a reality and it means that anyone proposing research that depends on cooperation of therapists (which most does) has to factor in this fear. Therapists will not automatically support research and may even passively or actively resist it. This is not a reason to pull back from research but it does mean that organisations and researchers need strategies to bring clinicians on board. The following strategies are likely to help.

  • Manage therapist fears. The dominant fear is unfair personal evaluation. Explain that the focus is not on the therapist but on overall outcomes for clients undertaking therapy. Emphasize that it is likely that individual therapists will have different outcomes, not because of the quality of the therapist but because of the clients they happen to be treating. Make it clear that no judgments will be made about the therapists based on research findings. A secondary fear is that the research will disrupt the therapeutic relationship – for example, that clients will resent having to fill out questionnaires and terminate therapy. Explain that clients are not under any duress to take part in research and that part of the informed consent process requires that clients understand they will not be denied services if they decide not to participate. As a general rule, clients do not find filling out questionnaires a burden. In fact, they often find the process quite helpful as it gives them cause to reflect on how they are travelling.
  • Appeal to altruism. Therapists are mostly altruistic people – it is one of the core motivators for the profession. Explain that while research is unlikely to confer benefits to the immediate participants, it is in the interests of future clients. Research showing that therapy is effective increases the likelihood of referral, maximizes any placebo effect (and in most therapy there is quite a large placebo effect) and increases the likelihood of third party support, such as Medicare rebates making therapy more accessible for clients. Research may also identify ways by which the therapy can be enhanced, meaning future clients will have the benefit of higher quality therapy.
  • Point to the likelihood that findings will affirm therapy. There is now a large body of research into the effectiveness of psychotherapy and counselling and the findings indicate that all therapies are effective and valued by clients – and in approximately equal measure. It is unlikely that an evaluation of a new therapy will find differently.
  • Acknowledge burden. The reality is that there is unlikely to be much if any personal benefit to the therapist and there may be some cost such as completion of forms or questionnaires, or remembering to collect questionnaires from clients. Look at ways of minimizing any costs and if possible compensating by reducing some other kind of administrative burden for the duration of the research.
  • Promote the status enhancing value of research. Being part of an organization engaged in high quality research can add status to the organization and its therapists. Make this clear in both internal and external communications. Enhanced status is probably the only personal benefit likely to accrue to therapists so it is important to make the most of it.
  • Make commitment to research part of the organization’s strategy and work process. It is important that the organization ‘owns’ the research, even if a specific study is being managed by an outside body such as a university. Therapists who work for the organization need to understand that participation in research is part of their job and not an optional extra. This should be made clear when therapists join and reinforced on a regular basis. Participation in research should be clearly linked to core organizational values and goals. Decisions to undertake research or participate in research should be widely discussed within the organization, benefits and costs identified, and any practical difficulties resolved. Once a decision has been made to engage in a research project, it should be treated like any other activity and properly quality managed.

Collaboration

While some organizations have ‘in-house’ capacity for research and evaluation, there is enormous mutual benefit when service providers collaborate with universities. Universities bring expertise, energy and resources. Service providers bring therapists, clients and the setting. In the first instance, the resources are likely to consist of one or more students undertaking research projects. However, the development of a successful collaboration may ultimately, through research grants, bring additional therapists or other useful resources.

Collaborations are most likely to be successful when both parties have a clear vision of what they want from the collaboration but are also prepared to be flexible and negotiate to achieve mutually satisfactory outcomes.

I have a number of tips for service providers to get the most out of collaboration:

  1. Be proactive. Approach universities and set the agenda rather than waiting for someone to approach you.
  2. Target individuals and schools or departments. Most universities have well-organized websites with good search capacity. Using terms like psychotherapy or counselling (or even a specific form of therapy) will identify people with an interest in your field. Psychology usually has a strong focus on quantitative research but may also have qualitative expertise. Qualitative expertise is often found in schools of Social Work and Nursing.
  3. Contact research coordinators and honours student coordinators. Each year there are hundreds of students looking for projects. Coordinators have the responsibility of helping students to find suitable projects. They will be very interested to hear about your research interest.
  4. Plan ahead. Most collaborations take a year or two to bear fruit. Don’t expect things to happen quickly.
  5. Be open to projects that don’t quite meet your needs or expectations. Building a collaborative relationship is important. As long as you are working towards your project, it may be worth contributing to something that does not quite meet your needs.
  6. Seek formal standing. If someone from your organization is granted an honourary or adjunct position in the university, that person will have access to university resources such as library facilities. These can be extremely valuable.

The research process

Whether research is internal or in collaboration with an external agency, the processes are much the same. Understanding the processes is important whether the research is undertaken in-house or in collaboration with a university.

Clarify the research question. In psychotherapy and counselling there are outcome questions, process questions and process-outcome questions. Outcome questions concern the effect of the therapy. Do clients resolve their problems or difficulties as a result of the therapy? Do clients’ symptoms improve as a result of the therapy? Are client relationships with partners more satisfactory as a result of therapy? Process questions concern what happens during the therapy. Was the therapeutic alliance strong? Did clients practice mindfulness? Did the therapist deliver the therapy in the standard way? Process-outcome questions investigate the relationship between therapy processes and outcomes. Did clients who regularly practiced mindfulness show greater reduction in symptoms? Did clients who reported better therapeutic alliances resolve problems to a greater degree? Did couples therapy result in better relationships between partners when conducted strictly in accordance with the manual?

As a general rule, it makes sense to start with outcome questions. Until we are clear about outcomes it does not make much sense to investigate processes or process-outcome questions. Once outcomes are well established, investigation of processes and process-outcome becomes important to understand what it is about the therapy that makes a difference. This kind of research can lead to improvements in therapy technique.

Find out what is already known. The research question determines the focus of the research but not much else. The next step is to find out what is already known. This is usually done by means of a literature review, which is increasingly easy. Access to a good university library helps but is not essential. Medline (Pubmed) is freely available and is the single most useful resource for accessing research into psychotherapy and counselling. Medline has good search features and gives access to abstracts. In some cases these lead to full access. PsychInfo is also a very good database but is not freely available. The Google search engine does interface with Medline but it is a rather inefficient search process as it turns up a lot of low quality information as well. Google Scholar is useful as it provides access to full text articles and books and not just abstracts. However, the range is limited and the articles that turn up on a search are often old. Using Medline, supplemented by Google will give a good indication of the scope of work done in the area and provide some pointers to the approach taken and the quality. In some cases it will actually answer the question.

Address Ethical Issues

Most but not all research will be subject to formal ethical review. Ethical review is desirable and is one of a number of benefits of collaborating with a university. Therapists who feel ambivalent about, or antagonistic towards, research may query the ethical standing of the proposed work. Having the research reviewed by a well-constituted ethical review process will address these concerns. In general, ethical review will consider questions such as:

  1. Do the possible benefits justify the effort? This includes issues such as whether the research is well enough designed to answer the key questions. Useless research will usually be considered unethical.
  2. Will participants engage in the research voluntarily and with a clear understanding of what they are engaging in? There should be no coercion of any kind and no exploitation of vulnerable people. Information about the research must be in plain language and explain procedures and time demands adequately.
  3. Is the privacy and confidentiality of participants respected? This goes especially to collection and storage of sensitive information. Risks of unauthorized people obtaining access to information must be effectively managed.
  4. Are there any risks of participation beyond the normal risks of everyday life? Completing a questionnaire or being interviewed may cause distress. Some therapies may expose people to risks. What measures are in place to manage any risks?

When the work is not subject to formal review, such as in a quality assurance project, the research or evaluation team must address these issues.

Design the Study. A new study should address a question that we do not already have the answer to. In the case of therapies that have little research history, it may be a very broad question such as ‘is this therapy effective?’ In the case of therapies that have been more thoroughly investigated, it may be a much more specific question, such as is this therapy effective with this or that particular problem, or this or that particular population. Alternatively, the focus may be more on process or process-outcome issues.

The form and methodology of the study will depend on factors such as resources and expertise. Studies may be qualitative, quantitative or use mixed methods. All have advantages.

Qualitative studies are good for intensive exploration of small samples. They are especially important when the characteristics of key variables are not well understood or when there are not reliable and valid measures for key variables. They can be very helpful in exploring process issues. Qualitative studies tend to be time consuming, more demanding on participants, and data analysis is difficult. They also have less impact on decision makers because they are by nature exploratory, and tend not to provide definitive answers to questions. Qualitative studies are not recommended unless the researchers have substantial expertise in this approach.

Quantitative studies use standardized measures to quantify outcome variables such as symptoms, quality of life or relationship satisfaction, or process variables such as the therapeutic alliance, homework compliance or fidelity to treatment. There are two widely used designs: uncontrolled repeated measures and controlled repeated measures.

Uncontrolled repeated measures studies have a single sample with all participants receiving the treatment. Participants’ status on variables of interest is measured at various intervals. At a minimum, measurement is prior to treatment and post treatment, but there may also be measurements during treatment and at follow up. The effect of the treatment is inferred from changes in scores after treatment compared with before treatment. Process measures such as the therapeutic alliance can be administered during treatment to evaluate change over the course of treatment, or to investigate the relationship between process and outcome. Uncontrolled studies are less credible than controlled studies because it is possible that any improvements may have occurred without the treatment. However it is often possible to benchmark treatment effects for uncontrolled studies with effects reported in the literature, which enhances credibility.

Controlled repeated measures studies use the same approach (explained above) except that there is a control sample group who do not receive the treatment but who complete the same measures as the study group. The control sample may be on a waiting list and receive treatment after an interval, or they may receive an alternative treatment or ‘treatment as usual’ which may be something like routine case management. It is always best to randomly assign participants to the treatment and the control so as to minimize risk of bias in assignment. Controlled studies (and especially randomized controlled studies) have greater credibility than uncontrolled studies.

Quantitative studies are relatively easy to implement so long as a few basic principles are followed:

  1. Use measures that have sound psychometric properties and a good track record in research. It is acceptable to use some novel measures so long as there is a core of sound measures. Fortunately there are plenty of freely available measures of acceptable quality. Our recently published Manual of Psychosocial Rehabilitation (King et al., 2012) has a substantial section devoted to such measures.
  2. Ensure measures are administered properly. Missing data can compromise quantitative studies, and studies that rely on therapists administering measures are especially vulnerable. I recommend administration of measures through client reception or some other administrative process rather than through therapists. Therapist administration may bias scores but the greater risk is non-administration.
  3. Take care with data storage and data management. Most research will be subject to ethical review, which will ensure sound data management processes. However, internal research with a quality assurance focus may not need ethical review and in such circumstances it is incumbent upon the organization to ensure that original questionnaires are securely stored, and that when data is transcribed to a database that identifying features other than a participant code are removed.

Data analysis. A study design gives consideration not just to how data will be collected but also how it will be analyzed.

Data analysis processes for qualitative research are highly variable. There is a common aim of extracting themes and ideas but no single procedure for doing this. The aim is to distill the meaning of the information. The risk is that the meaning will be distorted by the subjectivity and biases of the researcher. A range of strategies have been developed to maximize the extraction of meaning and minimize bias, but there is no consensus as to the best method or methods to employ. In my opinion, it is best not to attempt analysis of qualitative data without the assistance of someone with expertise in one or more of the established methods. Furthermore, it is better not to wait until the data analysis phase to bring in expertise because the approach to data collection and the approach to data analysis are often intimately linked.

Quantitative data analysis can be technical but is relatively straight forward because there are well developed and widely accepted statistical procedures. There are more and less sophisticated approaches but quite a lot of useful analysis can be done using commonly available databases such as Excel or online programs. Many health professionals have sufficient basic training in statistics to conduct this kind of analysis.

Practice-based Evidence

There is growing interest in building data collection into routine clinical practice. Michael Lambert’s OQ system (http://www.oqmeasures.com/) and Len Bickman’s CFS (http://www.cfsystemsonline.com/) are both examples of well designed systems that provide feedback to therapists about client progress. They also compile aggregated data summaries that provide information about outcomes for the organization. Both systems charge users, but the fees are modest.

Typically, clients complete standardized measures of outcome and/or process each session. Both OQ and CFS offer a suite of well developed measures. The advantage of routine outcome measurement is that data collection is built into the standard operation of the service. Data is available for answering specific research questions if and when required.

This approach has two clear benefits. The first and most obvious is that the ‘research’ informs practice on a case-by-case basis. The therapist receives information which can be used to adjust treatment or manage ruptures in the therapeutic alliance. The available research suggests that this feedback does have a positive impact on treatment although with modest effect size. The second is that embedding data collection within clinical processes means that the service can engage in ongoing data collection, which is available for research, without the intrusions and disturbances associated with episodic research projects.

Conclusion

Research is central to quality improvement and provides a means by which services or associations can demonstrate commitment to evidence based practice. It is also a means by which new or untested therapies can establish their effectiveness. However, research is likely to generate anxieties among therapists and make demands on organizations. It is important to acknowledge these impacts and to make use of strategies to manage them. Collaboration with external agencies, especially universities, is an important means of bringing research expertise to an organization. Embedding routine data collection into organizational practice has the potential to both yield clinical benefits and facilitate research activities.

Reference

King R, Lloyd C, Meehan T, Deane, F & Kavanagh D. (eds) (2012) A Manual of Psychosocial Interventions. Oxford: Blackwell-Wiley.

 


Return to Journal Articles