Efficacy indicators of four methods in outpatient addiction treatment

Abstract

Background

In Portugal, as far as we know, there are no recent studies that evaluated the comparative efficacy of therapeutic modalities in addiction problems by reference to a holistic and psychosocial model of effectiveness.

Objectives

Using a sample of Portuguese patients in outpatient treatment for drug and alcohol abuse, this study aimed to examine if a combined treatment modality (group therapy with individual intervention) had greater overall efficacy when compared to other three types of treatment without group therapy.

Methods

This is a correlational and cross-sectional study using a convenience sample of patients (N = 254) from an outpatient treatment in the Intervention Service on Addictive Behaviors and Substance Dependence. At the time of data collection, the patients were attending four types of treatment, such as receiving intervention based on individual psychological counseling (n = 66); receiving individual psychiatric counseling (n = 68); receiving both individual psychological and psychiatric counseling (n = 102); and receiving not only individual counseling (i.e., psychology or psychiatry), but also attending group therapy (n = 18).

Results

Using MANOVA and Wilks’s multivariate test criterion, there was a significant effect of treatment modality on the global efficacy, Λ = 0.88, F(9, 603) = 3.75, p < 0.0001. Examination of mean estimates indicated that patients in a combined therapeutic modality revealed more treatment involvement compared to patients in other therapeutic modalities without group therapy.

Discussion

The results obtained in this study highlight the importance of integrating interventions in a collaborative way. A combined therapeutic modality, adding group therapy, was associated with positive effects, such as more levels of peer support and involvement in treatment, and increasing the individual’s probability to remain abstinent.

Addiction; outpatient treatments; group therapy; treatment effectiveness evaluation

Introduction

Outpatient treatments for drug or alcohol dependence can include a variety of treatment methods, such as individual or group therapy. The literature suggests that these specific types of treatment are more effective as more they take into account the diversity and the patients’ deficit areas, through an eclectic and systematic approach¹. It is not just a question of finding a good match between patient and treatment modality², but of acting in a concerted manner upon the various deficit areas of the patient^3,4. It is an established fact in the scientific community that a combined and integrated treatment by itself increases the likelihood of obtaining better results not only in addiction problems^3,4, but also in various treatments for mental disorders¹.

However, some studies suggest that certain therapies and therapy combinations in treatment programs seem to increase the success of interventions in the treatment of addiction. This is the case of group therapy that has been considered by some authors as the intervention of choice in the treatment of addictions^5,6. Group therapy increases the perception of better peer support, being a privileged tool of persuasion, stabilization and social support in the treatment of addiction⁷. The positive social support and peer support help to maintain abstinence⁸, and improve treatment outcomes⁹. Group therapy has also been associated with the effectiveness of the treatment in psychological distress symptoms^10,11 and has been found to be effective in reducing the negative affects¹². Additionally, group therapy also provides social, emotional, and relational skills in order to increase their perception of being accepted and accept others, as well as dealing with emotional states in the relationship with others¹³. Some factors that arise from the group therapy include catharsis, cohesion, interpersonal learning¹⁴, affiliation, confrontation, support, gratification, and identification, which seem to promote the involvement of patients in their treatment^5,15, and promote a better prognosis¹⁶. These factors promoted by group therapy increase the success of interventions, not only in the addiction problems, but also in psychotherapy in general¹⁷.

Given the advantages of group therapy, some studies specifically seek to understand if a combined modality of individual and group therapy would increase the probability of success of treatment programs. In fact, some literature corroborates this view stating that group therapy and individual therapy seem to complement each other in addiction treatment. The patients appear to be more likely to remain engaged in their recovery process if they have been involved in group therapy¹⁵. In this sense, some authors confirm that a combined intervention of an intensive individual counseling (individual drug counseling – IDC) added to group counseling (group drug counseling – GDC) increases the likelihood of effective treatment in cocaine dependence, raising abstinence rates¹⁸.

However, as far as we know, there are no studies in Portugal that have evaluated the comparative efficacy of therapeutic modalities by reference to a holistic and psychosocial model of effectiveness. Considering the previous discussion, this study aimed to examine, in a sample of Portuguese patients in outpatient treatment for drug and alcohol abuse, if a combined treatment modality (group therapy with individual intervention) had better outcomes, when compared to other three types of treatment without group therapy: group receiving intervention based on individual psychological counseling; group receiving intervention based on individual psychiatric counseling; and group receiving intervention based on individual psychological and psychiatric counseling. The groups were compared in terms of some indicators of efficacy that are described in the literature¹⁹ (i.e., abstinence, involvement in treatment, social and peer support, psychological distress symptoms and negative affects). We expected that patients who attended the combination of both individual and group therapies reported lower rates of psychological distress symptoms and negative affects, as well as higher rates of abstinence, involvement in treatment, and social and peer support, than other patients involved in individual therapies.

Methods

Participants

The present study is part of a larger research project funded by Fundação para a Ciência e Tecnologia (Foundation for Science and Technology – Portuguese and European funding) on the study of 608 Portuguese patients in treatment for drug addiction and alcohol abuse. For this study, as inclusion criteria, the participants had to be on outpatient alcohol and drug treatment (including the attending of at least one of the four treatments described below) and age over 18 years old. Exclusion criteria were having apparent psychosis, intoxication, or having mental retardation, in order to ensure that informed consent could be given. The participants who meet these criteria were 254 patients, male (85.8%) and female (14.2%) with ages range from 18 to 73 years; mean age for the sample was 41.58 (SD = 8.71), recruited from outpatient treatment programs at the Intervention Service on Addictive Behaviors and Substance Dependence (SICAD) - Regional Northern Section. There were no differences between the included vs. excluded participants in terms of age (t(603) = 1.65, p = 0.099) and gender (χ²(1) = 0.340, p = 0.56). At the time of the data collection (between January 2012 and May 2013), the participants were attending different four types of treatment for drug and alcohol, such as receiving intervention based on individual psychological counseling (n = 66) (therapeutic modality I), where 92.4% of this subsample was men, with ages ranging from 20 to 56 years (M = 41.09; SD = 7.08); receiving individual psychiatric counseling (n = 68) (therapeutic modality II), where 77.9% of this subsample was men, with ages ranging from 28 to 56 years (M = 41.96; SD = 6.85); receiving both individual psychological and psychiatric counseling (n = 102) (therapeutic modality III), where 88.2% of this subsample was men, with ages ranging from 18 to 66 years (M = 40.88, SD = 10.24); and receiving not only individual counseling (i.e., psychology or psychiatry), but also attending group therapy (n = 18) (therapeutic modality IV), which 77.8% of this subsample was men, with ages ranging from 33 to 73 years (M = 45.94, SD = 24.10). This was a cross-sectional and correlational study with a convenience sample, which no randomization of the participants by groups was made. The group treatment allocation was performed after the data collected considering the type of treatment that the patients were receiving. Further details about the descriptive analysis and the homogeneity of the 4 groups can be found in Table 1.

Table 1
Descriptive statistics of the sample

	Therapeutic Modality I (n = 66)	Therapeutic Modality II (n = 68)	Therapeutic Modality III (n = 102)	Therapeutic Modality IV (n = 18)
Psychophathology symptoms in time of questionnaire administration (%)
(≥ 1.7) (%)	63.6	60.3	55.9	55.6
Treatment retention (months)
M	29.81	29.52	26.35	30.79
SD	21.04	23.86	21.93	17.1
Comsumption patterns in time of questionnaire administration (%)
Abstinence	41.6	55.2	55.9	77.8
Recreational use	35.4	28.4	18.6	22.2
Regular use	16.9	9.0	12.7	0.0
Substance abuse	4.6	6.0	7.8	0.0
Dependence	1.5	1.4	4.9	0.0
Comsumption patterns prior to treatment (%)
Regular use	0.0	1.5	2.0	0.0
Substance abuse	3.1	13.8	14.7	16.7
Dependence	96.9	84.7	83.3	83.3
Primary substance of abuse
Heroine	51.5	26.5	34.4	11.1
Cocaine	4.5	22.1	17.6	5.6
Alcohol	4.5	19.1	28.4	83.3
Hashishe	1.5	0.0	2.9	0.0
Polydrug use	38.0	32.3	15.7	0.0
Psychopharmacotherapy	0.0	0.0	1.0	0.0

Measures

A Socio-Demographic Questionnaire^20,21 collected information about gender, age, birthplace, treatment program, primary substance of abuse, treatment time, pattern of substance use and abstinence. The outcome variable abstinence was operationalized as follows: dichotomous variable (“absence of substance use” = 0 and “substance use” = 1) defined in terms of the presence vs. absence of substance use during treatment. The minimum time during which the individual restrained from using substances was of at least 1 month.

Brief Symptoms Inventory (BSI)²². This is a reduced version of the Symptom Check List 90 Revised (SCL-90-R), a self-report inventory consisting of 53 items. The individual must specify the degree to which each problem has affected them over the past week, on a Likert-type scale (1-Never, 2-Rarely, 3-Sometimes, 4-Often, 5-Very often). This inventory assesses psychological distress symptoms along nine basic dimensions (somatization, obsession-compulsion, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation and psychoticism) and three global indices (positive symptoms distress index – PSDI, global severity index – GSI, and positive symptom total – PST). In the current study, the various dimensions have an internal consistency ranging from moderate to high, with Cronbach’s alpha values between 0.67 to 0.86. The internal consistency of the overall BSI in this sample was 0.96.

Monitoring and Evaluation of Effectiveness and Progress (MEEP) –Portuguese short form^20,21. A short Portuguese measure (MEEP) with psychometric qualities demonstrated²¹, covers 4 overall factors and 22 items: (a) Therapeutic Progress subscale (α = 0.88) – includes items intended to measure treatment engagement, treatment satisfaction and counselling rapport; (b) Negative Affects subscale (α = 0.81) – includes items that measure the psychological functioning; (c) Social support subscale (α = 0.83) – seeks to measure the perception of social support received; and (d) Peer support scale (α = 0.85) – includes items that measure the perception of social support received by peers. On average, it takes 15-20 minutes to complete the questionnaire properly. The response format is a 7-point Likert scale (1 – I strongly disagree to 7 – I strongly agree). The internal consistency of the overall MEEP-22 in this sample was .81.

Global Efficacy Measure was defined by a composite variable that included 3 components – absence of substance abuse (defined in terms of the absence or presence of substance abuse during treatment), psychological distress symptoms below the cutoff (ISP < 1.7) and high levels of involvement in treatment (PT ≥ 6). These different components were evaluated through the Socio-Demographic Questionnaire^20,21; the Positive Symptoms Index (PSI) from BSI (Portuguese version of Canavarro, 1999)²² and the MEEP^20,21 based on TCU-CEST²³. Higher scores reflect higher indicators of treatment efficacy.

Procedure

A formal contact with SICAD was established to obtain permission to conduct the investigation, as well as with the ethics committee of the Faculty of Psychology and Education Sciences (University of Porto). Professionals of these institutions made the first contact with the participants, where a general explanation of the study was provided to them, and then they were asked if they agreed to participate in the study. Written and verbal consent were obtained from each participant. To ensure the confidentiality of all participants, their names and personal information have been coded. The questionnaires were administered in an interview format by trained female psychologists at the facilities of SICAD (Regional Northern Delegation), in a private and quiet room.

Data analysis

Data analyses were carried out using the SPSS version 18 for Windows (United States, New York, IBM Corporation). Descriptive statistics were calculated to characterize the study variables. We used Chi-square analyses to examine differences among groups in terms of the dependent variable abstinence and Kruskal-Wallis test to explore differences among groups in terms of treatment retention. We used MANOVA and Wilks’s multivariate test criterion to examine differences among groups in terms of the dependent variables treatment involvement, negative affects, psychological distress symptoms, social support, and peer support. Pearson correlation analyses was conducted previously to verify if the dependent variables were correlated among them. Finally, we used a Mann-Whitney test to confirm the findings from MANOVA due to the lack of normal distribution to conduct parametric tests, with a Bonferroni correction by reducing the p value to < 0.0125, adjusted for four groups.

Results

Further details about the descriptive analysis can be found in Table 1. Chi-square analyses revealed no significant associations between the four intervention modalities in terms of previous severity of the addiction (χ² = 7.94, p = 0.242). Kruskal-Wallis test revealed no significant differences between groups in terms of treatment retention, χ²(3) = 2.62, p = 0.454. Considering the time interval between sessions, this interval ranged from weekly to monthly for all individual therapeutic modalities I, II and III (69% to 87% of cases). In combined therapeutic modality IV, this percentage was 75% of cases. Chi-square analyses also revealed that patients in therapeutic modality I reported less drug therapy (42%, n = 27), compared to patients in therapeutic modality IV (77.8%, n = 14), therapeutic modality II (65.7%, n = 44), and therapeutic modality III (58%, n = 57) (χ² = 11.02, p = 0.012).

Chi-square analyses shown that patients in therapeutic modality IV significantly reported more abstinence (78%; n = 14) than patients in other treatment program modalities, including therapeutic modality III (56%; n = 57), therapeutic modality II (55%; n = 37), and therapeutic modality I (42%; n = 27), (χ² = 8.34, p < 0.05). Pearson correlation analyses showed that peer support was significantly correlated with social support (r = 0.19, p = 0.002). Treatment involvement was significantly correlated with social support (r = 0.49, p < 0.0001), and peer support (r = 0.13, p = 0.03), but was not significantly correlated with negative affects (r = -0.02, p = 710). Negative affects was also not significantly correlated with peer support (r = -0.02, p = 740). Similarly, psychological distress symptoms was not significantly correlated with treatment involvement (r = -0.09, p = 0.13), and peer support (r = -0.02, p = 70). Considering these results, the variables negative affects and psychological distress symptoms were not included in multivariate analyses.

Using MANOVA and Wilks’s multivariate test criterion, there was a significant effect of treatment modality on the global efficacy, Λ = 0.88, F(9, 603) = 3.75, p < 0.0001. Univariate tests showed statistically significant effects for treatment involvement (F(3,254) = 5.04, p = 0.002) and also peer support (F(3,254) = 6.15, p < 0.001). Examination of mean estimates indicated that patients in therapeutic modality IV revealed more treatment involvement (M = 52.89; SD = 2.93) than patients in therapeutic modalities I (M = 48.3; SD = 5.56) and II (M = 47.69; SD = 6.61). There were no significant differences between patients in therapeutic modality IV and patients in therapeutic modality III (p = .23). The patients in therapeutic modality IV also revealed more peer support (M = 9.56; SD = 3.24) compared to patients in therapeutic modalities II (M = 6.75; SD = 3.61), and I (M = 5.63; SD = 3.68). There were no significant differences between patients in therapeutic modality IV and patients in therapeutic modality III (p = 0.06).

We verified that all nonparametric analyses confirmed the multivariate analyses, with exception of the differences between therapeutic modality IV and therapeutic modality III in terms of peer support. Particularly, in terms of treatment involvement, the therapeutic modality IV was significantly different from therapeutic modalities I (U = 292.00, p = 0.001) and II (U = 326.00, p = 0.002), as well as the therapeutic modality IV was significantly different from therapeutic modalities I (U = 251.00, p < 0.0001) and II (U = 319.50, p = 0.002), in terms of peer support. Regarding the treatment involvement, there were no significant differences between patients in therapeutic modality IV and patients in therapeutic modality III (U = 640.00, p = 0.04). However, considering the peer support, significant differences were found between patients in modality IV and patients in modality III (U = 534.00, p = 0.004). Patients in modality IV revealed more peer support (Mean Rank = 81.81) than patients in modality III (Mean Rank = 56.74).

Discussion

This study aimed to compare a combined treatment modality, including individual counseling and group therapy, with other three treatment modalities without group therapy, in terms of efficacy indicators among Portuguese patients in outpatient treatment for drug and alcohol abuse. According to the results, we confirmed our hypothesis regarding the efficacy indicators of abstinence, involvement in treatment, and peer support. Specifically, we found that patients engaged in therapeutic modality IV (attending individual and group therapy) showed significantly better therapeutic involvement than patients in therapeutic modality I (psychological counseling) and therapeutic modality II (psychiatric counseling). Comparatively to individual therapeutic modalities (I and II), the therapeutic modality IV seems to have clear advantages in terms of patient involvement in treatment. These results were expected since this view is corroborated in literature, stating that group therapy and individual therapy seem to complement each other in addiction treatment¹⁵. The patients appear to be more likely to remain engaged in their recovery process if also have been involved in group therapy¹⁵. Group therapy is indeed considered by some authors as the intervention of choice in the treatment of addictions^5,6, given that several aspects are strengthened, particularly the catharsis, group cohesion, interpersonal learning¹⁴, affiliation, confrontation, support, gratification, and identification¹⁵. These aspects seem to promote the involvement of patients in their treatment^5,15, which in turn increase the likelihood of better patient prognosis¹⁶.

Moreover, it is possible that patients who have attended a multimodal treatment (individual and group therapy) had felt more valued for having received more assistance and/or a better service, thus increasing their satisfaction with treatment and therapeutic involvement. In fact, according to a conceptual heuristic model oriented to the addiction area^24,25, the involvement in treatment implies some essential aspects, as therapeutic relationship, satisfaction with the treatment, participation and treatment compliance^26,27. This hypothesis seems to be supported by the fact that there were no differences between therapeutic modality IV and therapeutic modality III (individual counseling in both Psychology and Psychiatry). Indeed, both the combined treatment modalities (III and IV) add at least two complementary and concerted interventions in the same treatment program. The inclusion of two complementary interventions directed to the patients’ deficit areas can promote a positive effect in treatment outcomes^3,4, specifically in the patient’s involvement in treatment.

As hypothesized, the patients in treatment modality IV reported more positive peer support compared to the patients from the other individual therapeutic modalities (I and II). This result is consistent with empirical evidence showing that treatment modalities with group therapy increase the perception of better peer support⁷, being privileged ways of persuasion, stabilization and social support in the treatment of addiction⁷. Actually, one of the central goals of group therapy is to increase in patients the perception of being accepted and accept others, and to deal with emotional states in interpersonal relationships¹³. In the addictions area, the promotion of positive social and peer support increases the likelihood of success in interventions, helping to maintain abstinence⁸ and to obtain better treatment results⁹.

Additionally, our hypothesis was confirmed because patients engaged in therapeutic modality IV reported significantly better results in terms of abstinence than those who had attended in any of the other treatment modalities, consistent with literature previously discussed^14,18. In general, in the area of mental health, empirical evidence has shown that treatment protocols which include group therapy, particularly cognitive behavioral therapy, are more effective compared to other treatment modalities^12,18. Considering that this treatment modality (adding group therapy) was associated with better results in terms of involvement in treatment and positive peer support and these factors have been associated with better treatment outcomes^16,9,15, it was also expected that abstinence would be higher in this group.

Finally, this study also confirms the results of a previous study showing that a combined intervention of an intensive individual counseling (individual drug counseling – IDC), added to group counseling (group drug counseling – GDC), increased the likelihood of effective treatment in cocaine dependence, raising abstinence rates¹⁸. According to the authors of this study, these two combined interventions may have focused their similar goals in the maintenance of abstinence, boosting the effect of both interventions¹⁸.

Limitations

First, this is a cross-sectional study, which is not the best method to assess treatment efficacy because the participants were not randomly allocated to the group treatments and we have not pre-test data. Therefore, we cannot guarantee the homogeneity in terms of outcome variables among groups before the treatments. For instance, we have no data about the patients´ health status at the time of the admission to treatment (e.g., psychological distress symptoms and negative affects). These limitations reduce the internal validity of the study when groups are compared in terms of outcome variables, and any causality relationships between the variables under study must be interpreted with care. Additionally, this research focuses on patients with different types of dependence, from either drugs or alcohol, which can be considered a limitation because increase the heterogeneity of the sample. Finally, we did not assess marital status and income and they are important markers of treatment effectiveness.

Conclusions

Despite the limitations, the contribution of this study focus on the novelty in testing a psychosocial model that conceptualizes the effectiveness of dependence treatments, identifying specific therapeutic modalities that best enhance these predictive factors of therapeutic success. It is known that most studies only use abstinence as an effective treatment criterion, or psychological and social variables alone, and not in an integrated manner. As far as we know, there are no studies in Portugal and Brazil that evaluated the comparative efficacy of therapeutic modalities by reference to a holistic and psychosocial model of effectiveness. The results of this study suggest the importance of integrating interventions in a collaborative way, showing that individual and group therapies are not mutually exclusive and that they should be integrated. A combined therapeutic modality, adding group therapy, was associated with positive effects in peer support and involvement in treatment, while increasing the individual’s probability to remain abstinent.

References

• ¹
  Beutler LE, Consoli AJ, Lane G. Systematic treatment selection and prescriptive psychotherapy: an integrative eclectic approach. In: Norcross JC, Goldfried MR, editors, Handbook of psychotherapy integration, 2nd ed. New York: Oxford University Press; 2005;121-43.
• ²
  McLellan AT, Grissom GR, Zanis D, Randall M, Brill P, O’Brien CP. Problem-service ‘matching’ in addiction treatment: a prospective study in 4 programs. Arch Gen Psychiatry. 1997;54(8):730-5.
• ³
  Schütz C, Linden IA, Torchalla I, Li K, Al-Desouki M, Krausz M. The Burnaby treatment center for mental health and addiction, a novel integrated treatment program for patients with addiction and concurrent disorders: results from a program evaluation. BMC Health Serv Res. 2013;13(1):288.
• ⁴
  McLellan AT, McKay JR, Forman R, Cacciola J, Kemp J. Reconsidering the evaluation of addiction treatment: from retrospective follow‐up to concurrent recovery monitoring. Addiction. 2005;100(4):447-58.
• ⁵
  Galanter M, Hayden F, Castañeda R, Franco H. Group Therapy, Self-Help Groups, and Network Therapy. In: Frances RJ, Miller SI, Mack AH, editors. Clinical textbook of addictive disorders. Guilford Press; 2004.
• ⁶
  Matano RN, Yalom ID. Approaches to chemical dependency: Chemical dependency and interactive group therapy – a synthesis. Int J Group Psychother. 1991;41(3):269-93.
• ⁷
  Center for Substance Abuse Treatment (2005). Substance Abuse Treatment: Group Therapy. Treatment Improvement Protocol (TIP) Series 41. DHHS Publication No. (SMA) 05-3991. Rockville, MD: Substance Abuse and Mental Health Services Administration.
• ⁸
  Matano RA, Yalom ID, Schwartz K. Interactive group therapy for substance abusers. In: Spira JL, editor. Group Therapy for Medically Ill Patients. New York: Guilford Press; 1997. p. 296-325.
• ⁹
  Akers RL, Krohn MD, Lanza-Kaduce L, Radosevich M. Social learning and deviant behavior: a specific test of a general theory. Am Sociol Rev. 1979;44(4):636-55.
• ¹⁰
  Gomes BC, Lafer B. Group psychotherapy for bipolar disorder patients. Rev Psiquiatr Clín. 2007;34(2):84-89.
• ¹¹
  Weiss RD, Griffin ML, Greenfield SF, Najavits LM, Wyner D, Soto JA, et al. Group therapy for patients with bipolar disorder and substance dependence: results of a pilot study. J Clin Psychiatry. 2000;61(5):361-7.
• ¹²
  Dolbeault S, Cayrou S, Bredart A, Viala AL, Desclaux B, Saltel P, et al. The effectiveness of a psycho‐educational group after early‐stage breast cancer treatment: results of a randomized French study. Psychooncology. 2009;18(6):647-56.
• ¹³
  Fehr SS. Introduction to group therapy: a practical guide. Routledge; 2014.
• ¹⁴
  Ahmed S, Abolmagd S, Rakhawy M, Erfan S, Mamdouh R. Therapeutic factors in group psychotherapy: a study of Egyptian drug addicts. J Groups Addict Recover. 2010;5(3-4):194-213.
• ¹⁵
  Project MATCH Research Group. Matching Alcoholism Treatments to Client Heterogeneity: Project MATCH post treatment drinking outcomes. J Stud Alcohol. 1997;58(1):7-29.
• ¹⁶
  Joe GW, Simpson DD, Broome KM. Retention and patient engagement models for different treatment modalities in DATOS. Drug Alcohol Depend. 1999;57(2):113-25.
• ¹⁷
  Grencavage LM, Norcross JC. Where are the commonalities among the therapeutic common factors?. Prof Psychol Res Pr. 1990;21(5):372.
• ¹⁸
  Crits-Christoph P, Siqueland L, Blaine J, Frank A, Luborsky L, Onken LS, et al. Psychosocial treatments for cocaine dependence: National Institute on Drug Abuse collaborative cocaine treatment study. Arch Gen Psychiatry. 1999;56(6):493-502.
• ¹⁹
  Marlatt GA, Gordon JR. Relapse Prevention – Maintenance strategies in the treatment of addictive behaviors. New York: The Guilford Press; 1985.
• ²⁰
  Moura A, Ferros LC, Negreiros JN. Evaluation and monitoring instrument: Client Evaluation of Self and Treatment. Arch Clin Psychiatry. 2013;40(4):165-66.
• ²¹
  Moura A, Ferros LC, Negreiros JN. The effectiveness of substance abuse treatment: Development of a brief questionnaire. Arch Clin Psychiatry. 2015;42(4):83-9.
• ²²
  Canavarro MC. Inventário de sintomas psicopatológicos – BSI [Brief symptom Inventory (BSI)]. In: Simões MR, Gonçalves M, Almeida LS, Editors. Testes e provas psicológicas em Portugal [Psychological tests and proofs in Portugal]. Braga: SHO/APPORT; 1999. p. 87-109.
• ²³
  Joe GW, Broome KM, Rowan-Szal GA, Simpson DD. Measuring patient attributes and engagement in treatment. J Subst Abuse Treat. 2002;22(4):183-96.
• ²⁴
  Simpson DD. A conceptual framework for transferring research to practice. J Subst Abuse Treat. 2002;22(4):171-82.
• ²⁵
  Simpson DD. A conceptual framework for drug treatment process and outcomes. J Subst Abuse Treat. 2004;27(2):99-121.
• ²⁶
  Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abuse Treat. 2002;22(4):197-209.
• ²⁷
  Greener JM, Joe GW, Simpson DD, Rowan-Szal GA, Lehman WE. Influence of organizational functioning on client engagement in treatment. J Subst Abuse Treat. 2007;33(2):139-47.

Publication Dates

• Publication in this collection
  Sep-Oct 2017

History

• Received
  17 Apr 2017
• Accepted
  15 Aug 2017