Construction and validation of an instrument for evaluating Lean Healthcare in healthcare institutions

Construcción y validación de un instrumento para evaluar el Lean Healthcare en instituciones de salud

ABSTRACT

Objective:

To build and validate an instrument to evaluate Lean Healthcare in healthcare institutions.

Method:

Methodological study conducted in three stages: 1) Instrument construction; 2) Content validity using the Delphi technique with 14 experts; and 3) Construct validation using Structural Equation Modeling with sample consisted of 113 professionals with experience in Lean Healthcare. Data collection carried out from October/2020 to January/2021 using a digital form. Data analysis performed with the SmartPLS2.0/M3 software.

Results:

Items were developed after an integrative review and divided into the dimensions Structure, Process and Outcome, according to Donabedian’s theoretical framework. Content validation in two rounds of the Delphi technique. Final instrument, after model adjustment, containing 16 items with Cronbach’s alpha of 0.77 in Structure, 0.71 in Process and 0.83 in Outcome.

Conclusion:

The instrument presented evidence of validity and reliability, enabling its use in healthcare institutions to evaluate Lean Healthcare.

Descriptors:
Nursing; Validation study; Data accuracy; Health evaluation; Health management; Total quality management

RESUMEN

Objetivo:

Construir y validar un instrumento para evaluar Lean Healthcare en instituciones de salud.

Método:

Estudio metodológico realizado en tres etapas: 1) Construcción del instrumento; 2) Validez de contenido mediante técnica Delphi con participación de 14 expertos; 3) Validez de constructo mediante Modelado de Ecuaciones Estructurales con muestra compuesta por 113 profesionales con experiencia en Lean Healthcare. La recopilación de datos se realizó de octubre/2020 a enero/2021 mediante formulario digital. El análisis de datos se realizó con el software SmartPLS2.0/M3.

Resultados:

Ítems elaborados después de revisión integradora y divididos en las dimensiones Estructura, Proceso y Resultado, según referencial teórico de Donabedian. Validación de contenido en dos rondas de la técnica Delphi. Instrumento final, después del ajuste del modelo, contiene 16 ítems con alfa de Cronbach 0,77 en Estructura, 0,71 en Proceso y 0,83 en Resultado.

Conclusión:

El instrumento presentó evidencias de validez y confiabilidad, permitiendo uso para evaluar Lean Healthcare.

Descriptores:
Enfermería; Estudio de validación; Exactitud de los datos; Evaluación en salud; Gestión en salud; Gestión de la calidad total

RESUMO

Objetivo:

Construir e validar um instrumento para avaliar o Lean Healthcare nas instituições de saúde.

Método:

Estudo metodológico realizado em três etapas: 1) Construção do instrumento; 2) Validade de conteúdo pela técnica Delphi com 14 especialistas; e 3) Validade de constructo por Modelagem de Equações Estruturais, em amostra de 113 profissionais com experiência no Lean Healthcare. Coleta de dados realizada de outubro/2020 a janeiro/2021 por formulário digital. Análise de dados realizadas com o software SmartPLS2.0/M3.

Resultados:

Itens elaborados após revisão integrativa e divididos nas dimensões Estrutura, Processo e Resultado, conforme referencial teórico de Donabedian. Validação de conteúdo em duas rodadas da técnica Delphi. Instrumento final, após ajuste do modelo, contendo 16 itens com alfa de Cronbach de 0,77 em Estrutura, 0,71 em Processo e 0,83 em Resultado.

Conclusão:

O instrumento apresentou evidências de validade e confiabilidade, permitindo seu uso nas instituições de saúde para avaliar o Lean Healthcare.

Descritores:
Enfermagem; Estudo de validação; Confiabilidade dos dados; Avaliação em saúde; Gestão em saúde; Gestão da qualidade total

INTRODUCTION

After World War II, Japanese manufacturers faced shortages of material, financial and human resources. To counter this scenario, Toyota leaders, Eiji Toyoda and Taiichi Ohno developed a disciplined process-oriented management system known as the Toyota Production System (TPS), which was called Lean from 19911^,2. Lean is a management philosophy that was adapted for use in the healthcare sector, called Lean Healthcare and its first publications in this area emerged in the United Kingdom in 2001 and in the United States in 2002. Lean Healthcare presents an expectation of reducing costs and optimize resources by reviewing processes to generate value for the patient/user3.

Changes are constant in the healthcare field and its main objective is to achieve quality care4. In this systematic management approach, the objective is to do more with less, that is, organizations must seek to develop high-quality services or products, avoiding waste, reducing costs and time to perform the service5. The three terms used to describe waste in Lean are: 1) Muda - activity that consumes resources without creating value for the customer; 2) Mura - variation in the process that makes control difficult and generates intense peaks of work and, later idle moments; and 3) Muri - overloading of people and equipment1.

Lean Healthcare provides tools and practices that enable professionals to think and develop solutions to improve the efficiency, quality and sustainability of the organizations they are part of6. In its management model, it values all professionals involved and seeks the root cause of problems together with those who most implement actions. Professionals are encouraged to reveal situations that require improvements, as this is the first necessary condition to fix them. This allows professionals to participate in solving problems within the workplace, which contributes to their professional satisfaction and, in the practice of Lean, it is up to leadership to engage the team7.

Evidence points out to positive outcomes with the practice of Lean Healthcare, such as: reduction in surgical cancellation and increase in installed service capacity5, reduction in waiting time8, reduction in hospitalization time9, increased satisfaction of patients and employees10, increased operational efficiency and time optimization11.

In the Brazilian scenario, the Lean Healthcare philosophy has contributed to reducing waste and improving quality in the healthcare field12^-14. Continuous improvement and maintenance of achieved results are challenges for management in the use of Lean in healthcare practice. Better results occur as the team gets involved in the change process, modifying and maturing a new way of thinking about their work process15. It is up to health management to periodically evaluate and monitor Lean to sustain the results achieved in its implementation.

The Donabedian model, based on the evidence, strengthens the healthcare evaluation process, enabling to identify the factors involved in the management models and a better understanding of service quality through existing criteria in each of the proposed dimensions of structure, process and outcome16.

There are some instruments available to evaluate Lean implementation only. For the healthcare field, the following instruments stand out: Lean in Healthcare Questionnaire (LiHcQ), aimed specifically at primary health care17⁾ and the Employee Perception to assess Lean Implementation Tool (EPLIT), which applies to the Lean implementation18. There is a lack of instruments to monitor/evaluate Lean Healthcare outcomes over time in hospital services. Given the above, the present study aimed to build and validate an instrument to evaluate Lean Healthcare in healthcare institutions.

METHOD

Methodological study conducted in three stages: 1) Instrument construction; 2) Content validity using the Delphi technique; and 3) Construct validation using Structural Equation Modeling to evaluate the measurement model and the structural model.

In the first stage, of instrument construction, a literature review was performed19, and consultation of other existing instruments, the opinion of experts and the experience of the target audience for: a) items construction ; b) distribution by dimensions, according to Donabedian’s theoretical framework16 (structure, process and outcome); c) instrument layout development; e) proposition of a bidirectional Likert response scale with five points (1=strongly disagree, 2=partially disagree, 3=neutral, 4=partially agree, 5=strongly agree)20.

The first items were developed based on the findings of the literature review19and other existing instruments. The opinion of experts and the target audience were also consulted through a focus group conducted with participants in a research group on nursing management. Three meetings lasting two hours were held, until opinions were exhausted regarding the items necessary to compose the instrument, as well as the appropriate dimension for each item, considering Donabedian’s theoretical framework16.

For the second stage of the study, using the Delphi technique, a panel of experts was formed after invitations on national digital platforms that gather professionals with experience in Lean. The inclusion criteria were: 1) Having a minimum of five years of experience with Lean Healthcare; and/or 2) Holding Green Belt certification; and/or c) Having experience with research related to instrument construction.

Two rounds of the Delphi technique were necessary, conducted remotely between January and June 2020. Originally, a feedback was requested within 14 days, but this occurred after 36 days for the first round and 22 days for the second round and the interval between rounds was 54 days.

The experts who expressed interest in participating in this second stage of the study received a formal email invitation, which outlined the objectives, and theoretical concepts adopted in the study and included files of the Informed Consent Form, the instrument, and the guidelines for completing the analysis of the instrument in the content validation process. From the 23 professionals who met the inclusion criteria, 60.8% (14 professionals) adhered to this stage and only one expert did not complete the second round of the Delphi technique.

The representativeness and clarity of each item were analyzed using the Content Validity Index (CVI). The calculation was based on a four-point ordinal Likert scale, and judges could mark the following responses for representativeness: 1 = not representative, 2 = item needs major revision, 3 = item needs minor revision, or 4 = representative item. To assess comprehensiveness, clarity and relevance, the following options were used: 1 = not clear, 2 = unclear, 3 = quite clear, 4 = very clear. Items with CVI below 80% should be reviewed, as suggested by experts21.

Finally, in the third stage, the measurement model and the structural model were analyzed for construct validation. Data collection for construct validity was carried out using the free online platform Google Forms and the data collection period was from October 2020 to January 2021. The study population consisted of members of the multidisciplinary team working in healthcare services that adopt Lean Healthcare.

The recruitment of professionals was conducted by invitations on the social medias Instagram, Facebook and LinkedIn, to groups of healthcare professionals working and researching on Lean; to members of SOBECC (Brazilian Association of Surgical Center Nurses, Anesthesia Recovery and Material and Sterilization Center - Associação Brasileira de Enfermeiros de Centro Cirúrgico, Recuperação Anestésica e Centro de Material e Esterilização) and REBRAENSP (Brazilian Network of Nursing and Patient Safety - Rede Brasileira de Enfermagem e Segurança do Paciente). The invitations included a request for widespread dissemination of the research.

It was decided to send an invitation to members of SOBECC and REBRAENSP as they are official groups with professionals working in hospital services with great potential to meet the inclusion criteria of the study. Furthermore, the use of Lean philosophy has been the subject of scientific events and discussions organized by both groups due to their focus on continuous improvement. In this context, there was a lack of any official Brazilian certification, group or platform that registered healthcare services adopting Lean Healthcare.

The inclusion criterion adopted was three months of previous experience with Lean Healthcare and no participants were excluded. The minimum sample required for validation was calculated using a significance level of 0.05, medium effect size and statistical power of 0.80, using the free software G*POWER. The resulting value was 55 cases for using the estimation model22.

For participant characterization, the following information was sought: 1) Profile of the institution in which they work (public, private or philanthropic); 2) Brazilian state of institution location; 3) Position; 4) Professional role; 5) Work area; 6) Sector in which works; 7) Age; 8) Gender; 9) Time of experience working with Lean Healthcare; 10) Time working with Lean Healthcare in current job.

The data collected in Google Forms were exported to a spreadsheet in Microsoft Excel and imported into the Statistical Analysis System® (SAS) software version 9.4. Data adherence was checked using Mardia’s PK test based on its distribution to check whether the statistical tests would be parametric or non-parametric.

It was used the Structural Equation Modeling (SEM) technique, more specifically the second-order confirmatory analysis22with the SmartPLS 2.0 M3 software. This technique was chosen because the instrument contains three constructs defined by the Donabedian model16(O - Outcome, S - Structure and P - Process) and the data do not adhere to a multivariate normal distribution. Using the SEM, the measurement model and the structural model were analyzed. The measurement model evaluated: convergent validity, reliability, discriminant validity and significance of correlations and regressions. In the structural model, the following were evaluated: the path coefficients, the Pearson determination coefficients- R2, the effect size- f2and the predictive validity- Q2.

Once the initial model was calculated by the SEM, seven stages were followed to adjust the model. In the first stage, convergent validity was analyzed considering the AVE value ≥ 0.50 (AVE - Average Variance Extracted). Regarding reliability, Composite Reliability (CR) and Cronbach’s alpha were evaluated, and it was defined that both measures should be greater than 0.7020. Discriminant validity was verified using the Fornell and Larcker criterion by comparing the value of the AVE square root of each construct, which should present a higher value than the correlations of the AVE with the other constructs22. The significance of correlations and regressions were evaluated using the resampling module (bootstrapping) of the smartPLS software, with calculation of Student’s t(reference - t ≥ 1.96)22.

After adjusting the measurement model, the structural model was analyzed using Pearson’sdetermination coefficients (R²). For the field of social and behavioral sciences, Cohen (1988) suggests thatR²=2% be classified as a small effect, R²=13% as a medium effect and R²=26% as a large effect22. For the effect size(f²) of the constructs, values of 0.02, 0.15, and 0.35 were considered small, medium and large respectively to weigh the importance of each construct in the model. Finally, the predictive validity(Q²), or accuracy of the model, was evaluated with reference to Q²>020.

The study was approved by the Research Ethics Committee under opinion 3,739,373 (CAAE 20454519,2,0000,5404). Professionals from the expert panel and other participants accepted the ICF. The confidentiality of participant identification and data confidentiality was ensured, in compliance with Resolution No. 466/12.

RESULTS

First stage: Instrument construction

The first version of the instrumentwas developed based on an extensive literature review, consultation of other instruments and focus groups with experts and members of the target population (professionals with experience in Lean Healthcare) resulted in the construction of 54 items. We had the participation of 16 professionals in this stage. From the total items, 47 items were created based on the results of the literature review and consultation of other instruments, while eight (8) items emerged from the focus groups. In total, there were three focus groups, in which participants were also able to give their opinion on the 47 items present in the instrument.

In the distribution by dimensions, conceptual aspects of Donabedian’s theory⁽¹⁶⁾, discussed in the focus groups, were considered: a) Structure: items about resources, norms, routines, system of values and expectations; b) Process: items about the assistance actions provided to patients/users; and c) Outcomes: items that demonstrated the consequences of the activities performed by healthcare professionals. From 54 items, 18 were allocated to the Structure dimension, 17 to Process and 19 to Outcome.

Second stage: Content validity using the Delphi technique

In the content validation stage, two rounds of the Delphi technique were carried out. The first round took place with 14 experts, seven nurses, two hospital administrators, a consultant pharmacist and four engineers, all with experience in Lean. The experts’ response time varied from 10 to 36 days.

Out of the 54 items in the instrument, related to representativeness, 14 items (Q11, Q17, Q18, Q19, Q21, Q25, Q26, Q30, Q33, Q37, Q38, Q39, Q41, Q43) obtained a CVI below 80%. Regarding clarity, 14 items (Q1, Q2, Q3, Q6, Q19, Q25, Q27, Q28, Q38, Q41, Q42, Q47, Q50 and Q51) obtained a CVI below 80%. Of these items, four (Q11, Q18, Q21 and Q30) were excluded due to low representativeness and because they were covered by other items. There was also the inclusion of an item. Therefore, at the end of the first round of the Delphi technique, the instrument contained a total of 51 items.

Still in the first round of the Delphi technique, experts were asked to indicate, among the alternatives presented (structure, process and outcome), the dimension in which each item best fit. There was agreement with the proposed dimension for 45 items (83%). From the analysis of divergences for the remaining nine items, a change in dimension was made for five items (Q1, Q9, Q12, Q26 and Q50) and maintenance of the dimension originally proposed for four items, according to the theoretical framework.

In the second round, 51 items, along with the title, filling instructions, and response options, were sent for analysis to the experts. This round took place 54 days after the first round and involved 13 experts, as one judge had COVID-19 and was unable to participate in this stage. Return time in the second round varied from seven to 22 days. The CVI for clarity in the title was 82%, for filling instructions it was 89%, and for answer options, 89%. Therefore, there were no changes made to these elements.

In this second round, all 51 items had a CVI greater than 80% for representativeness and 50 items obtained a CVI greater than 80% for clarity. Item Q24 obtained a CVI of 71% and had its wording changed, reformulated from: “In my sector there is standardization of conduct (protocols, norms) between professionals in the same category” to “In my sector there is standardization of conduct (protocols, norms) between professionals of the same category, such as nursing, physicians, cleaning and others”. The experts suggested changing the wording of three other items (Q5, Q30 and Q51), which was accepted.

At the end of this stage, the 51 items evaluated were grouped as follows: 19 items in the Structure dimension, 17 items in the Process dimension and 15 items in the Outcome dimension (Chart 1). With the high degree of agreement among experts in validating the content of the instrument’s title, the abbreviation IALEH is used to refer to the Lean Healthcare Evaluation Instrument.

Third stage: Construct validation through Structural Equation Modeling to evaluate the measurement model and the structural model

In the third stage of the study, 113 professionals participated, the majority (76.1%) female, aged between 23 and 67 years old, an average of 42.2 years old and a median of 40.0 years old. From these, 62% from the state of São Paulo, 14% Rio de Janeiro, 7% Minas Gerais, 7% Ceará, 4.4% Rio Grande do Sul and other states (Santa Catarina, Pernambuco, Paraná, Espírito Santo, Distrito Federal and Bahia - 0.9% in each state).

The majority of participants held positions as Managers or Coordinators (58.5%), the others worked as Support Professionals (17.7%), Directors/Administrators (8.8%), Consultants (6.2%), Quality Analysts (5.3%) and Advisors (3.5%). Regarding their educational background, 66.3% were Nurses, 19.4% were administrative professionals, 4.4% Physicians, 2.6% Pharmacists, 2.6% Engineers, Nursing Technicians (1.7%), Psychologist, Quality Technical Assistant, and Assistant (0.9% each).

Participation was obtained of professionals from the hospital area (94.7%), Basic Health Unit (1.8%), Support and Diagnosis (1.8%), Consulting (0.9%) and Outpatient (0.9%). The representation of the participants’ work sectors, in descending order, was: Surgical (35.4%), Quality (24.8%), Clinical (14.1%), Support areas (11.5%), Urgency/ Emergency (9.7%), Intensive Care (2.6%), Basic Health Unit (1.7%).

Once the sample characterization was completed, validation through Structural Equation Modeling (SEM) began. The verification of convergent validity, obtained by observing AVE > 0.50, was carried out in six rounds of Structural Equation Modeling, until all items with AVE ≤ 0.50 were removed from the instrument (Table 1). In the fifth round of SEM, the Structure dimension presented AVE ≤ 0.50, with variables removed from the model. The sixth round resulted in model adjustment, with AVE > 0.50 in all dimensions.

Table 1 -
Values of Average Variance Extracted in rounds of Structural Equation Modeling for the 51 items of the instrument. Campinas, São Paulo, Brazil, 2022

Models	Structure	Process	Outcome	Removed items
1st Round	0.309	0.284	0.313	S01, S10, S12, S19, P01, P02, P03, P07, P09, P13, O02, O08, O10, O11 e O13
2nd Round	0.370	0.362	0.427	S05, S13, S15, S17, S18, P06, P08, P11, P12, R06 e O12
3rd Round	0.417	0.455	0.475	S14, P17, O04
4th Round	0.429	0.491	0.503	S08, P10 e P16
5th Round	0.446	0.528	0.503	S04, S11 e S16
6th Round	0.527	0.537	0.503

• Source: Research data, 2022.
• Legend: S - Structure; P - Process; O - Outcome (dimensions of the instrument).

Chart 1 -
Description of the 51 items and dimensions corresponding to the Lean Healthcare Evaluation Instrument. Campinas, São Paulo, Brazil, 2022

ITEMS
S01	Q1	The infrastructure of my sector was changed to facilitate service flows
S02	Q2	Professional performance evaluation is used positively to correct failures/problems
S03	Q3	The culture of continuous improvement in my sector, with investments in infrastructure, material resources and personnel development, favors my professional development
S04	Q5	Continuing education focuses on the demands of patients/users
S05	Q6	Continuing education focuses on the demands of professionals
S06	Q7	The physical structure (flow organization and layout) enhances work performance
S07	Q16	I feel safe in my work environment
S08	Q17	I have good conditions to perform my job
S09	Q24	My team is always encouraged to incorporate new knowledge into practices
S10	Q25	My team is open to new information/knowledge
S11	Q26	In my institution there is a training and updating program for professionals
S12	Q29	In my sector, changes regarding improvement processes are the exclusive responsibility of my supervisors
S13	Q30	In my sector, quality indicators are available and/or easily accessible to the entire team
S14	Q32	My sector has material resources in quantity and quality to meet healthcare demands
S15	Q33	My sector has technological resources in quantity and quality to meet healthcare demands
S16	Q38	Quality indicators (costs, process and patients/users) are presented to the team periodically
S17	Q43	I feel that my opinion is valued to promote change
S18	Q44	I feel satisfied working in this sector
S19	Q46	I am encouraged to report incidents related to patient/user safety
P01	Q11	The standardization of healthcare promotes physical and emotional safety for the patient/user
P02	Q12	Active participation in change processes in the sector is part of the routine of medical team members
P03	Q14	In my sector there is standardization of conduct (protocols, standards) between professionals in the same category, such as nursing, physicians, cleaning and others
P04	Q18	There is a routine for the team to periodically discuss the sector’s quality indicators
P05	Q19	I am part of a team that maintains continuous communication on a daily basis
P06	Q20	Materials are requested and used without excess or surplus in the sector
P07	Q21	My work process is centered on value to the patient/user
P08	Q23	My supervisor encourages positive discussions about errors, difficulties and lessons learned
P09	Q27	Overall, the team I work for makes decisions together
P10	Q31	Flow mapping enables discussions on continuous improvement in my sector
P11	Q34	The patient/user is promptly attended to in this sector
P12	Q35	The work process flows without interruptions
P13	Q37	The use of standardized procedures and actions increased the recording and documentation of activities in my sector
P14	Q41	Action plans for improvement projects are discussed with the team periodically
P15	Q48	I am sure that now my functions/attributions are very well defined
P16	Q49	All sectors are involved in continuous improvement
P17	Q50	We work to reduce waste in service, such as: unnecessary inventory, no waiting queues, no excess production, no rework, no failures and no lengthy transport
O01	Q4	The culture of continuous improvement has contributed to increasing productivity year after year
O02	Q8	Continuous improvement in healthcare is the result of the work of the team in each sector
O03	Q9	Continuous improvement, based on Lean principles, is part of the routine in my work sector
O04	Q10	My team is recognized for good results in their work
O05	Q13	Currently my sector is more organized and efficient
O06	Q15	I would feel safe being served in this sector
O07	Q22	My sector had immediate positive results with the implementation of Lean
O08	Q28	In my sector, improvements in service and organization are only discussed during audits
O09	Q36	Teamwork improved after implementing Lean in our sector
O10	Q39	Patients/users recognize this institution for its excellence
O11	Q40	I participate in decisions made in my sector
O12	Q42	Little or nothing changed in my sector after implementing Lean
O13	Q45	Problem solving with long-term projects is valued by the team
O14	Q47	We have less waste in the sector compared to last year
O15	Q51	Lately, the perception of value delivered to the patient/user by workers has improved

• Source: Research data, 2022.
• Legend: S - Structure; P - Process; O - Outcome (dimensions of the instrument)/ Q - Question

The final analysis model containing the 16 items that remained in the instrument (Figure 1) resulted in statistically acceptable values and can be considered adjusted. The analysis of the path coefficients (between the arrows that connect the constructs) shows that these values can be considered high, suggesting a good fit of the model. Composite Reliability and the Cronbach’s Alpha (AC) test presented values above 0.70 (Table 2).

Table 2 -
Values of model fit quality tests. Campinas, São Paulo, Brazil, 2022

Dimensions	Average Variance Extracted	Composite Reliability	Cronbach’s Alpha	R2
Structure	0.527	0.847	0.774	0.794
Process	0.537	0.822	0.712	0.630
Outcome	0.503	0.875	0.833	0.818

• Source: Research data, 2022.
• Note: R2 values of 0.02, 0.13 and 0.26 are considered small, medium and large, respectively.

Figure 1 -
Final model of the Lean Healthcare Evaluation Instrument. Campinas, São Paulo, Brazil, 2022

Through the correlation between the constructs of the instrument, it was identified discriminant validity, as the square roots of the correlation between the constructs (square root of the AVE) have a higher value than the correlation with the other constructs. The square roots obtained were Structure=0.726; Process=0.733 and Outcome=0.709. For the other correlations, the following were obtained: Structure-Process=0.604; Structure-Outcome=0.705 and Process-Outcome=0.574.

Moving on to the analysis of the structural model, the values of Pearson’s coefficients of determination(R2 - R Square) present high values, showing that the model has strong relationships between the constructs (Table 2). The significance (Student’s t) of the regressions and correlations were calculated using the resampling (bootstrapping) module of the SmartPLS 2.0/M3 software. In this process, Student’s t values >1.96 were identified for all items.

Regarding predictive validity (Q2), the values obtained were Structure=0.419, Process=0.333, Outcome=0.402 and IALEH=0.302. The effect size values were Structure=0.294, Process=0.227, Outcome=0.334 and IALEH=0.302. The final instrument consisted of 16 items, with five items in the Structure dimension (S_02, S_03, S_04, S_06, S_07), four items in the Process dimension (P_05, P_09, P_14, P_15) and seven items in the Outcome dimension (O_01, O_03, O_05, O_07, O_09, O_14, O_15) (Chart 2).

Chart 2 -
Final version of the Lean Healthcare evaluation instrument. Campinas, São Paulo, Brazil, 2022

IALEH - Lean Healthcare evaluation instrument
Instructions: Please answer the items below considering the extent to which you agree that they are present in your sector or area of work. Physicians and other healthcare professionals who are not assigned to a single unit must respond based on their experiences in the workplace where they spend most of their time.		Completely disagree	Partially disagree	Neutral	Partially agree	Completely agree
ITEMS
1	Professional performance evaluation is used positively to correct failures/problems	1	2	3	4	5
2	The culture of continuous improvement in my sector, with investments in infrastructure, material resources and personnel development, favors my professional development	1	2	3	4	5
3	Continuing education focuses on the demands of patients/users	1	2	3	4	5
4	The physical structure (flow organization and layout) enhances work performance	1	2	3	4	5
5	I feel safe in my work environment	1	2	3	4	5
6	I am part of a team that maintains continuous communication on a daily basis	1	2	3	4	5
7	Overall, the team I work for makes decisions together	1	2	3	4	5
8	Action plans for improvement projects are discussed with the team periodically	1	2	3	4	5
9	I am sure that now my functions/attributions are very well defined	1	2	3	4	5
10	The culture of continuous improvement has contributed to increasing productivity year after year	1	2	3	4	5
11	Continuous improvement, based on Lean principles, is part of the routine in my work sector	1	2	3	4	5
12	Currently my sector is more organized and efficient	1	2	3	4	5
13	My sector had immediate positive results with the implementation of Lean	1	2	3	4	5
14	Teamwork improved after implementing Lean in our sector	1	2	3	4	5
15	We have less waste in the sector compared to last year	1	2	3	4	5
16	Lately, the perception of value delivered to the patient/user by workers has improved	1	2	3	4	5

• Source: Research data, 2022.
• Legend: S - Structure (items 1-5); P - Process (items 6-9); O - Outcome (items 10-16) - dimensions of the instrument.

The score obtained from the application of IALEH can vary from 16 to 80 points, where higher scores (closer to the value of 80) indicate a better perception of the team in adopting Lean Healthcare. For the dimensions, the scores can vary from 5 to 25 for Structure; 4 to 20 for Process; and 7 to 35 for Outcome.

DISCUSSION

In the first stage of this study, an integrative review19, was conducted, consulting other existing instruments on Lean Healthcare and the opinion of experts, with the objective of contemplating different evaluative aspects to produce a reliable and appropriate instrument21. For content validation, success depends on the size of the expert panel, their qualifications and experience. The number of experts in other studies is variable and heterogeneous23. In this regard, the study had the participation of 14 experts in the first round and 13 experts in the second round, all with different backgrounds and experiences.

The remote implementation of the Delphi Technique enabled the participation of experts from different locations, without the need for traveling, avoiding unnecessary costs and facilitating the sending of suggestions. The online model and anonymity allowed everyone to express themselves freely and comfortably, as did studies that also used the Delphi technique24. The return rate at each stage of performing the Delphi technique is variable, generally around 35 to 87%25. In this study, the return rate was 60.8% for the first round of the Delphi technique and higher than expected for the second round, in which 92.8% of experts sent their feedback.

Researchers, in general, faced several intervening factors due to the Covid-19 pandemic. Restrictions regarding social contact, access to healthcare services, staying in closed spaces, overcrowding in hospitals, among others, impacted research. However, it enabled technological advancement in several areas, including data collection26.

Data collection took place during the Covid-19 pandemic, which was the only alternative at the time. During the four months of collection, 321 professionals were interested in participating in the study, with 113 professionals meeting the inclusion criteria. It is believed that more professionals would have participated in the study if it were not for the atypical period experienced, which substantially changed the work routine27, and also a trivialization of web-based research28.

The sample consisted of 113 professionals from different categories, all with experience in Lean Healthcare for more than three months, was substantial for IALEH validation considering that, in Lean Healthcare, the involvement and commitment of professionals is one of the main factors for achieving satisfactory results29.

Regarding the position, most participants were Nurses, Managers and/or Service Coordinators in the hospital area. However, there was participation from other team members. This diversification is relevant in validating IALEH as it confirms the engagement of all professional categories in the practice of the Lean philosophy. Professional education and training of team leaders are also essential for success in Lean management, as it reflects on the engagement of other team members7.

Representatives from various areas participated in the research (Surgical/Sterilized Material Center (SMC), Quality, Clinical Units, Support areas, Urgency/Emergency, Intensive Care and Basic Health Unit), as occurred in other studies on the implementation of the Lean Healthcare29.

Moving towards validation, the SEM was used to analyze: convergent validity, reliability, discriminant validity, significance of correlations and regressions, Pearson’s coefficients of determination - R2, predictive validity- Q2 and effect size- f2. For convergent validity, AVE is the average of the squared factor loadings and indicates how positively the variables correlate with their respective constructs or latent variables. Therefore, when AVEs are greater than 0.50, it is assumed that the model converges to a satisfactory outcome22.

Each dimension, in the final model (6^th round of the SEM), was greater than 0.50 for all dimensions and inferential statistics continued. The traditional indicator used for reliability analysis is Cronbach’s alpha, which is based on intercorrelations of the variables, while Composite Reliability prioritizes the variables according to their reliability, not being sensitive to the number of variables in each construct like the alpha of Cronbach. Both indicate whether the sample is free from bias or whether the responses are reliable, measuring what is proposed. As Cronbach’s alpha is more sensitive to the number of variables in each construct, joint analysis with CR is justified22.

For the analysis of internal consistency, CR measures ranged from 0.82 to 0.88 and Cronbach’s alpha ranged from 0.70 to 0.77, as well as other studies in relation to the reliability of scales, which can be interpreted as satisfactory and demonstrating that the sample is free of bias20. These findings are similar to those found in a Swedish instrument on team perceptions regarding the application of Lean Healthcare, also with 16 items in its final version, in which internal consistency with Cronbach’s Alpha ranged from 0.60 to 0.8617.

The discriminant validity of the SEM is an indicator that the constructs or latent variables are independent on of each other20. Regarding discriminant validity of the instrument, the values obtained were greater than the correlations of the constructs, showing that the model has discriminant validity according to the Fornell-Larcker criterion22. With discriminant validity, the analysis of the measurement model is concluded, and the analysis of the structural model begins using linear correlation tests between observed variables and latent variables for analysis of Pearson’s coefficients of determination(R2) and the significance of the model with Student’s t, a useful measure also in the analysis of the measurement model.

For the field of social and behavioral sciences, Cohen (1988) suggests thatR2=2% be classified as a small effect, R2=13% as a medium effect andR2=26% as a large effect. In the final model of SEM, R2values represent a medium-large effect, showing strong relationships between the constructs. The Student’s t of the model greater than 1.96 shows its significance22.

Relevance or Predictive Validity(Q2) and Effect size(f2) are other indicators analyzed to adjust the model. Q2evaluates the quality of prediction or accuracy of the adjusted model and ranges from zero to 1. In this study, the values obtained forQ2ranged from 0.30 to 0.42, demonstrating accuracy20. The effect size(f2) is obtained by including and excluding constructs from the model one by one, evaluating how “useful” each construct is for adjusting the model. Values of 0.02, 0.15 and 0.35 are considered small, medium and large, respectively22. In this study, the values obtained demonstrate that all constructs are important for the overall adjustment of the model.

In its final version, the instrument contains 16 items distributed across the dimensions Structure (items 1 to 5), Process (items 6 to 9) and Outcome (items 10 to 16). The performance of the scale in validity and reliability analyses aligns with recommendations. The items that remained in the instrument measure the following elements: a) Structure: characteristics of the physical structure, evaluation of professionals' performance, investment in infrastructure, acquisition of necessary resources, ongoing education focused on the patient and safety in the work environment; b) Process: communication, teamwork, action plans with everyone's involvement and the need for definitions in the responsibilities of each professional; and c) Outcome: culture of continuous improvement, reduction of waste and perception of value delivered to the patient. Thus, the three dimensions proposed based on Donabedian’s theoretical framework were addressed16.

These elements corroborate findings from other studies that highlight the importance of generating support and resources, “flow review” and “pull”, where it is necessary to include the entire team and use standardized work, in addition to eliminating waste and practice of activities that add value to the quality of healthcare, aligned with the principle of seeking perfection to advancement of the Lean philosophy throughout the institution6.

It is worth noting that the findings of this study refer to the perception of different professionals, with different realities, all with experience in Lean Healthcare. The implication of the study for the advancement of scientific knowledge is the relevance of systematic monitoring of the implementation of Lean Healthcare. The main reasons for Lean failures are generally associated with inappropriate problem-solving strategies and the pursuit for quick and timely solutions30.

The main limitation of the study is the change in the dynamics of healthcare services due to the COVID-19 pandemic, which interfered with the performance of Lean in institutions and in the routine of healthcare professionals. Therefore, it is suggested that the instrument be applied to other samples and working conditions, continuing to evaluate its psychometric properties. Its application is also recommended in healthcare institutions with different stages of Lean Healthcare adoption.

Other limitations, or aspects of improvement for future studies using IALEH, refer to the possibility of methodical analysis of cut-off points for interpreting the score obtained by applying the scale, in addition to the need to create an official group that records healthcare services that adopt the Lean Healthcare philosophy in their strategic planning.

CONCLUSION

The objective of the study was achieved by following internationally recommended methodological steps for the construction and validation of the Lean Healthcare Evaluation Instrument (IALEH), consisting of 16 items, and divided into three dimensions (Structure, Process and Outcome).

For nursing and healthcare management, the instrument can be useful in different scenarios/sectors in identifying weaknesses that compromise the maintenance of the results achieved in the implementation of Lean Healthcare. The application of IALEH is quick and easy, therefore, it can occur periodically, enabling continuous improvement to be achieved, which is one of the principles of the Lean philosophy.

Acknowledgments:

To the Coordination for the Improvement of Higher Education Personnel - Brazil, (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES)- Funding Code 001- Process 88882.434708/2019-01.

To the Brazilian Association of Surgical Center Nurses (Associação Brasileira de Enfermeiros de Centro Cirúrgico - SOBECC) and the Brazilian Network of Nursing and Patient Safety (Rede Brasileira de Enfermagem e Segurança do Paciente - REBRAENSP) for their support in disseminating information for data collection.

REFERENCES

• 1. Liker JK. Toyota way: 14 management principles from the world's greatest manufacturer [Internet]. New York: McGraw-Hill Education; 2004 [cited 2023 Apr 03]. Available from: Available from: https://www.accessengineeringlibrary.com/content/book/9780071392310
  » https://www.accessengineeringlibrary.com/content/book/9780071392310
• 2. Mahmoud Z, Angelé-Halgand N, Churruca K, Ellis LA, Braithwaite J. The impact of lean management on frontline healthcare professionals: a scoping review of the literature. BMC Health Serv Res. 2021;21(1):383. doi: https://doi.org/10.1186/s12913-021-06344-0
  » https://doi.org/10.1186/s12913-021-06344-0
• 3. Henrique DB, Godinho Filho M. A systematic literature review of empirical research in Lean and Six Sigma in healthcare. Total Qual Manag. 2020;31(3-4):429-49. doi: https://doi.org/10.1080/14783363.2018.1429259
  » https://doi.org/10.1080/14783363.2018.1429259
• 4. Gomes JA, Martins MM, Tronchin DMR, Fernandes CS. Avaliação da qualidade do centro cirúrgico na estrutura, processo e resultados. Cogitare Enferm. 2021;26:e71083. doi: https://doi.org/10.5380/ce.v26i0.71083
  » https://doi.org/10.5380/ce.v26i0.71083
• 5. Schretlen S, Hoefsmit P, Kats S, Merode G, Maessen J, Zandbergen R. Reducing surgical cancellations: a successful application of Lean Six Sigma in healthcare. BMJ Open Qual. 2021;10:e001342. doi: https://doi.org/10.1136/bmjoq-2021-001342
  » https://doi.org/10.1136/bmjoq-2021-001342
• 6. Zimmermann G dos S, Siqueira LD, Bohomol E. Lean Six Sigma methodology application in health care settings: an integrative review. Rev Bras Enferm. 2020;73(suppl 5):e20190861. doi: https://doi.org/10.1590/0034-7167-2019-0861
  » https://doi.org/10.1590/0034-7167-2019-0861
• 7. van Elp B, Roemeling O, Aij KH. Lean leadership: towards continuous improvement capability in healthcare. Health Serv Manage Res. 2022;35(1):7-15. doi: https://doi.org/10.1177/09514848211001688
  » https://doi.org/10.1177/09514848211001688
• 8. Muharam R, Firman F. Lean management improves the process efficiency of controlled ovarian stimulation monitoring in IVF treatment. J Healthc Eng. 2022;2022:6229181. doi: https://doi.org/10.1155/2022/6229181
  » https://doi.org/10.1155/2022/6229181
• 9. Moffatt S, Garry C, McCann H, Teeling SP, Ward M, McNamara M. The use of lean six sigma methodology in the reduction of patient length of stay following anterior cruciate ligament reconstruction surgery. Int J Environ Res Public Health. 2022;19(3):1588. doi: https://doi.org/10.3390/ijerph19031588
  » https://doi.org/10.3390/ijerph19031588
• 10. Hung DY, Mujal G, Jin A, Liang SY. Patient experiences after implementing lean primary care redesigns. Health Serv Res. 2021;56(3):363-70. doi: https://doi.org/10.1111/1475-6773.13605
  » https://doi.org/10.1111/1475-6773.13605
• 11. Tsai HW, Huang SW, Hung YL, Hsu YS, Huang CC. Use of the Smart Lean Method to conduct high-quality integrated perioperative management prior to hospitalization. Int J Environ Res Public Health . 2021;18(24):13391 doi: https://doi.org/10.3390/ijerph182413391
  » https://doi.org/10.3390/ijerph182413391
• 12. Vieira LCN, Menezes MO, Pimentel CA, Juventino GKS. Lean Healthcare no Brasil: uma revisão bibliométrica. Rev Gest Sist Saúde. 2020;9(3):381-405. doi: https://doi.org/10.5585/rgss.v9i3.16882
  » https://doi.org/10.5585/rgss.v9i3.16882
• 13. Silva TO, Vieira LM, Lemos TS, Sant’Anna FP, Sanches RS, Martinez MR. Hospital management and nursing management in the light of the lean philosophy in healthcare. Cogitare Enferm . 2019;24:e60003. doi: https://doi.org/10.5380/ce.v24i0.60003
  » https://doi.org/10.5380/ce.v24i0.60003
• 14. Vieira LCN, Juventino GKS, Pimentel CA, Menezes MO, Silva MFSB, Santos BJ. Contribuições da simulação no lean healthcare para o combate a Covid-19. Rev Tecnol Soc. Curitiba. 2020;16(45):184-201. doi: https://doi.org/10.3895/rts.v16n45.12198
  » https://doi.org/10.3895/rts.v16n45.12198
• 15. Cielusinsky V, Anholon R, Rampasso IS, Silva D, Quelhas OLG. Análise das principais métricas utilizadas por profissionais na avaliação da maturidade de projetos de Lean. Rev Prod Online. 2020;20(1):202-20. doi: https://doi.org/10.14488/1676-1901.v20i1.3470
  » https://doi.org/10.14488/1676-1901.v20i1.3470
• 16. Tossaint-Schoenmakers R, Versluis A, Chavannes N, Talboom-Kamp E, Kasteleyn M. The challenge of integrating ehealth into health care: systematic literature review of the Donabedian model of structure, process, and outcome. J Med Internet Res. 2021;23(5):e27180. doi: https://doi.org/10.2196/27180
  » https://doi.org/10.2196/27180
• 17. Kaltenbrunner M, Bengtsson L, Mathiassen SE, Engström M. A questionnaire measuring staff perceptions of Lean adoption in healthcare: development and psychometric testing. BMC Health Serv Res . 2017;17(1):235. doi: https://doi.org/10.1186/s12913-017-2163-x
  » https://doi.org/10.1186/s12913-017-2163-x
• 18. Sarantopoulos A, Spagnol GS, Newbold D, Li LM. Establishing face validity of the EPLIT questionnaire. Br J Healthc Manage. 2017;23(5):221-7. doi: https://doi.org/10.12968/bjhc.2017.23.5.221
  » https://doi.org/10.12968/bjhc.2017.23.5.221
• 19. Fernandes HMLG, Jesus MVN, Silva D, Guirardello EB. Lean Healthcare in the institutional, professional, and patient perspective: an integrative review. Rev Gaúcha Enferm. 2020;41:e20190340. doi: https://doi.org/10.1590/1983-1447.2020.20190340
  » https://doi.org/10.1590/1983-1447.2020.20190340
• 20. Leguina A A primer on partial least squares structural equation modeling (PLS-SEM). Int J Res Method Educ. 2015;38(2):220-1. doi: https://doi.org/10.1080/1743727X.2015.1005806
  » https://doi.org/10.1080/1743727X.2015.1005806
• 21. Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saúde. 2017;26(3):649-59. doi: https://doi.org/10.5123/S1679-49742017000300022
  » https://doi.org/10.5123/S1679-49742017000300022
• 22. Ringle CM, Silva D, Bido DS. Structural equation modeling with the smartpls. Revista Brasileira de Marketing. 2014;13(2):56-73. doi: https://doi.org/10.5585/remark.v13i2.2717
  » https://doi.org/10.5585/remark.v13i2.2717
• 23. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol. 2021;11(4):116-29. doi: https://doi.org/10.5662/wjm.v11.i4.116
  » https://doi.org/10.5662/wjm.v11.i4.116
• 24. Silva MR, Montilha RCI. Contribuições da técnica Delphi para a validação de uma avaliação de terapia ocupacional em deficiência visual. Cad Bras Ter Ocup. 2021;29:e2863. doi: https://doi.org/10.1590/2526-8910.ctoAO2163
  » https://doi.org/10.1590/2526-8910.ctoAO2163
• 25. Romero-Collado A. Essential elements to elaborate a study with the (e)Delphi method. Enferm Intensiva . 2021;32(2):100-4. doi: https://doi.org/10.1016/j.enfie.2020.09.003
  » https://doi.org/10.1016/j.enfie.2020.09.003
• 26. Lobe B, Morgan D, Hoffman KA. Qualitative data collection in an era of social distancing. Int J Qual Methods. 2020;19:1609406920937875. doi: https://doi.org/10.1177/1609406920937875
  » https://doi.org/10.1177/1609406920937875
• 27. Acioli DMN, Santos AAP, Santos JAM, Souza IP, Silva RKL. Impacts of the COVID-19 pandemic on nurses’ health. Rev Enferm UERJ. 2022;30:e63904. doi: https://doi.org/10.12957/reuerj.2022.63904
  » https://doi.org/10.12957/reuerj.2022.63904
• 28. Daikeler J, Bošnjak M, Lozar Manfreda K. Web versus other survey modes: an updated and extended meta-analysis comparing response rates. J Surv Stat Methodol. 2020;8(3):513-39. doi: https://doi.org/10.1093/jssam/smz008
  » https://doi.org/10.1093/jssam/smz008
• 29. Tlapa D, Zepeda-Lugo CA, Tortorella GL, Baez-Lopez YA, Limon-Romero J, Alvarado-Iniesta A, et al. Effects of lean healthcare on patient flow: a systematic review. Value Health. 2020;23(2):260-73. doi: http://10.1016/j.jval.2019.11.002
  » http://10.1016/j.jval.2019.11.002
• 30. Akmal A, Greatbanks R, Foote J. Lean thinking in healthcare-findings from a systematic literature network and bibliometric analysis. Health Policy. 2020;124(6):615-27. doi: https://doi.org/10.1016/j.healthpol.2020.04.008
  » https://doi.org/10.1016/j.healthpol.2020.04.008

Edited by

Associate editor:

Aline Marques Acosta

Editor-in-chief:

João Lucas Campos de Oliveira

Publication Dates

• Publication in this collection
  12 Feb 2024
• Date of issue
  2024

History

• Received
  12 May 2023
• Accepted
  08 Sept 2023