Programmatic situation from the perspective of health vulnerability: Item database validation

ABSTRACT

Objective

To validate an item database to assess the programmatic situation of health vulnerability.

Method

A validation study carried out in 2021. Two scoping reviews were elaborated to identify operational definitions and, afterwards, meetings were held with the authors to formulate items. As a follow-up, the items were sent via Google Forms to specialists with expertise in the areas of health vulnerability or construction and validation of instruments. To validate the items, we used the content validity coefficient and binomial test, in addition to the intraclass correlation coefficient to verify reliability, all via SPSS® version 25.

Results

Seven experts answered with item evaluations in the language clarity, practical relevance and theoretical relevance criteria. Of the 88 items organized in the infrastructure and work process sub-concepts, most were modified following the experts' suggestions and had content validity coefficients greater than 0.80. The intraclass correlation coefficient was 0.80 for clarity, 0.94 for relevance, and 0.92 for relevance (p < 0.05). Two items were excluded after the authors' meeting for consensus on the final item database and ten were merged.

Conclusion and implications for the practice

The items were internally validated and there was good reliability among the judges, enabling their use by health professionals to investigate vulnerability.

Keywords:
Validation study; Workflow; Health infrastructure; Public health; Health vulnerability

RESUMO

Objetivo

Validar banco de itens para avaliação da situação programática na perspectiva da vulnerabilidade em saúde.

Método

Estudo de validação realizado em 2021. Construíram-se duas scoping reviews para identificar definições operacionais e, após, realizaram-se reuniões com os autores para formular itens. Em seguimento, enviaram-se os itens via Google Forms para especialistas com expertise na área de vulnerabilidade em saúde ou construção e validação de instrumentos. Para validar os itens, utilizou-se coeficiente de validade de conteúdo, teste binomial, além do coeficiente de correlação intraclasse para verificar confiabilidade, todos via SPSS® versão 25.

Resultados

Sete especialistas retornaram com avaliações dos itens nos critérios clareza da linguagem, pertinência prática e relevância teórica. Dos 88 itens organizados nos subconceitos infraestrutura e processo de trabalho, a maior parte foi modificada por sugestão dos especialistas e teve coeficiente de validade de conteúdo maior que 0,80. O coeficiente de correlação intraclasse foi 0,80 para clareza, 0,94 para pertinência e 0,92 para relevância (p < 0,05). Dois itens foram excluídos após reunião dos autores para consenso sobre os itens do banco final e dez foram mesclados. Conclusão e implicação para prática: O banco de itens foi validado internamente e houve boa confiabilidade entre os juízes, possibilitando o uso por profissionais da saúde para investigar a vulnerabilidade.

Palavras-chave:
Estudo de validação; Fluxo de trabalho; Infraestrutura sanitária; Saúde pública; Vulnerabilidade em saúde

RESUMEN

Objetivo

Validar una base de datos de ítems para evaluar la situación programática de vulnerabilidad en salud.

Método

Estudio de validación realizado en 2021. Se construyeron dos revisiones de alcance para identificar definiciones operativas y, posteriormente, se realizaron reuniones con los autores para la formulación de ítems. Como seguimiento, los ítems se enviaron a través de formularios de google a especialistas con experiencia en el área de vulnerabilidad en salud o construcción y validación de instrumentos. Para validar los ítems utilizamos el coeficiente de validez de contenido, prueba binomial, además del coeficiente de correlación intraclase para verificar la confiabilidad, todo a través de SPSS® versión 25.

Resultados

Siete expertos regresaron con evaluaciones de ítems en los criterios claridad de lenguaje, relevancia práctica y relevancia teórica. De los 88 ítems organizados en los subconceptos de infraestructura y proceso de trabajo, la mayoría fueron modificados por sugerencia de expertos y tuvieron un coeficiente de validez de contenido superior a 0,80. El coeficiente de correlación intraclase fue 0,80 para claridad, 0,94 para pertinencia y 0,92 para relevancia (p < 0,05). Se excluyeron dos elementos después de la reunión de los autores para el consenso sobre los elementos finales de la base de datos y diez se fusionaron.

Conclusión e implicaciones para la práctica

Los ítems fueron validados internamente y hubo una buena fiabilidad entre los jueces, lo que permitió su uso por parte de los profesionales de la salud para investigar la vulnerabilidad.

Palabras clave:
Estudio de Validación; Flujo de Trabajo; Infraestructura Sanitaria; Salud Pública; Vulnerabilidad en Salud

INTRODUCTION

Health Vulnerability (HV) is a human life condition expressed in all its dimensions from the (re)arrangements of power relations that constitute its essential elements: the subject, understood as human life constituted from subjective relations; and the social, the appearance scene where the subject relates to other lives or institutions in the health field. These essential elements are HV attributes and are supported by concepts and sub-concepts.¹

Among these, the Programmatic Situation (PS) stands out, which is defined as “characteristics and processes of institutions that provide various types of services to the population, especially those related to health”.^1:15 In this conceptual model, the PS concept belongs to the social element and has the following analytical components: the infrastructure, characterized by the set of human and material resources that produce care; and the work process, understood as articulation between health professionals and subjects on which the action of this worker focuses.¹

From the perspective of expanding health promotion, the discussion about HV through the PS and its relationship with subjects is relevant, as it presupposes impacts on access to health and on care quality. Health services sometimes tend to produce HV situations when characteristics that should produce excellent care, for example, infrastructure and the work process, are insufficient, scarce, ineffective or absent.^1,2

Therefore, considering that the PS is inserted in the HV field and involves other sub-concepts such as human and material resources, the health professionals' techno-scientific preparation, care practices, and management and organization of the services, it is assumed that it is necessary to study these elements. This is because they are directly related to the health-disease process and because there is a gap in the literature² when seeking to understand the PS from the social-subject perspective,¹ given that it is a new conceptual model in the context of research and care of subjects among professionals because it involves several factors such as inputs, staffing, service protocols, permanent education or care availability.²

The research shows that, although the health network has expanded access, care quality has been questioned because it results from several factors such as “availability of equipment, staffing pattern, availability of different health professionals, poor coordination, variations in management, availability of specialized service and institutional support”.^2:26 Thus, it is necessary to monitor indicators aimed at improving the functioning of the health system to reduce disparities in care because there are regional, social and schooling level differences.³ Both the infrastructure and the work process have improved in recent decades in the services, but the transformations were not homogeneous,⁴ as there is great variability across the states, which allows for a vast number of vulnerability scenarios for subjects who use these services.

In the HV context, there is no such evaluation through indicators or instruments. Therefore, it is opportune to construct and validate an item database for PS evaluation from the HV perspective, so that the professionals can identify vulnerable scenarios for subjects that are caused by ineffectiveness, insufficiency, scarcity or absence of infrastructure and the work process. Thus, the objective was to validate an item database to evaluate the programmatic situation from the perspective of health vulnerability.

METHOD

This is a content validation study of an item database. These studies verify the reliability and validity of a given instrument with which it is intended to take measurements.⁵ In this case, we sought the internal validation of an item database for PS evaluation. This study complied with Resolution 466/2012 of the National Health Council and was approved by the Committee of Ethics in Research with Human Beings of Universidade Estadual do Ceará (UECE) under number 4.393.432.

Experts in the lattes platform were searched and they were selected based on the Guimarães et al.⁶ parameters. According to the parameters, the experts were classified as follows: 1. Junior, clinical experience of at least four years in the specific area of the study (mandatory); 2. Master, score between six and twenty points; or 3. Senior expert, score greater than twenty points, supported by years of experience. The higher the score, the better the level, and the minimum score to be considered an expert in the area was five points. Among the parameters were clinical experience of four years, teaching experience, article published in the area, participation in research groups, PhD/Master's degree in the study area and residency in health.

Elaboration of the items arose from stages that have existed prior to this study. The first stage consisted of a review on the conceptual clarification of health vulnerability;¹ in the second stage, an update of this review was prepared for the year 2020, only referring to aspects of the programmatic situation that consisted of the health professionals' techno-scientific preparation, human and material resources, management and organization of the health services and care practices. These two reviews resulted in 120 articles. Subsequently, operational definitions related to the infrastructure and work process were extracted from each article. Thus, it was possible to elaborate the items, according to the HV model.¹

Elaboration was made up from two meetings between three authors of this study, lasting three hours each. The first took place on April 6^th and the second on April 8^th, both in 2021. From these meetings, 88 items distributed between both PS sub-concepts were initially elaborated to be submitted to evaluation.

After identifying the experts, they were sent an email message with an invitation containing a link to the form. This form was built on Google Forms and contained eight sections: 1. Welcome, invitation letter and space to suggest other specialists; 2. Free and Informed Consent Form (FICF) and post-informed consent; 3. Instructional script for filling out the form containing the evaluation criteria of the items; 4. Sociodemographic and professional identification of the specialists; 5. Conceptual model, concepts and sub-concepts of health vulnerability; 6. Indication to the next section (evaluation of the items); 7. Infrastructure assessment; and 8. Assessment of the work process. Seven of the 42 eligible specialists answered the invitation.

Data collection took place in May and June 2021. The data were organized in a Microsoft Excel® spreadsheet. The socio-professiographic variables were name, age, gender, city, time since graduation in years, basic profession, time in teaching, scientific production, and experience in care, teaching and management. The criteria for evaluating the items were as follows: language clarity, which analyzes whether the writing style is difficult to understand; practical relevance, which aims at knowing if the item is representative of the situation to be observed and if it is important; and theoretical relevance, which considers the degree of association between the item and the construct that intends to be measured.

The criteria received scores through categories based on a five-point Likert scale: 1) The item is very little indicative of health vulnerability (programmatic situation); 2) The item is little indicative of health vulnerability (programmatic situation); 3) The item is considerably indicative of health vulnerability (programmatic situation); 4) The item is very indicative of health vulnerability (programmatic situation); and 5) The item is very much indicative of health vulnerability (programmatic situation). For this, the grades were considered high when scored 4 or 5, medium when scored 3, and low when scored 1 or 2.

The data were processed and analyzed using the Statistical Package for the Social Sciences (SPSS®), version 25. The Content Validity Coefficient (CVC)⁷ was used to verify the items' content validity, and items whose scores were equal to or greater than 0.80 were considered valid. This result is obtained by calculating the mean value of the grades obtained by the specialists, consisting in calculating the initial CVC for each item in the list, calculating the error and final calculation for each item in the list.

A binomial test was used to estimate statistical reliability to the CVC, with a significance level of p > 0.05, as well as to know the proportion of criteria with scores ≥ 0.80, if the proportion is 70% (0.80 in two criteria) or 30% (0.80 in only one criterion). Reliability was verified with the Intraclass Correlation Coefficient (ICC), with an estimated cutoff point of 0.80 to be considered reliable. The ICC was used to calculate reliability among specialists with a 95% significance level (p < 0.05) and it was not possible to perform a test-retest, as the instrument has not yet passed the external validation phase.

RESULTS

The content was evaluated by seven specialists in the topic of vulnerability or construction and validation of health materials, one male, with basic training in speech therapy, with scientific publications in the field of health vulnerability and 13 years of experience in teaching; and six female, five of them nurses, also with experience in the topic of vulnerability; only one does not have teaching practice and another has a Bachelor's degree in Physical Education with 11 years of experience in teaching. Chart 1 describes the expert judges' expertise (E) and highlights some contributions of the evaluation of the items. The experts were randomly identified in numerical order.

Chart 1
Profile of the experts and their contributions. Fortaleza, Ceará, Brazil, 2021.

E	Category/Degree	Level of expertise	Evaluation of the items
E1	Nurse. Master and PhD student in Nursing. Member of the Brazilian Association of Gerontological Nursing (CE)	Junior	Agreed with 97% (86) of the items. Suggested changing and removing terms to improve the reader's understanding of the items, standardizing between being the “service used” or the “most used service”.
E2	Speech therapist. Specialist, Master and PhD student in Collective Health.	Senior	Agreed with 75% (63) of the items. Mostly reported that the items would be difficult for the reader to understand, and that it is necessary to qualify the term “not applicable”.
E3	Nurse. Master in Collective Health. PhD in Public Health.	Senior	Agreed with 84% (74) of the items. Suggested involving aeration in some infrastructure conditions, making clear what “specific”, “regular”, “periodic form” and “routine” are in the items, and suggested removing item 26.
E4	Nurse. Master and PhD student in Nursing.	Junior	Agreed with 95% (84) of the items. Suggested replacing some terms for the reader's understanding. And criticized item 26, for considering that it did not portray a situation of vulnerability.
E5	Nurse. Master and PhD student in Clinical Health Care.	Junior	Agreed with 99% (87) of the items. Suggested standardizing terms such as “health service” or “health unit”. Gave examples to be included in the items for the subject to better understand and replace other terms.
E6	Nurse. Master in Family Health.	Junior	Agreed with 96% (85) of the items. Redrafted a few items. Suggested specifying some terms for the subject to understand what is being evaluated.
E7	Physical Education Professional. Master and PhD student in Collective Health.	Senior	Agreed with 98% (86) of the items. Suggested changing the order of some items, reformulating and changing terms and clarifying some items so that the subject understands.

• Source: Research data.

Regarding the CVC, the experts evaluated the criteria of the items (language clarity, practical relevance and theoretical relevance) and high grades (4 and 5) were noticed, with excellent mean values for most of the items. Three had a final mean below 0.80 and a p-value of 0.48 (Table 1). Item 26 was suggested to be removed, as it had a very low mean and for being the only one that had a score lower than 0.80 in all criteria. Thus, it was understood that this item does not represent a programmatic situation for HV, therefore being excluded from the final item database, as well as other items. The ICC was calculated and evidenced excellent reliability among the judges, being 0.80 for the language clarity criterion, 0.94 for practical relevance and 0.92 for theoretical relevance (p < 0.05), being possible to assert that the reliability parameter was met.

Table 1
Content validity coefficient and binomial test of the items. Fortaleza, Ceará, Brazil, 2021.

Item	Clarity	Pertinence	Relevance	Mean CVCi	p-value*	Item	Clarity	Pertinence	Relevance	Mean CVCi	p-value*	Item	Clarity	Pertinence	Relevance	Mean CVCi	p-value*
1	0.89	0.91	0.91	0.90	0.51	31	0.71	0.77	0.94	0.81	0.10a	61	0.83	0.91	0.91	0.89	0.51
2	0.94	0.86	0.89	0.90	0.51	32	0.80	0.80	0.91	0.84	0.51	62	0.91	0.97	0.94	0.94	0.51
3	0.94	0.97	0.97	0.96	0.51	33	0.97	0.86	0.83	0.89	0.51	63	0.97	0.97	0.97	0.97	0.51
4	0.77	0.80	0.80	0.79	0.48a	34	0.97	0.97	0.97	0.97	0.51	64	0.97	0.97	0.97	0.97	0.51
5	0.91	0.91	0.91	0.91	0.51	35	0.89	0.83	0.91	0.88	0.51	65	0.91	0.97	0.97	0.95	0.51
6	0.91	0.94	0.94	0.93	0.51	36	0.97	0.94	0.94	0.95	0.51	66	0.97	0.97	0.97	0.97	0.51
7	0.91	0.91	0.91	0.91	0.51	37	0.97	0.91	0.91	0.93	0.51	67	0.94	0.94	0.94	0.94	0.51
8	0.86	0.91	0.91	0.90	0.51	38	0.80	0.83	0.80	0.81	0.51	68	0.94	0.91	0.94	0.93	0.51
9	0.71	0.83	0.89	0.81	0.48a	39	0.94	0.86	0.89	0.90	0.51	69	0.97	0.94	0.94	0.95	0.51
10	0.83	0.83	0.89	0.85	0.51	40	0.89	0.97	0.97	0.94	0.51	70	0.94	0.97	0.97	0.96	0.51
11	0.83	0.91	0.91	0.89	0.51	41	0.97	0.94	0.97	0.96	0.51	71	0.91	0.97	0.97	0.95	0.51
12	0.89	0.94	0.94	0.92	0.51	42	0.94	0.97	0.97	0.96	0.51	72	0.97	0.97	0.97	0.97	0.51
13	0.86	0.91	0.94	0.90	0.51	43	0.89	0.94	0.97	0.93	0.51	73	0.97	0.97	1.00	0.98	0.51
14	0.86	0.97	0.97	0.93	0.51	44	0.77	0.83	0.86	0.82	0.48a	74	0.80	0.97	0.97	0.91	0.51
15	0.91	0.94	0.94	0.93	0.51	45	0.94	0.97	0.97	0.96	0.51	75	0.97	0.97	0.97	0.97	0.51
16	0.83	0.83	0.86	0.84	0.51	46	0.91	0.91	0.91	0.91	0.51	76	0.86	0.97	0.97	0.93	0.51
17	0.89	0.89	0.94	0.90	0.51	47	0.97	0.97	0.97	0.97	0.51	77	0.94	0.86	0.89	0.90	0.51
18	0.91	0.97	0.97	0.95	0.51	48	0.94	0.97	0.97	0.96	0.51	78	0.86	0.86	0.86	0.86	0.51
19	0.86	0.89	0.89	0.88	0.51	49	0.97	0.97	0.97	0.97	0.51	79	0.97	0.86	0.89	0.90	0.51
20	0.71	0.89	0.89	0.83	0.48a	50	0.91	0.86	0.94	0.90	0.51	80	0.97	0.86	0.89	0.90	0.51
21	0.91	0.86	0.94	0.90	0.51	51	0.94	0.97	0.97	0.96	0.51	81	0.97	0.86	0.89	0.90	0.51
22	0.94	0.97	0.97	0.96	0.51	52	0.86	0.94	0.97	0.92	0.51	82	0.97	0.86	0.89	0.90	0.51
23	0.91	0.94	0.94	0.93	0.51	53	0.97	0.97	0.97	0.97	0.51	83	0.97	0.86	0.89	0.90	0.51
24	0.94	0.94	0.94	0.94	0.51	54	0.89	0.97	0.97	0.94	0.51	84	0.97	0.86	0.89	0.90	0.51
25	0.91	0.91	0.94	0.92	0.51	55	0.86	0.97	0.97	0.93	0.51	85	0.97	0.86	0.89	0.90	0.51
26	0.74	0.77	0.77	0.76	0.008a	56	0.74	0.94	0.94	0.88	0.48a	86	0.97	0.86	0.89	0.90	0.51
27	0.86	0.94	0.94	0.91	0.51	57	0.80	0.91	0.91	0.88	0.51	87	0.86	0.86	0.89	0.87	0.51
28	0.91	0.97	0.97	0.95	0.51	58	0.94	0.97	0.97	0.96	0.51	88	0.97	0.97	0.97	0.97	0.51
29	0.69	0.80	0.83	0.77	0.48a	59	0.80	0.89	0.89	0.86	0.51
30	0.89	0.91	0.97	0.92	0.51	60	0.91	0.97	0.97	0.95	0.51

• Source: Prepared by the authors.
• Note: *Binomial test: (a) the alternative hypothesis states that the proportion of cases in the first group is <0.8.

Thus, the list of items that evaluate HV through the infrastructure and work process is presented (Chart 2). The final version consists of 76 items: 22 for the infrastructure concept and 54 for evaluation of the work process, each item with 7 answer options varying from 1 to 7, where 1 is the lowest HV degree and option 6 is the highest HV degree found. Answer option 7 should be checked when a particular item does not apply to the subject interviewed. Two items were excluded after a consensus meeting to review the experts' answers and 10 items were merged for being similar.

Chart 2
Item database to assess the programmatic situation of health vulnerability. Fortaleza, Ceará, Brazil, 2021.

Item database to assess the programmatic situation of the concept of health Vulnerability
Infrastructure
According to the service you use the most, please answer:
1 - Is there any area with compromised infrastructure (broken chairs, aeration conditions, rooms with mold, cracks, leaks, lack of water, etc.)?	41 - In the health service, are the care routines frequently modified?
1) None; 2) At least one area; 3) Few areas; 4) Some areas; 5) The vast majority of the areas; 6) All areas; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
2 - In your consultations or appointments, do the professionals provide materials that explain health topics (pamphlets, booklets, educational folders, applications, etc.)?	42 - Have you ever been mechanically treated (which procedure, what it is for, etc.) by the professional during the consultation or appointment?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
3 - Have you ever lacked medication?	43 - Do the health professionals record your appointments in any medical chart?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
4 – Do you have a room with ample space that can be used for collective/educational activities?	44 - Have you had access to the health bulletin of the city or neighborhood in which you live (publications of the health department or similar bodies)?
1) Yes, very wide; 2) Yes, wide; 3) Yes, wide enough; 4) Yes, little wide; 5) Has room, but very small; 6) No space for activities; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
5 - Have you ever failed to undergo examinations/procedures due to absence of materials/equipment?	45 - In the health service, is there a responsible and active group that discusses the organization of the health service (health council)?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Yes; 2) Probably; 3) I can't say; 4) At least once; 5) Probably not; 6) No; 7) Not applicable.
6- Have you ever failed to undergo exams due to absence of personal protective equipment for the health professionals (glove, hat, mask, etc.)?	46 - When you are referred to another service, do you take the referral paper?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
7 - Is there equipment to perform the expected diagnostic tests (mammography, tomography, ultrasound, blood collection, etc.)?	47 - When returning to the original service where you were treated, do you bring with you the document that explains how the consultation went and your health condition?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
8 - Are there specific exams for your health condition (e.g., routine exams)?	48 - Did you have difficulty being referred to a specialized service?
1) Always; 2) Most of the the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) I don't know how to answer.	1) Never; 2) I had difficulty at least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
9 - Have you ever not been treated due to problems with the online medical records system (slow system, system down, etc.)?	49 - Have you been referred to a specialized service and been waiting a long time to be called for this consultation/exam?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) I don't know how to answer/There is no medical record	1) No; 2) Yes, at least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
10 - Is rapid testing available for diagnosis (HIV, syphilis, Diabetes, Hepatitis C, etc.)?	50 - When you need to use other services, is your unit of origin informed by this other service?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never, it is generally me that reports; 7) Not applicable.
11 - Is there privacy in the appointment rooms (without interruptions, one person at a time)?	51 - Do the health professionals who assist you report that they are following pre-established routines (flows established by the health department or similar bodies)?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
12 – Are disposable materials available for personal use (syringes, needles, condoms, etc.)?	52 - Were you offered any health education activities, such as groups, conversation circles, etc.?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
13 - Did the health professional ever ask about your social condition (work and salary, housing, government benefits, etc.)?	53 - Are there many people waiting for care/long queues in the health service?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
14 - Do you usually have quick consultations or appointments?	54 - Do you take a long time to receive the results of the exams performed?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
15 - In the medical consultations or appointments, do the health professionals usually make a more detailed evaluation/examination (did they search for antecedents and talk about what can happen, did they search for the probable cause)?	55 - Do you receive the result of your health exams at the time it was agreed upon?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
16 - Have you ever not performed a procedure that you needed due to lack of trained health professionals in the service?	56 – Do health professionals request your exams periodically (once a year)?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
17 - Was there any difficulty or delay in your appointment or consultation with a health professional due to your disability?	57 - Have you ever received exams with results that were not in accordance with your diagnosis?
1) Never; 2) I had difficulty at least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
18 - Did the health professional ever not know how to interpret your exams (gynecological prevention, laboratory exams, imaging exams, etc.)?	58 - Do you attend routine consultations to assess your health (once a year)?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
19 - Has the health professional ever focused only on filling out papers and making the referral without giving further explanations about your health status during your consultation or appointment?	59 - Have you ever been mistreated in the health service?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
20 - Did you have any health conditions that took a long time to be diagnosed?	60 - In the health service, are there people who advise you to go to the right places/rooms?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
21 - Have you ever had complications in your health condition due to a delay in diagnosis (delay in test results, lack of vacancies, difficulty scheduling returns)?	61 - Do you feel safe being treated at the nearest health service?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
22 - When you arrived at the health service to be assisted, did you return due to the reduction of the service hours?	62 - Did you feel unsafe with any guidance during a consultation or appointments by a health professional?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
	63 - Do you perform any activities related to your health outside the health service that you use the most?
Work processes	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
According to the service you use the most, please answer:
	64 - Have you ever had confidential information about your health shared with third parties without your permission?
23 - Are there teams responsible for care planning (e.g., is there a team responsible for you)?	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
1) Always; 2) Most of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) I don't know how to answer.
	65 - Do the health technologies (app, websites, crime reporting service) provided by the health services work when you need them?
24 - Do you believe that there are few health professionals working in the service you use the most?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
	66 - In the health service you use, are there specific consultations/appointments for various health conditions (hypertension, diabetes, leprosy, tuberculosis, respiratory diseases, etc.)?
25 - Do you usually see health professionals of the service in courses, case discussion, staff training, etc.?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) Most of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) I don't know how to answer.
	67 - When you attend a consultation, is it usual to be assisted by a multidisciplinary team (several professionals)?
26 - Is there a good relationship between professionals in the service?	1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
1) Always; 2) Most of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) I don't know how to answer.
	68 - In the health service that you use the most, are there appointments/consultations with nurses available?
27 - Is alternate consultation offered between nurse and physician in the service?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
	69 - In the health service that you use the most, are there appointments/consultations with a physician available?
28 - Have you stopped being assisted due to the limited number of forms/passwords for consultations with health professionals?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
	70 - In the health service that you use the most, are there appointments/consultations with a physiotherapist available?
29 - Do you wait more than two hours to be served?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
	71 - In the health service that you use the most, are there appointments/consultations with a physical education professional available?
30 - During your consultations or appointments, do the health professionals pay more attention to your disease, leaving aside other aspects of your life that you wanted to share?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
	72 - In the health service that you use the most, are there appointments/consultations with a dentist available?
31 - Do you receive guidance on all the procedures and exams you need to undergo (objective of the exam, risks, benefits, what is used, how long it will take etc.)?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
	73 - In the health service that you use the most, are there appointments/consultations with a social worker available?
32 - Do you receive regular (1x/month) visits from health professionals at your home to address issues related to your health?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
	74 - In the health service you use the most, are there appointments/consultations with a psychologist available?
33 - Do the home visits you receive from health professionals meet your needs?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
	75 - In the health service you use the most, are there appointments/consultations with a nutritionist available?
34 - Has the health professional ever informed you that you have a disease that needs to be notified (informed to the health department)?	1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
	76 - In the health service you use the most, are there complementary health practices (dance groups, physical exercise groups, yoga, acupuncture, auriculotherapy, flower therapy, etc.)?
35 - Is there a space for reception in the health service (private space that allows you to have an initial conversation with a health professional)?	1) Always; 2) Most of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) I don't know how to answer.
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
36 - Do health professionals pay attention when you talk about your health condition?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.
37 - Do health professionals give excessive personal opinions* related to their health condition?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
*Note: Based on quick information and without a more extended elaboration on important issues
38 - Have you ever felt dissatisfied with the care you received when using the health service?
1) Never; 2) At least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
39 - Do you have difficulty interacting with the health professional during your consultation/appointment?
1) Never; 2) I had difficulty at least once; 3) Few times; 4) Sometimes; 5) The vast majority of the times; 6) Always; 7) Not applicable.
40 - In your perception, do the professionals put themselves in your shoes when dealing with your health condition (welcoming, understanding, patient)?
1) Always; 2) The vast majority of the times; 3) Sometimes; 4) Few times; 5) At least once; 6) Never; 7) Not applicable.

• Source: Prepared by the authors.

DISCUSSION

When elaborating and validating the items to evaluate the programmatic situation of the HV concept in the collective health context, it is expected that it contributes to the discussion of subjects' vulnerabilities and, thus, that there are impacts on the services in order to rethink the processes that can make care for the subjects precarious.

The validation procedures took into account the specialists' expertise in two main dimensions, the scientific papers produced with a focus on HV or the construction and validation of instruments. The suggestions made by the professionals who comprised the sample of specialists were important to improve understanding of the items, highlighting the language clarity criterion for the reader to understand.

The ICC had good results, and it is possible to state that there is good reliability among the judges and that the items are representative of the programmatic situation of HV in the public health context. It is noteworthy that the language clarity criterion had a lower value than the others (0.80), with most of the suggestions of the items related to this aspect. Therefore, the items were reviewed according to the suggestions, as difficulty understanding the meaning of the items can lead to the elaboration of fragile and non-applicable models in the external validation phase. Thus, it is emphasized that the authors met all the considerations to improve language clarity, making it simpler and more objective.

Therefore, use of these items should be considered. The PS concept deals with the subjects' HV and the health professional must critically judge this human life condition produced from the service-person relationship. The PS does not seek to identify the vulnerability of the health service, although it is acknowledged that some characteristics of the health institutions exert direct effects on the subjects' HV. Confusion can lead to an inadequate understanding of what the PS really aims at, which is to investigate the subjects' HV and not that of the health services, failing to identify and prioritize phenomena that interfere with the care provided to the subject.

The greatest highlight among the items was question number 62: “Did you feel insecure with any guidance during consultation or care by a health professional?”. This item had the highest CVCi score: 0.98 (item 73 in Table 1). For the professionals, the main objective is to make the patient follow their guidelines to change their lifestyle, when it is unfavorable to health. However, insecurity can be a strong factor in determining this relationship. Therefore, the health professionals' techno-scientific preparation and the way in which the care practices are implemented are important categories in this type of subject-service relationship. It is important to show that item 61, which is also related to safety, had good scores (CVCi of 0.97, item 72 in Table 1). The difference between them is that the first refers to the guidelines provided by the professionals and the second to the subject's physical safety in relation to the health service.

When referring to safety of and for subjects, the discussion is in the field of patient safety in relation to the errors by health professionals that may occur and of safety in relation to the context of violence in certain territories. Some people are forbidden to use closer health services because they are inserted amid the oppression and cruelty of other subjects. It is this sense that item 72 is concerned with evaluating. It is fundamental that the health services are able to prevent and reduce situations of violence using multidisciplinary approaches that stimulate commitment and a culture of peace.⁸

Other items also stood out with CVCi scores equal to or greater than 0.90 (76%, 67 items) with a value of 0.97 in the three dimensions evaluated. All these items belong to the work process category and refer to aspects such as the limited number of care modalities, feeling heard by the professional, interaction with health professionals, records in medical charts, offering health education activities, safety, confidential information and the presence of long queues.

The “Do you have medical appointments just in hospitals?” item (item 26 in Table 1) was removed from the list because the experts understood that this question does not represent a programmatic situation; therefore, its CVC was low and the binomial test proved to be significant not to include it in the final item database sample. This item was thought by an operational definition withdrawn from the study⁹ that indicated that consultations only in hospitals could impair the monitoring of people over time. However, it is observed that, although the hospital is historically a place only for rehabilitation or cure, this service is important in the comprehensive care provided to the subjects, as long as the professionals are not limited to the biological aspects of care, plan in a multidisciplinary way or provide clarification in the appointments, so that the subjects may come to have autonomy to follow the path that is most appropriate to their care line¹⁰, which affects the improvement of the workflow.

On the contrary, items 04 and 23 (item 29 in Table 1) had a low validation coefficient, although the binomial test was significant. It is pointed out that these items had low scores because the language was not clear to the experts' understanding and there would be important implications at the time of the evaluation with the subjects. After the experts' suggestions and a consensus meeting with all three authors of this study, it was decided to keep them.

Items 05 and 06, 16 and 17 were merged because it was observed that they were quite similar in the objective they evaluate. Both items had good CVCi scores. Thus, the questions were “Have you ever failed to undergo exams/procedures due to absence of materials/equipment?” and “In consultations or appointments, do health professionals usually make a more detailed evaluation/examination (did they search for antecedents and say what could happen, did they search for the probable cause)?”.

Division of the items in the two dimensions proposed by the theoretical framework¹ helps the professionals to appropriate aspects of the infrastructure and the work process in their professional practice and to identify situations other than those found in the items, which may break a cycle of production of vulnerabilities. Understanding health needs by health workers adds to the development of practices that go beyond intervention strategies focused on biological actions that do not reflect human complexity.¹¹

Analysis of the infrastructure and the work process becomes an additional tool for indirect evaluation of some indicators necessary for the functioning of health services, as a study¹² indicates that only 35% of the basic health units are adequate in terms of health infrastructure aspects and 8% for the work process in the context related to health nutrition actions in all the states referring to the country observed.

The field in which the analysis of these items is situated is in the precariousness conditions caused to people in situations of vulnerability in a condition of reduced capacity of health systems;¹³ these are meanings attributed to the HV concept. It concerns the dynamic and changing state of the services to be able to provide an effective response to an event that involves from the absence of inputs to the difficulty adapting to certain contexts related to both intrinsic and extrinsic factors of users or health professionals who are imbued with their care.

It should be noted that the programmatic situation has an important space in contexts of health crisis. There is a need for mobilization in the work process and in the infrastructure, as it increases the capacity of the service to provide effective responses. When these aspects are under HV perspective, there is a scenario of scarcity of resources and investments in health, which deepens the situations of precariousness in services.¹⁴

Problematization of these items goes beyond blaming the professional or individual for placing or being placed in a vulnerable situation. It is important to understand that the issues pointed out in the items involve macrostructural situations that require management and state action as responsible for guaranteeing the right to health.¹⁵ As an example, there are items that assess scarcity or insufficiency of equipment or inputs.

Broader aspects, such as biopolitics and biopower, permeate the arrangements and the ways in which health care is produced. Taking into account that the biological body is the most important means for regulating life, unlike the know-how of the professionals who act in these ways of caring for the lives of others, the final result is a reorientation of habits and behaviors. But this active force in the health services, when workers only guide thinking about the biological body, seeking the “normal” and acting on the ways of living of others, puts subjects in socially vulnerable positions.

It is in this sense that the items point to gaps in care that can help identify more than exclusively biological situations. HV analyses using validated indicators or items help to focus the efforts on those who are most likely to suffer observable harms or not, as in this case, resulting from health services.

HV has this potential to be worked on in the field of health promotion. The inherent physical and social conditions are included in the vulnerability analyses. But the innovation that the use of the concept contributes to the clinical practice is to consider that health services are producers of vulnerability and compromise the provision of comprehensive care, even if in proportions different from those exclusively dependent on the individual aspects, means and social relations experienced by the subjects.

The low response rate of experts eligible for evaluation is highlighted as a limitation. However, the vast experience of those who answered the invitation in the HV theme, programmatic situation or construction and validation of instruments is highlighted. In addition to that, after suggestions from each specialist, the items are simpler and more objective. In this context, when analyzing the results of this item database in relation to the concepts investigated, greater understanding of the possibilities for applying the questions is the creation of data on HV or quality of the health services, transforming them into health indicators.

With this scope, it is possible to go beyond the subjective perceptions of subjects and health professionals themselves about HV or quality of the services, although the latter is an indirect perspective treated in the item database, so that elements are provided for management, subjects and health professionals alike to elaborate intervention modalities and improve health care quality.

CONCLUSION AND IMPLICATIONS FOR THE PRACTICE

Based on the elaboration of the items and on the specialists' contribution, it is possible to state that these items are validated internally and that, through good reliability among the specialists, they can be used by health professionals, managers or students, which is their contribution to the evaluation of the subjects' HV. However, these items still need to go through other processes that verify their applicability in the elaboration of instruments to assess the subjects' HV levels, with the recommendation to conduct tests in different scenarios and audiences in the health network.

In addition, the infrastructure and work process analytical components allow for a more targeted evaluation when the objective is to identify possible programmatic HV situations and when aggregated individual and social aspects encourage its agency through health promotion actions. Thus, it is expected that from, their application in research, new practices can be outlined that improve health care so that vulnerabilities are reduced.

ACKNOWLEDGMENTS

To the Graduate Program in Collective Health (Programa de Pós-graduação em Saúde Coletiva, PPSAC) of Universidade Estadual do Ceará (UECE).

• FINANCIAL SUPPORT

  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES; Master's degree scholarship).

REFERÊNCIAS

• ¹
  Florêncio RS, Moreira TMM. Modelo de vulnerabilidade em saúde: esclarecimento conceitual na perspectiva do sujeito-social. Acta Paul Enferm. 2021;34:eAPE00353. http://dx.doi.org/10.37689/acta-ape/2021AO00353
  » http://dx.doi.org/10.37689/acta-ape/2021AO00353
• ²
  Macinko J, Mendonça CS. Estratégia Saúde da Família, um forte modelo de Atenção Primária à Saúde que traz resultados. Saúde debate. 2018;42(esp. 1):18-37. http://dx.doi.org/10.1590/0103-11042018s102
  » http://dx.doi.org/10.1590/0103-11042018s102
• ³
  Stopa SR, Malta DC, Monteiro CN, Szwarcwald CL, Goldbaum M, Cesar CLG. Use of and access to health services in Brazil, 2013 National Health Survey. Rev Saude Publica. 2017 jun 1;51(suppl 1):3s. http://dx.doi.org/10.1590/S1518-8787.2017051000074 PMid:28591351.
  » http://dx.doi.org/10.1590/S1518-8787.2017051000074
• ⁴
  Mey APGFV, Morais APP, Guimarães JMX, Campelo ILB, Vieira NFC, Machado MFAS et al. Infraestrutura e processo de trabalho na atenção primária à saúde: PMAQ no Ceará. Rev Saude Publica. 2020;54(62):1-15. http://dx.doi.org/10.11606/s1518-8787.2020054001878
  » http://dx.doi.org/10.11606/s1518-8787.2020054001878
• ⁵
  Polit D, Beck CT. Fundamentos da pesquisa em enfermagem: avaliação para evidências para a prática da enfermagem. Porto Alegre: Artmed; 2019.
• ⁶
  Guimarães HCQCP, Pena SB, Lopes JL, Lopes CT, Barros ALBL. Experts for validation studies in nursing: new proposal and selection criteria. Int J Nurs Knowl. 2016;27(3):130-5. http://dx.doi.org/10.1111/2047-3095.12089 PMid:25782343.
  » http://dx.doi.org/10.1111/2047-3095.12089
• ⁷
  Hernández-Nieto RA. Contributions to statistical analysis. Mérida: Universidad de Los Andes; 2002. pp. 119.
• ⁸
  Fernandes H, Sala DCP, Horta ALM. Violence in health care settings: rethinking actions. Rev Bras Enferm. 2018;71(5):2599-601. http://dx.doi.org/10.1590/0034-7167-2017-0882 PMid:30304196.
  » http://dx.doi.org/10.1590/0034-7167-2017-0882
• ⁹
  Doucet H. “Responding to vulnerability”: ethics and CHSLD in the age of COVID-19. Ethique Sante. 2020;17(3):142-6. http://dx.doi.org/10.1016/j.etiqe.2020.07.003 PMid:32837545.
  » http://dx.doi.org/10.1016/j.etiqe.2020.07.003
• ¹⁰
  Toldrá RC, Ramos LR, Almeida MHM. Em busca de atenção em rede: contribuições de um programa de residência multiprofissional no âmbito hospitalar. Cad Bras Ter Ocup. 2019;27(3):584-92. http://dx.doi.org/10.4322/2526-8910.ctoAO1670
  » http://dx.doi.org/10.4322/2526-8910.ctoAO1670
• ¹¹
  Santos DS, Mishima SM, Merhy EE. Processo de trabalho na Estratégia de Saúde da Família: potencialidades da subjetividade do cuidado para reconfiguração do modelo de atenção. Cien Saude Colet. 2018;23(3):861-70. http://dx.doi.org/10.1590/1413-81232018233.03102016 PMid:29538566.
  » http://dx.doi.org/10.1590/1413-81232018233.03102016
• ¹²
  Machado PMO, Lacerda JT, Colussi CF, Calvo MCM. Estrutura e processo de trabalho para as ações de alimentação e nutrição na Atenção Primária à Saúde no Brasil, 2014. Epidemiol Serv Saude. 2021;30(2):e2020635. http://dx.doi.org/10.1590/s1679-49742021000200015 PMid:33886807.
  » http://dx.doi.org/10.1590/s1679-49742021000200015
• ¹³
  Florêncio RS, Cestari VRF, Pessoa VLMP, Silva MRF, Borges JWP, Santiago JCS et al. Significados do conceito de vulnerabilidade em saúde: uma revisão de escopo. Brazilian J Health Review. 2021;4(3):12817-34. http://dx.doi.org/10.34119/bjhrv4n3-243
  » http://dx.doi.org/10.34119/bjhrv4n3-243
• ¹⁴
  Silva GAB, Saraiva EV, Ferreira GJSN, Peixoto RM Jr, Ferreira LF. Capacidade do sistema de saúde nos municípios do Rio de Janeiro: infraestrutura para enfrentar a COVID-19. Rev Adm Pública (Online). 2020;4(4):578-94. http://dx.doi.org/10.1590/0034-761220200128
  » http://dx.doi.org/10.1590/0034-761220200128
• ¹⁵
  Carmo ME, Guizardi FL. O conceito de vulnerabilidade e seus sentidos para as políticas públicas de saúde e assistência social. Cad Saude Publica. 2018;34(3):e00101417. http://dx.doi.org/10.1590/0102-311X00101417 PMid:29590243.
  » http://dx.doi.org/10.1590/0102-311X00101417

Edited by

ASSOCIATED EDITOR

Gerson Luiz Marinho 0000-0002-2430-3896

SCIENTIFIC EDITOR

Ivone Evangelista Cabral 0000-0002-1522-9516

Publication Dates

• Publication in this collection
  09 May 2022
• Date of issue
  2022

History

• Received
  06 Sept 2021
• Accepted
  24 Mar 2022