Documento sin título

ORIGINAL ARTICLE / ARTÍCULO ORIGINAL/ ARTIGO ORIGINAL

 

Themes and situations that cause embarrassment among participants in research in which questionnaires or interviews are used

 

Temas y situaciones que ocasionan vergüenza a los participantes en investigaciones que utilizan cuestionarios o entrevistas

 

Temas e situações que ocasionam vergonha aos participantes em investigações que utilizam questionários ou entrevistas

 

 

Juliana Dias Reis Pessalacia1; Cléa Regina de Oliveira Ribeiro2; Dinéia Massuia3

 

1 RN, Ph.D., Professor. Universidade Federal de São João del Rei, Divinópolis (MG),Brazil. email: juliana@pessalacia.com.br.

2 Philosopher, Ph.D., Retired professor. University of São Paulo São Paulo, São Paulo (SP), Brazil. email:clearib@terra.com.bremail:clearib@terra.com.br.

3 RN. Centro Universitário de Rio Preto, São José do Rio Preto (SP), Brazil. email:dineiamassuia@hotmail.com.

 

Receipt date: April 28th 2012. Approval date: September 19th 2012.

 

Article associated with the research: Bioethics and research: research subjects' perception about embarrassing topics and situations in research involving inquiries

Subventions: none.

Conflicts of interest: none to declare.

How to cite this article: Pessalacia JDR, Ribeiro CRO, Massuia D. Themes and situations that cause embarrassment among participants in research in which questionnaires or interviews are used. Invest Educ Enferm. 2012;31(1): 70-77

 


ABSTRACT

Objective. To recognize the themes and situations that could make research participants feel embarrassed when questionnaires or interviews are used. Methodology. Quantitative and descriptive study, developed in 2008, involving a stratified sample of 1,1149 subjects who qualified the degree of shame in view of potentially embarrassing themes and situations. Results. For the research participants, it is embarrassing to answer questions related to the following themes: betrayal (50%), physical violence (42%), sexual harassment (42%), psychological violence (40%) and death of loved ones (38%). The situations that most frequently causes embarrassment were: start of the survey or interview without requesting informed consent (83%); lack of information about the type of questions that would be addressed (79%), lack of guaranteed anonymity (78%), or use of images (66%) or a recorder (58%). Conclusion. Themes and situations were identified that caused embarrassment among participants in research in which questionnaires or interviews were used, which should be considered in the ethical evaluation of studies.

Key words: bioethics; ethics, research; data collection.


RESUMEN

Objetivo. Reconocer los temas y las situaciones que podrían ocasionar vergüenza en los participantes de investigaciones en las que utilizan cuestionarios o entrevistas. Metodología. Estudio cuantitativo de tipo descriptivo realizado en 2008, en el que se hizo muestreo estratificado de 1 149 sujetos, quienes calificaron el grado de vergüenza ante temas y situaciones potencialmente embarazosas. Resultados. Para los encuestados es embarazoso responder preguntas relacionadas con los siguientes temas: traición (50%), violencia física (42%), acoso sexual (42%), violencia psicológica (40%) y muerte de personas cercanas (38%). Las situaciones que produjeron frecuentemente vergüenza fueron: inicio de la encuesta o entrevista sin solicitar el consentimiento informado (83%), la falta de información sobre el tipo de preguntas que serían abordadas (79%), la falta de garantía del anonimato (78%), el uso de imágenes (66%) o de grabadora (58%). Conclusión. Los temas y situaciones que ocasionan vergüenza en los participantes de investigaciones que utilizan cuestionarios o entrevistas deberían ser tenidos en cuenta en la evaluación ética de los estudios.

Palabras clave: bioética; ética en investigación; recolección de datos.


RESUMO

Objetivo. Reconhecer os temas e as situações que poderiam ocasionar vergonha nos participantes de investigações nas que utilizam questionários ou entrevistas. Metodologia. Estudo quantitativo de tipo descritivo realizado em 2008, no que se fez amostragem estratificada de 1,149 sujeitos que qualificaram o grau de vergonha ante temas e situações potencialmente embaraçosas. Resultados. Para os interrogados é embaraçoso responder perguntas relacionadas com os temas de: traição (50%), violência física (42%), acosso sexual (42%), violência psicológica (40%) e morte de pessoas próximas (38%). As situações que mais freqüentemente produziram vergonha foram: início da enquete ou entrevista sem solicitar o consentimento informado (83%) a falta de informação sobre o tipo de perguntas que seriam abordadas (79%), a falta de garantia do anonimato (78%), ou o uso de imagens (66%) ou de gravadora (58%). Conclusão. Identificaram-se temas e situações que ocasionavam vergonha nos participantes de e investigações que utilizam questionários ou entrevistas, os que deveriam ser tidos em conta na avaliação ética dos estudos.

Palavras chaves: bioética; ética em pesquisa; coleta de dados.


 

 

INTRODUCTION

The use of research instruments containing questions such as questionnaires and interviews is frequent in nursing and in the fields of research in the Humanities, Social Sciences, and various areas of health. In research, however, ethical limits in the handling of these instruments are often unconsidered - in the sense of focusing attention on the possible risks to those participating in an investigation. It is thought that the risk to such subjects - as opposed to those participating in a clinical investigation - is almost zero, as physical harm is not involved. In addition to this, it is considered that that it is difficult to define and measure the potential harm, that is, to evaluate its long-term effects.1 For this reason, the importance of reflecting on these types of collection instrument (questionnaires and interviews) as possible causes of non-physical risks (psychological, moral, social and spiritual) seems not to be very clear to the researchers. It follows that administering such instruments to research subjects may cause harm, unease and embarrassment.

In one study undertaken with researchers concerning the perception of risk in using questionnaires and interviews, it was identified that the researchers believe the word 'risk' to be too strong for the research participants, and that it is generally associated with physical risks, and the relationship between life and death. Possibilities of moral, emotional and social risks were mentioned only a few times.2 Although when speaking of risk, its relation to the physical dimension is inevitable, to take into account physical harm alone is also to fail to conceive of the human being as a social individual, imbued with values, culture, beliefs and emotions.2 It should be highlighted that the possibility of risk exists for any type of issue addressed in research. Depending on the context in which the subjects find themselves, and when issues are being addressed which lead them to share personal and - frequently - intimate aspects of their lives, the possibility of embarrassment may be greater. Here one should include matters concerning facts which profoundly interfere in people's lives or personal experiences, whether in the present or the past.3 In addition, one should take into account the research subjects' conditions of circumstantial vulnerability. Vulnerability may be described as a person's inability to protect their own interests due to impediments4 where the main characteristic is the limitation of capacity or freedom.5 The subject's condition and the type of topic, however, are not the only things which can influence the risk of embarrassment; the way the research is carried out itself can also do so, with the lack of elucidated information, confidentiality, anonymity and respect for the subjects' privacy standing out. The probable impact of the use of questionnaires or interviews on the patients is frequently not considered, and consequently the balance between benefit and harm is not entirely explored.6

Given that the present study focuses on situations and issues which can bring risks to research subjects, with the degree of embarrassment being used as the parameter for the evaluation of risk, it is appropriate to explain some conceptions of the term 'embarrassment'. Embarrassment, also termed 'unease', is a central construct when dealing with investigating aspects linked to the feelings. One can assert that the feeling of embarrassment influences many behaviors of subjects across different cultures. The feeling of embarrassment is an emotional experience or state which may be expressed as a form of social anxiety, related to shyness and shame, resulting from the perspective or evaluation of others in real or imaginary situations.7 Embarrassment may be perceived as the result of the worry which people show concerning their observed behavior and the desire to act in line with others' expectations and interests. It covers a psychological and also physiological response, resulting from the effects of going against social demands.7 Thus, taking the concept of embarrassment as a base, one cannot ignore the possibility that any intervention, even in the psychological or social fields, may mobilize content with which the subject is not yet disposed to enter into contact.8

The present study aimed to identify and compare risk questions and situations in surveys with questioning (questionnaires and/or interviews) based on the evaluation of the 'degree of embarrassment' attributed by possible research subjects, subdivided into the groups: vulnerable and not vulnerable.

 

METHODOLOGY

This is a prospective, cross-sectional study, with a quantitative approach and a descriptive nature. Data collection was carried out in sectors of hospital with over 500 beds, located in the north-western region of the state of São Paulo, Brazil. The subjects were selected using the technique of stratified sampling, proportional to the size of the population. A total of 1 149 (100%) subjects were approached, covering the following groups: vulnerable: oncology patients (201, 17.5%), nursing personnel from the critical care sector (44, 3.8%), undergraduate students (209, 182%), pregnant women (200,17.4%), mothers of newborns (96, 8.4%), and oncology patients' companions (200, 17.5%) and not vulnerable: passers-by (199, 17.3%).

The choice of the vulnerable populations was based on the conditions of vulnerability described in the Brazilian resolution on ethical norms in research involving human beings. This resolution describes as criteria the cases in which there may be restriction on freedom or on the necessary clarification, and the cases in which the subjects may be exposed to specific conditioning or to the influence of authority.9 Such a choice was also based on a study which identified such populations as being the most frequently approached in research by professional nurses.10 The 'not vulnerable' population, on the other hand, could not present any of the conditions of vulnerability as specified in the above-mentioned resolution. The 'not vulnerable' population was approached in a public square near a commercial center in the municipality where the research was carried out. A specific script was prepared for approaching the populations, based on assumptions found in the literature concerning the type of topics and situations that can be discomforting for research subjects and containing the items necessary for achieving the desired objectives.

Before beginning data collection, the research project was evaluated and approved by the Research Ethics Committee (CEP) of the institution where the data on the vulnerable populations was to be collected, under approval decision no 037/2007. Following the approval, a pilot study was carried out with 10 subjects so as to test the instrument's applicability, regarding its content, clarity and understandability, the aim being to improve it and correct shortcomings. It should be noted that this instrument constituted a simulation of research, that is, the subject was approached in the following way: If you were invited to participate in research, which of the following topics and situations would most embarrass you to respond to or talk about? In this way, the subjects were able to indicate which types of topics, questions and/or situations in surveys with questioning would make them embarrassed, without needing to talk about them. The responses identified were classified into the categories: 1: Not discomforting, 2: A little discomforting, 3: Discomforting, 4: Very discomforting, and were later discussed according to data found in the literature, indicating those responses which most called attention.

For statistical analysis of the data, the researchers used frequency distribution (absolute and percentage values), the Z-test for comparison of two proportions, and the Chi-squared test for comparison of more than two proportions. The data was analyzed, considering a level of significance of 0.05, given the dispersion of the phenomenon studied. The statistical software used for the analysis was Minitab, version 14.

 

RESULTS

When a script containing sensitive questions and situations was administered to possible research subjects, [N=1 149 (100%) subjects] subdivided into the Vulnerable and Not-vulnerable groups, the authors found some topics/situations indicated as causing greater embarrassment when spoken about during surveys, in both groups, with the results presented below: betrayal (575, 50.0%), physical violence (499, 43.4%), sexual harassment at work (482, 41.9%), sexual abuse (468, 40.7%), psychological violence (458, 39.9%) and the death of people close to one (438, 38.1%). Also identified as potentially-discomforting topics were: sexual behavior (777, 67.6%); sexually-transmitted diseases - STDs - (708, 61.6%), number of sexual relations per week (666, 58.0%), number of sexual partners (659, 57.4%), sexual difficulties (575, 50.0%) and questions about behavior in matters of romance, dating etc. (551, 48.0%). These results were identified based on the combination of the collection instrument's constant categories Discomforting and Very discomforting. It should also be noted that each subject was able to mark more than one item in the instrument.

Note that although the topic of Betrayal was mentioned as being potentially discomforting by the majority of the subjects (575, 50.0%), the result of the Z test for comparison of proportions between the sexes, for each situation of marital status, and according to the degrees of Discomforting and Very discomforting did not identify significant differences (p>0.05) between men and women or among the different marital statuses when talking about this topic. The questions involving human sexuality were cited by the subjects as being the most discomforting for talking about in surveys. The results of the Z test for comparing proportions between the sexes (male and female) according to the categories Very discomforting and Discomforting for the topics of sexual behavior, number of sexual partners and number of sexual relations, showed the degree of embarrassment to be significantly higher (p< 0.05) for women. Also through the Z test, percentages for embarrassment for the topic of sexual behavior were compared between Roman Catholics (692, 60.2%) and Protestants/Evangelicals (240, 20.9%), with no significant difference (p >0.05) in embarrassment being identified between the two branches of Christianity. The choice of these churches was due to their higher absolute frequencies identified in the study.

The results of the Chi-squared test for comparing between three populations (vulnerable oncology patients, vulnerable companions of oncology patients, and not-vulnerable passers-by) for the topics of serious accident, experience of illness, illness of a family member, illness of a friend, hospitalization and death of people close to one, evidenced that in the population studied, there was no significant difference (p>0.05) in the degree of embarrassment for the vulnerable populations (patient and companion) and not vulnerable. A statistically significant difference (p<0.05) was, however, identified for the vulnerable population 'companion of oncology patient' and the vulnerable population 'oncology patient', evidencing greater embarrassment in the first population. Statistical tests were undertaken with the aim of comparing degrees of embarrassment, considering the 'vulnerable' and 'not vulnerable' groups. However, the comparative tests administered did not evidence a significant difference in embarrassment among the populations mentioned.

The present study also evidenced greater embarrassment resulting from the conditions of the place where interviews or questioning could take place, the following situations being cited: the researcher initiating the survey without requesting the subject's authorization (956, 83.3%), lack of prior information concerning the type of issues to be addressed (902, 78.6%), lack of guarantee of confidentiality and anonymity (895, 78.0%), the use of images (757, 65.9%) and the use of a recorder (668, 58.2%).

 

DISCUSSION

The topic of betrayal was cited as potentially discomforting by the majority of the subjects. The literature indicates that the issue of betrayal has an important relationship with the changes that have occurred in the concept of 'marriage'. The valuing of love and fidelity as fundamental elements for marriage has been altered as a result of the advent of capitalism and the dissemination of a liberal and feminist ideology.11 Therefore, such liberalism may be associated with the increase in rates of betrayal. The results of the statistical test comparing the sexes on this topic, however, did not evidence a significant difference for embarrassment for men and women. In this way, it was observed from the results that although liberalism and feminism have caused an increase in the rates of betrayal, the topic is still considered discomforting for both sexes.

The topic 'violence' was also mentioned frequently by the participants. 'Violence' is conceptualized as the use of physical strength capable of overcoming the victim's strength and 'threat' as the promise to cause considerable harm. The literature further indicates that the true incidence of violence is unknown, it being believed to be one of the world's most under-reported and under-recorded conditions.12 In this way, this study's results suggest that one of the possible causes for the under-reporting may be the fact that the subjects feel embarrassed by such situations. However, the importance of undertaking complementary studies on the real causes for this under-reporting is emphasized. Another topic mentioned was sexual harassment at work. The literature indicates that this practice has been recurrent in recent years, featuring in the media in general, in claims made by various categories of workers and in the specialized literature.13 In this respect, the present study's results support the data from the literature, given that although the workers feel embarrassed, spaces for discussion and mobilization so as to control such situations have been occurring.

It stands out that issues involving sexual and amorous behavior were mentioned most as being potentially discomforting in surveys. Regarding the administration of the Z test for comparing proportions between sexes (female and male) for the topics of Sexual Behavior, Number of Sexual Partners and Number of Sexual Relations per Week, it was evidenced that the embarrassment was greater for the women in the group studied. Such results may be related to the understanding of human sexuality as a production of the culture rather than as a biologically determined instinct. In this context, the representation produced is that female sexuality is passive, directed at men's pleasure and that sexual pleasure not only is not a necessary component, but is dispensable for procreation.14

Another topic presenting a high level of embarrassment was the STDs; such results corroborate those found in the literature, since acquiring a Sexually Transmitted Disease (STD) can provoke intense suffering in the subjects. This suffering is objectified in the disease's symptoms and treatment, in the guilt for having infected the partner, and in the repercussions in the subject's affective and sexual life. Having a sexual disease is a situation of unease and discomfort. STDs are filled with negative social and cultural values. The stigma is very strong and the subjects are ashamed for having acquired an STD.15

It is relevant to discuss the results of the Chi-square test for comparing between the populations 'Vulnerable oncology patients' and 'vulnerable companions of oncology patients'; here it was identified that, for the population studied, the patient's companion could feel more discomforted/embarrassed to talk about issues related to accidents, hospitalizations and death that the patient population. In regard to this, the literature describes that although finitude is inherent to life, Western man still has death as one of his major concerns. Accepting the death of another has been frequently indicated as more problematic, since the loss may become an ordeal, as much for the person facing the loss as for those around her, who often do not know what to do. Mourning is a complex experience which can transcend the individual scope.16

Another datum which called the authors' attention is that the majority of the results from the comparative statistical tests did not evidence a significant difference in embarrassment between the 'vulnerable' and 'not vulnerable' groups. The authors suppose that the fact of there not being a difference in embarrassment between these categories is due to the dynamics and continuous character of the experience of vulnerability, which does not represent a sequence of linear events, but, rather, repetitive and interactive events, permitting there to be alternation in relation to the consequences.17 Thus, the subjects identified as not vulnerable circumstantially by the criteria proposed in the Brazilian resolution on ethics in research involving human beings could, at the time of responding to the collection instrument, be in other types of disadvantageous personal or social circumstances, just as the vulnerable subjects could present such conditions in a lesser intensity.

The data also indicates that the situations to which the subjects are exposed in surveys using questioning can result in greater embarrassment. The results indicated that the risk of embarrassment is mainly related to the lack of information and agreement/authorization and clarification on the part of the researchers vis-à-vis the participating subjects. It should be emphasized that such risks can be minimized or even eradicated in research through appropriate administration of the Terms of Free and Informed Consent. The ethicality of a research project entails, primarily, the free and informed consent of the target individuals and the protection of vulnerable groups and the legally incapable. For this, a term must be elaborated using accessible language, including all the information related to the study and offering the subjects complete liberty to refuse to participate or to withdraw their consent. The act of consenting must be genuinely voluntary and based on adequate disclosure of information.18

The lack of prior clarification regarding the type of questions to be addressed, therefore, deserves some consideration. Considering the ethical principle of autonomy, one of the main steps for risk prevention is the guarantee that all the subjects have truly understood the objectives, as well as the instruments to be used in the study. This principle refers to people's freedom of thought and action, in which they must feel free and capable to decide and choose between the alternatives presented to them, without internal or external coercion.19 Before initiating data collection, therefore, the researcher must duly clarify the subjects as to the type of issues to be addressed and, should the participant not feel comfortable conversing on such topics, the researcher must guarantee the same the freedom to withdraw from the study at any time. In this way, the risk of embarrassment resulting from participating in the research can be minimized or removed through the expression of the desire not to participate.20

The participants also attributed high levels of embarrassment to the issue of lack of guarantee of confidentiality and anonymity in research. The difference existing between anonymity and confidentiality should be stressed: in anonymity, the researchers are unable to establish a link between the data and the research subjects to which it refers, whereas with confidentiality, although the researchers can establish links between the two, they commit themselves not to reveal these.21 The issue of using images or recordings in research was also mentioned as discomforting. However, it is known that such methods are extremely important in observational and phenomenological research, and that if the appropriate preventive steps are taken, such embarrassment can be reduced to practically zero. The principal recommendation is that the subject should be previously elucidated about the objective of the use of such images or recordings and how they shall be made, and that it should be made clear that the subject may withdraw from the research whenever he or she wishes.22 In this way one can see the importance of analyzing the differences between research projects of a clinical and pharmacological nature and those from the Humanities or social sciences, so as to show the latters' specific character, whose diversity and methodology is threatened by the biomedical and experimental hegemony.1

One should also note that the definition of the Brazilian resolution does not identify its grounding in the paradigms of positivism, and asserts that the provisions of the same must be applied to all types of research involving human beings. Thus the reader, and especially the ethics committees, are left with the false impression that there is one, and one only, way of undertaking a scientific investigation, and that the definition is applicable to this.23

Conclusion. Although the present study has neither exhausted the analysis of all the possible topics and situations of embarrassment or risk in research using questionnaires and/or interviews, nor explored such risks considering various other conditions of circumstantial vulnerability not addressed in the present study, given the diversity of such questions and populations, it brings important pointers concerning the risks of research which use such instruments. The identification of topics or situations or risk in research using questionnaires and/or interviews, therefore, is relevant for Nursing, since it uses such instruments frequently in its research, and also for the field of Research Ethics, as it allows researchers and CEPs to make use of support for better ethical analysis of research, contrasting risks and benefits.

 

REFERENCES

1. Rovaletti ML. Ethical review in humanities and social sciences. Scientific research: between freedom and responsibility. Acta Bioeth. 2006; 12(2):243-50.

2. Melo LMC. The comprehension of dental researchers about ethics in research involving in human beings [Dissertation]. São Paulo: Faculty of Dentistry, University of São Paulo; 2002.

3. Corbin J, Morse JM. The Unstructured Interactive Interview: Issues of Reciprocity and Risks When Dealing With Sensitive Topics. Qual Inq. 2003; 9(3):335-54.

4. La Rocca S, Martínez G, Rascio A, Bajardi M. Biomedical Investigation and informed consent in vulnerable populations and individuals. Acta Bioeth. 2005; 11(2):169-81.

5. Aran M, Peixoto Júnior CAP. Vulnerability and bare life: bioethics and biopolitics today. Rev Saude Publica. 2007; 4 (5):849-57.

6. Evans M, Robling M, Rapport FM, Houston H, Kinnersley P, Wilkinson C. It doesn't cost anything just to ask, does it? The ethics of questionnaire-based research. J Med Ethics. 2005;28(41):41-4.

7. Gouveia VV, Singelis TM, Guerra VM, Santos WS, Vasconcelos TC. Self-construal and embarrassment. Psico (Porto Alegre). 2005; 36(3):231-41.

8. Ministério da Saúde. Conselho Nacional de Saúde. Resolução n° 196, de 10 de outubro de 1996. Aprova as diretrizes e normas regulamentadoras de pesquisa envolvendo seres humanos. Diário Oficial da União (1996 Oct 16).

9. Jorge Dias HZ, Gauer GJC, Rubin R, Valério Dias A. Psichology and bioethics: dialogues. Psicol Clin. 2007; 19(1):125-35.

10. Reis JD, Tavares BB. Ethical issues in nursing. Analysis of ethical issues in research involving human. Nursing (São Paulo). 2004; 78(7):46-50.

11. Archanjo DR. A debate without struggle: a discussion of divorce in Congress (Brazil 1951-1977) [Dissertation]. Curitiba: Federal University of Paraná; 2008.

12. Dresset J, Caballero M, Juliano Y, Prieto ET, Marques JA, Fernandes CE. Study of mechanisms and factors related to sexual abuse in female children and adolescents. J Pediatr. 2001; 77 (5):413-9.

13. Meneguel SN, Barbiani R, Steffen H, Wunder AP, Rosa MD, Rotermund J, et al. The impact of women's groups on gender vulnerability. Cad Saude Publica. 2003; 19(4):955-63.

14. Meyer DEE, Klein C, Andrade SS. Sexuality, pleasures and vulnerabilities: educational implications. Educ Rev. 2007; 46:219-39.

15. Andrade LS, Nóbrega-Therrien SM. The male sexuality and the vulnerability to HIV/AIDS. DST J Bras. Doencas Sex Transm. 2005; 17(2):121-6.

16. Niconielo AP, Monteiro CZ, Assumpção MT, Lopes-Júnior C, Silva RHA, Sales-Peres A. Speech-language pathology researches: the importance of bioethics. Arq cienc saude. 2005; 12(4):200-5.

17. Pettengill MAM, Angelo M. Family vulnerability: concept development. Rev Latino-am Enfermagem. 2005; 13(6):982-8.

18. Domingos B, Maluf MR. Experiences of loss and mourning among school-children aged 13 to 18. Psic Reflexao Critica. 2003; 16(3):577-89.

19. Alves DCI, évora YDM. Ethical issues linked to the professional practice of nurses that are members of a hospital infection control commission. Rev Latino-am Enfermagem. 2002; 10(3):265-75.

20. Pessalacia JDR, Ribeiro CRO. Vulnerability and risk in research with interviews. Nursing (São Paulo). 2007; 10(114):509-13.

21. Pessalacia JDR. Risks, impairment and damages in Bioethics: a study on the risks in research with questionnaire and/or interview [Dissertation]. Ribeirão Preto (SP):University of São Paulo at Ribeirão Preto College of Nursing; 2006.

22. Berardineli LMM, Santos MLSC. Ethical questions in the research of nursing subsidized for audiovisual method. Texto Contexto Enferm. 2005; 14(1):124-30.

23. Guerriero ICZ, Dallari SG. The need for adequate ethical guidelines for qualitative health research. Ciênc. saúde coletiva. 2008; 13(2): 303-11.

 

 

Abstract : 347

Article Metrics

Metrics Loading ...

Metrics powered by PLOS ALM


Esta publicación hace parte del Sistema de Revistas de la Universidad de Antioquia
¿Quieres aprender a usar el Open Journal system? Ingresa al Curso virtual
Este sistema es administrado por el Programa Integración de Tecnologías a la Docencia
Universidad de Antioquia
Powered by Public Knowledge Project