Exploring the Potential for a Consensus on Reporting Guidelines for Qualitative Research using a Delphi Approach

1. PURPOSE OF THE STUDY Reporting guidelines have successfully been developed and disseminated for quantitative research designs such as experimental research, longitudinal research, cross-sectional research and meta-analyses. These guidelines include criteria outlining how to report the research procedure, the methods and results section in a manuscript to achieve consistency between reports and to increase the quality of reporting (Tate Douglas, 2011). For an overview of reporting guidelines for the most commonly used quantitative research designs we refer to Table 1. Several high quality scientific journals have adopted these reporting guidelines and use them in their instruction policy towards authors and reviewers of manuscripts. It follows that the review work conducted by e.g. editors and peer reviewers is considerably enhanced. In addition, researchers may benefit from instructions on how to report a detailed audit trail of methodological and other choices that have been made. Most of the reporting standards for quantitative research mentioned in Table 1 have been developed based on a consensus between international, methodological experts, mainly from health care disciplines, and are supported by a broad range of researchers. More recently, qualitative research designs have gained impact within several scientific disciplines, including the field of education. There have been attempts to develop a standard for reporting qualitative studies, based on a literature search or as a result of individual, academic endeavor (Tong, Sainsbury, Craig, 2007). However, the criteria identified have not yet known a considerable uptake among researchers conducting basic qualitative research. The COREQ statement developed by Tong et al. (2007) is limited to interviews and focus groups as a research technique. To date, it has not been subject to a formal consensus procedure amongst experts in qualitative research. One particular reason why it takes the international qualitative research community so long to develop a consolidated reporting standard is the variety of different paradigms that steer qualitative research as well as the broad range of designs, data-collection and -analysis techniques that one could opt for when conducting qualitative research. Whether or not it is possible to reach a consensus on such a reporting standard is a legitimate question, given the diversity within the field of qualitative research methods. Firstly, it is our aim to explore the possibility of developing a consolidated standard for qualitative research. Secondly, in the hypothetical understanding that the idea of reporting guidelines will be supported amongst qualitative researchers, our next step in the research process will be an update of the review from Tong and colleagues (2007) and a consolidation of the reporting criteria identified through a consensus procedure. The lead questions for the initial steps in this research project that will be addressed in this contribution are: What are the arguments that would plead for or against the development and use of general or design-specific reporting guidelines for qualitative research and what needs to be considered in the future development of reporting guidelines for qualitative research? 2. THEORETICAL PERSPECTIVE The development of reporting guidelines for quantitative research has been promoted by the numerous authors involved in systematically reviewing scientific literature and conducting meta-analyses, mainly because a transparent audit trail facilitates the critical appraisal of the quality of original studies considered for inclusion in systematic reviews. More recently, the number of qualitative evidence syntheses or meta-syntheses conducted has increased (Dixon-Woods, Booth, Sutton, 2007; Hannes Macaitis, 2012). The research team involved in this study draws on the idea that in order to be included in a qualitative evidence synthesis, qualitative studies should meet an acceptable level of methodological quality and validity. Clarity and transparency in reporting facilitates the critical appraisal of the methodological quality of original, qualitative studies (Hannes, Lockwood, Pearson, 2010). The interpretation of the findings derived from this study will be influenced by the research teams’ involvement in basic qualitative research as well as qualitative evidence synthesis. 3. DATA SOURCES Research team: The multidisciplinary research team that was formed to initiate this research consisted of two educational scientists, a marketing researcher and one researcher from each of the following disciplines: health care, sociology, theology/religious studies, and criminology. All of them were trained in qualitative research methods. The team was responsible for the selection of experts, the construction of the questionnaires, the feedback sessions to respondents, the data-collection and -analysis process, and the critical input in every phase of the research project. Selection of experts: The multidisciplinary team of researchers approached approximately 30 experts in qualitative research, based on personal knowledge of expert profiles within their respective fields of science. Seventeen experts agreed to participate in the study. These experts represented a broad range of different disciplines, such as education, sociology, criminology, health care, history, literacy, arts and architecture. A maximum variety in methodological expertise was achieved on three different levels. Firstly, the experts differed in the research paradigm used to shape their research: post-positivism, interpretivism, constructivism, critical theory and pragmatism. Secondly, a rich pallet of methodologies was covered by the experts, including grounded theory, systematic review methodology, phenomenology, action research, arts based methods, ethnography and mixed methods. Thirdly, the experts differed on the level of techniques used for data-collection they mastered: focus groups, observation, interviews and unobtrusive measures. 4. METHODS Delphi procedure: An argument Delphi-research approach was chosen to facilitate the research process. The main advantage of such an approach is that it can stimulate anonymous discussion, while at the same time can bridge the geographical distance between the different experts (Hasson, Keeney, McKenna, 2000). It identifies ideas, communalities and differences in opinions between experts and allows researchers to evaluate items in terms of pros and cons. Furthermore, it allows for a group dynamic process to arise, without being hindered by certain contra productive group processes such a monopolization of the discourse, marginalization of deviant opinions, dominant positions of certain authorities, group thinking, etc. The Delphi technique used in this study is characterized by three important aspects: (1) the researchers draw on the knowledge and experience of experts in the field; (2) the method is an iterative process consisting of several survey rounds; and (3) the group interaction process is anonymous and runs via questionnaires. First round: For the first round the research team developed a questionnaire that explored how the experts generally felt about reporting guidelines for qualitative research, either general or design specific. General reporting guidelines were defined as a set of general criteria that holds true for different methods, methodologies, designs or paradigms of qualitative research. Specific reporting guidelines were defined as a set of specific criteria for each particular method, methodology, design or paradigm. Experts were asked to provide us with their viewpoint on potential positive and negative aspects of both types of reporting guidelines and under which conditions they would use them. We also asked them to indicate for which particular qualitative methodologies, methods or approaches these guidelines should specifically be considered or on the other hand, might be counterproductive. In addition, we asked for their opinion on the potential effectiveness, feasibility, appropriateness, benefits, and meaningfulness of reporting guidelines. For a definition of these five key concepts we refer to Table 2. Second round: The open questions from the first round resulted in a rich pallet of information that was summarized and analyzed to serve as a basis to construct a second questionnaire in which the viewpoints from the experts could be fed back to the whole expert panel and judged by their peers. The first part of the questionnaire consisted of a series of 55 statements that the research team considered challenging and open for discussion, and evolved some conceptual thinking. Each of the experts was asked to indicate the degree to which he or she agreed or disagreed with statements from colleague experts. A five point Likert scale was used. The second part of the questionnaire listed some of the advantages and disadvantages of reporting guidelines in general and for particular target groups. It also included a list with conditions that might facilitate the uptake or use of reporting guidelines. The expert panel was asked to mark all items they agreed on and comment on the items they disagreed on. In the third part of we addressed the form of the reporting guideline as well as some process related issues to be considered for the development and implementation of reporting guidelines. These issues were again presented as a comprehensive list of items that were subject to the approval or disapproval of the experts. In each part of the questionnaire experts were encouraged to clarify their viewpoint or add their understanding. As such, the second round questionnaire provides a means of validating individual opinions and meanings of individual experts by the rest of the panel. 5. RESULTS First round We identified four different situations in which reporting guidelines may be considered: (1) never; (2) for all approaches; (3) only for well-known approaches to qualitative research such as grounded theory, document analysis, ethnographic or phenomenological research, etc; or (4) only for specific parts in a research process such as sampling strategies, description of setting, type of analysis, data-collection techniques, etc. More advantages than disadvantages of reporting guidelines were identified by the experts. These are listed in Annex 1 and 2 of the preliminary results, as part of the ongoing questionnaire in round 2 of the Delphi study. The experts also identified over 15 facilitating factors for the use of guidelines, summarized in Annex 3 of the preliminary results. In addition, they identified six different target groups who would benefit from reporting guidelines: students, teachers, peer reviewers and editors, authors/researchers, end users of research and funding organizations. Six different alternatives for the form of reporting guidelines were retrieved from the data: (1) no reporting guideline at all, (2) a reporting guideline listing a set of general principles outlining the minimum essential features of qualitative research, (3) a comprehensive general guideline or an overall framework that can be suited to encompass different qualitative methodologies and can be used across different study designs, (4) a general guideline with an annex listing out specific features for specific designs, (5) a fully comprehensive reporting guideline that includes all proper elements from all types of qualitative research methodologies, and (6) one particular reporting guideline for each particular design or methodology. The group further identified a list of process related issues or concerns to be considered in the development and implementation of reporting guidelines for qualitative research. These are outlined in Annex 4 of the preliminary results. Second round The second round will provide us with information on the extent to which the experts have agreed or disagreed with the viewpoints of their colleagues. The questionnaires from the second round have been distributed in June 2012, with a deadline for collection at the end of July 2012. The findings of the second round will be analyzed in fall 2012 and will be available for presentation at the upcoming AERA 2013 Annual Meeting. 6. SIGNIFICANCE OF THE STUDY A broad variety of different viewpoints has been identified and analyzed in the first round of the Delphi study. The preliminary analysis reveales that there is a considerable amount of topics where individual experts oppose each other in whether or not guidelines are desirable, in what particular situations they should or should not be considered and in what form they should be presented. Whether or not we, as a research team, will engage in the development of general or specific guideline criteria and what particular direction this research exercise will take will largely depend on the level of agreement reached between experts in the second Delphi round. This study provides important background information to researchers involved in guideline development as well as editors from journals considering the use of reporting criteria for qualitative research. In addition, researchers involved in discussing the particularities of their own qualitative research methods might benefit from familiarizing themselves with the arguments provided by the expert team involved in this study. REFERENCES Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, P. P., Glasziou, P. P., Irwig, L. M., … De Vet, H. C. (2003). The STARD initiative. Standards for reporting of diagnostic accuracy. Annals of Internal Medicine, 138, 40-44. Des Jarlais, D. C., Lyles, C., Crepaz, N., The TREND group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94, 361-366. doi: 10.1093/intqhc/mzm042 Dixon-Woods, M., Booth, A., Sutton, A. J. (2007). Synthesizing qualitative research: A review of published reports. Qualitative Research, 7, 375-422. doi:10.1177/1468794107078517 Hannes, K., Lockwood, C., Pearson, A. (2010). A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qualitative Health Research, 20, 1736-1743. doi:10.1177/1049732310378656 Hannes, K., Macaitis, K. (2012). A move to more transparent and systematic approaches of qualitative evidence synthesis: Update of a review on published papers. Qualitative Research. doi:10.1177/1468794111432992. Hasson, F., Keeney, S., McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32, 1008-1015. doi:10.1046/j.1365-2648.2000.t01-1-01567.x Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., for The PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. British Medical Journal, 339. doi:10.1136/bmj.b2535. Schulz, K. F., Altman, D. G., Moher, D., for the CONSORT Group (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine, 8, 18. doi:10.1136/bmj.c332. Tate, R. L., Douglas, J. (2011). Use of reporting guidelines in scientific writing: PRISMA, CONSORT, STROBE, STARD and other resources. Brain Impairment, 12, 1-21. doi:10.1375/brim.12.1.1 Tong, A., Sainsbury, P., Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal of Quality in Health Care, 19, 349-357. doi:10.1093/intqhc/mzm042 von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., for the STROBE Initiative (2008). Strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. Journal of Clinical Epidemiology, 61, 344-349. doi:10.1016/j.jclinepi.2007.11.008  TABLES Table 1: Overview of consolidated standards for reporting quantitative research Reporting guidelines Type of studies Developed by* CONSORT randomized controlled trials Schulz et al., 2010 TREND nonrandomized trials Des Jarlais et al., 2004 STROBE observational research von Elm et al., 2008 STARD diagnostic research Bossuyt et al., 2003 PRISMA systematic reviews Moher et al., 2009 *Included in the reference list Table 2: Definition of key concepts used in the questionnaire from the first Delphi round Key concept Definition Feasibility the extent to which a reporting guideline might be practical and practicable, referring to the development of reporting guidelines as well as its use/application. Appropriateness the extent to which reporting guidelines might fit with or is apt to fit in current (methodological, epistemological and ethical) debates on and trends in qualitative research. Effectiveness the extent to which a reporting guideline or standard, when used appropriately, achieves an intended effect. Benefits the extent to which reporting guidelines might be helpful, advantageous or good to something or someone. It refers to persons who might benefit from reporting standards as well as e.g. the nature and field of qualitative research itself. Meaningfulness relates to personal opinions, values, thoughts, beliefs, positive or negative experiences and interpretations regarding reporting guidelines.   ANNEXES WITH PRELIMINAIRY RESULTS Annex 1: Advantages of reporting guidelines Experts are currently discussing whether reporting guidelines: improve rigor; promote transparency (i.e. a clear view on the choices made throughout the research process); improve clarity; add to the scientific character of qualitative research; create a common understanding of the basic requirements of reporting across paradigms; help to ensure that research can actually be replicated by future researchers, practitioners or policy-makers; stimulate the debates about good practice in qualitative research; ensure that the nuances of a methodology or ideology are captured when adapted from the originally developed methodology; allow for a judgement to be made on whether or not different research approaches can be compared; contribute to a better understanding of how results are generated (i.e. explaining methods); create a common language regarding quality, rigour and credibility among the various ways of conducting qualitative research; are beneficial to the accumulation of knowledge, in the sense of build-on-each other through comprehensive information on the research process (rather than the juxtaposition of results); improve the assessment of qualitative research (good reporting facilitates appraisal) in relation to funding and publication; facilitate the inclusion of qualitative research within systematic reviews; improve the original studies because researchers reflect on their choices as well as the design of the research and how they conducted the study beforehand; enable the possibility of accountability of the process; ensure a higher baseline standard of reporting research; increase the credibility of an entire research field; open up people’s eyes to the rigour inherent in qualitative studies; help to ensure that research publications include the essential information needed to assess the validity and applicability of research findings; point out inconsistencies in reporting; facilitate criticism leading to improvements that can be implemented across many fields; stimulate the process of seeking good performance in new areas of research, such as arts based methods or research that is under scrutiny; lead to an increased cooperation across members of particular research approaches, including inter- and transdisciplinary work; reduce misunderstandings and enhance communication on qualitative methods; enable comparability; move the process of seeking good performance in new areas such as arts based research forward; will make the field more accessible to those who consume and commission research; help to transcend some of the basic level debates in each method; help to clarify the core for each method; and ensure that the nuances of a methodology or ideology are captured when applied in different ways. Annex 2: Disadvantages of reporting guidelines Experts are currently discussing the following disadvantages of reporting guidelines: they may become so extensive as to be rendered quite unattractive for use; the overkill of information in the methods section which reduces the importance of the findings; they may go beyond content and request a certain formatting of writing; they may stifle the creativity and freedom needed to adapt particular research methods and methodologies to specific target groups and settings; the risk of reporting guidelines becoming minimum formal requirements leading to correct but insufficient reporting styles; the risk to disconnect methodologies and designs from methods, because of the potential requirement of reporting guidelines to address them separately; less discretion for researchers to write their reports; they can create a frame which makes open reading/listening difficult, they would make qualitative work rigid and structured as in quantitative work and this does not go well with the philosophy of qualitative research; the potential inappropriate rejection of articles; they could result in a more ‘rigid’ thinking of students (i.e. students like protocol and ‘right’ answers); they lead to all qualitative research being homogenous and formulaic; they face the challenge of isolating specific approaches in their development phase, because authors are not always consistent in naming their methodologies and approaches; they risk to emphasize differences and disagreement and draw attention away from the commonalities that qualitative approaches share; they will result in hundreds of standards for the hundreds of mutations of approaches. Annex 3: Facilitators for the use of reporting guidelines Experts are currently discussing whether they would potentially use reporting guidelines if they help them consider the effect on the findings that following an alternative way of reporting may have; a funding body requires them to; they are generic; they are narrow; the items they contain are kept general (and don’t go into e.g. the level of length for an interview); the questions and items in the reporting guidelines include explanatory notes; the questions and items of reporting guidelines permit adaptation (i.e., addition of items relevant to the research question, methods, methodology, population); they include examples of good standards of reporting of qualitative research; they are open enough to allow for complex problems to be reported on; they do not take too long to complete; they don’t require too much expertise in qualitative methodology to assess the quality of different approaches; they are sufficiently flexible for reporting highly challenging qualitative studies; they were adopted or referred to as ‘best practice’ by a research funding agency, government body, or other agency with whom the experts sought greater influence and recognition; they have some discipline specific wording or conceptualisation; they match what qualitative researchers would expect in any particular design or methodology. Annex 4: Process and implementation issues to be considered in reporting guideline development Experts are currently clarifying their viewpoint on the following issues: • It is important to consider ethics of research when compiling reporting guidelines. • The only way to make reporting work, would be to convince the major publishing houses of the value of such guidelines, so that they would ‘strongly encourage’ their editors to apply standards or encourage adherence to guidelines. • It is mainly the question of whether publication forums provide the necessary space for reporting (see academic journals that only have a limited number of pages on offer for each item). For example, professional journals may prefer to limit the discussion of certain theoretical aspects that underlie qualitative research, making it difficult for people writing qualitative papers to follow reporting guidelines. • There is a link between reporting guidelines and funding; guidelines can suggest research is conducted in a way which requires certain resources, but often the funding bodies will not cover such resources. • Successful development of reporting guidelines should go through several stages of feedback from diverse stakeholders, representing all demographics with vested interest in the research methodology. • A widespread effort should be employed to ensure that people are aware of the guidelines, accept it as the standard, and can use it easily. Multiple publications in peer-reviewed journals, books, endorsement of the guideline by all journals that publish studies using the methodology, and dissemination at conferences and professional networks would help ensure this goal. • Reporting guidelines should be based on a systematic literature review and include experts to comment on their appropriateness. • Reporting guidelines would need to undergo pilot testing and feedback should be obtained from researchers, reviewers, etc. • Reporting guidelines will be effective only if they are seen as useful by the academic community and are accepted by those who act as gatekeepers to publication (e.g. major professional associations). • Reporting guidelines for qualitative research hold the promise of indirectly improving the quality of research. If researchers know that they must adhere to a reporting guideline upon study completion and publication, and that this guideline will aid more rigorous critical appraisal of their study, then researchers may design more rigorous and high quality research at the outset. • Reporting guidelines are intended to follow and move with trends in research over time; feedback and adjustment of the original guidelines will be necessary. • Reporting guidelines should be part of information to (peer-) reviewers for all journals where qualitative articles might be published. • Editors of qualitative journals as well as academics writing qualitative papers should be trained on reporting guidelines. Journal editors should integrate standards into the journal’s style guides and reporting templates. • The reporting guidelines will need to be evaluated within five years after their introduction to see whether reports have improved. Feedback and adjustment of the original guidelines will be absolutely necessary. • Guidelines should be open for criticism. • Difficulties in implementation might arise from poor consideration of what researchers actually need in the field. • Coordination by a trusted organization might help to minimize territoriality.