Cochrane systematic review as a PhD thesis: an alternative with numerous advantages

 

Human resources are the European Union’s main asset. They are central to the creation and transmission of knowledge and a deciding factor in each society’s potential for innovation (1). European Commission’s objective for European universities is to attract and maintain highly-qualified staff and students in order to support their research capabilities. A specific measure of a country’s potential research capacity is provided by the number of PhD students and PhD holders (2). The overall number of PhDs, or equivalent, produced in the different European countries either shows stability over the years or a slight increase. This seems to be insufficient to meet the requirements of the European Commission objectives; therefore, a substantial increase in the number of PhDs is critical (3).

In developing countries, sometimes referred to as the countries of scientific periphery (4), the rates of successfully defended doctoral theses remain discouragingly low (5). A possible way of improving this situation was suggested in the recommendations of the European University Association conference where it is stated that “new models of doctoral programs should be established, studied and supported” (6). Hereby, we propose that production of a systematic review within The Cochrane Collaboration should be considered a valid mode of achieving doctoral title in medicine and related professions.

The aim of this article is to briefly describe The Cochrane Collaboration and its mission, explain why a Cochrane systematic review is an original research appropriate for the doctoral-level researchers, describe the advantages and challenges of accepting Cochrane systematic reviews as PhD theses, and suggest the ways of integrating Cochrane systematic reviews in the existing doctoral programs.

The Cochrane Collaboration is a unique worldwide non-profit organization that aims to help people make well-informed decisions about all forms of health care by preparing, maintaining and promoting accessibility of systematic reviews of the effects of health care interventions (7). A systematic review is a literature review focused on a single question and uses the evidence from a number of studies to appraise, to pool and to synthesize all available data relevant to that question. It may contain meta-analysis, a statistical method that combines numerical results of several different studies, but there are also systematic reviews where meta-analysis is not appropriate for various reasons (8,9). It is important to stress that systematic reviews are scientific studies so they use rigorous methods that are evidence-based, transparent and reproducible, which makes them very different from traditional narrative reviews (10). Cochrane systematic reviews are published in the Cochrane Database of Systematic Reviews (CDSR), a monthly publication within the Cochrane Library, which in 2009 reached an impact factor of 5,65.

The conclusions of the 2005 Salzburg conference state that doctoral study should be based on knowledge acquisition through original scientific research (11). Original research may be defined as formulating and testing a unique hypothesis with an observational or interventional study (10). The production of Cochrane systematic review follows the same basic steps as any primary observational research, but in some aspects it is particularly advantageous for PhD candidates (10). For example, the research question in all cases has to be well defined and clinically relevant, yet the choice of this question in systematic reviews is not dependant on available resources or existing research facilities, but mostly on the personal interest of the candidate.

The feasibility of a systematic review is easy to establish through a quick preliminary search of the existing literature. In The Cochrane Collaboration, great attention is given to the development of the review protocol, which is thoroughly assessed by editors and external peer-reviewers (12). In the protocol, authors have to specify in advance all the methodological aspects of their intended systematic review, including inclusion and exclusion criteria, search strategy, data extraction and management, dealing with missing data, assessment of risk of bias in included studies, methods of data synthesis, etc (13). In primary research conducted by PhD candidates, protocols are often written as a formality and scrutinized only superficially, which deprives the candidates of important learning opportunities in the beginning phases of their research endeavor.

After having the protocol accepted and published in the CDSR, candidates begin to work on the actual systematic review, which in itself is a valuable learning experience. Literature search is a chance to acquire thorough understanding of different electronic databases and other sources of scientific information. By screening thousands of titles and articles, candidates get a feeling of a broad universe of literature related to their topic of interest, learn how to discern relevant from non-relevant, and train their patience and focused attention. Good understanding of various study designs is necessary to accurately identify potential articles for inclusion.

Looking for the desired information in full reports of the included studies allows candidates to realize that many studies, especially older ones, are poorly reported (14). This offers a valuable lesson on numerous intricacies of conducting and reporting primary research studies. Another insight may be related to the fact that studies usually use a wide range of outcome measures for the same condition, which makes it difficult to compare the results across different studies. This may encourage candidates to start distinguishing outcomes that are important to the patients from the outcomes that are less important to the patients. The assessment of the risk of bias is an obligatory step in the making of Cochrane systematic reviews and teaches candidates about the possible sources of bias in primary research and how to identify them.

Data extracted from the included studies have to be classified and organized into evidence tables, and decisions have to be made on the appropriateness of data for meta-analyses. Writing the full report of a Cochrane systematic review is almost impossible without a careful documenting of every step in the making of review, which teaches candidates record-keeping and data-management skills.

Conducting a Cochrane systematic review can give PhD candidates the opportunity to acquire not only a high level of content and methodological expertise, but also the capacity to learn and solve problems by using critical and analytical thinking. This capacity is considered one of the key “generic” and “transferable” skills necessary for future researchers (6). Other key transferable skills are communication and organization skills, ability to lead projects and work in international teams, and flexibility (6). Cochrane review authors are in good position to develop these skills also, as The Cochrane Collaboration is characterized by a high level of internationality, volunteer-based work and constant negotiation between editors, referees, and review authors (12). Formation of international teams of review authors is strongly encouraged and sometimes even facilitated by the editors of Cochrane Review Groups (12), which is especially beneficial for the young researchers who seldom have the access to international networks of potential research collaborators. Joining a team of Cochrane review authors from different parts of the world may signify the beginning of long lasting partnerships and building of a new research network independent from the PhD candidate’s supervisor.

It is well recognized that high-quality supervision and mentorship are prerequisites for the development of research profession and careers (15). At the same time, inadequate supervision is wide-spread in the countries of scientific periphery, as low quality performance tends to fall into a vicious circle of inadequacy (4). The lack of quality control of doctoral programs and poor supervision are the most frequent causes of unnecessary delays in completion of doctoral studies, both in developing (5), and developed countries (16).

The Cochrane Collaboration can offer the review authors some elements of mentoring, primarily related to educational functions such as tutoring and coaching (17). Cochrane Centers and Branches regularly organize training events for prospective authors and Review Groups as editorial bases of The Cochrane Collaboration also offer ongoing support and advice to authors working on their reviews (12). Other mentoring functions such as networking or monitoring are exhibited when Cochrane Review Groups assist in finding coauthors from other countries or inquire about the progress of work on a systematic review (12). Together with the supervisor and other actors at the home university, persons from different Cochrane entities form a mentoring network that can support the candidates on their path to doctoral title (18).

As explained above, producing a Cochrane review is beneficial for individual PhD candidates and offers them opportunities to:

a) engage in an original research study;

b) gain topic-related and methodological expertise;

c) acquire key “generic” and “transferable” skills, and

d) develop functional networks of mentoring and research partnerships.

The benefit, however, extends beyond the individual PhD candidate: a high impact factor of the CDSR indicates that Cochrane reviews are widely cited in the scholarly literature, many clinical guidelines are based on the Cochrane reviews (19), and CDSR has to be consulted as a source of evidence before any product is put on the essential medicines list of the Croatian Institute for Health Insurance (20). Even when a Cochrane review is produced by a PhD student (of course, under the supervision of more experienced authors), such a review represents substantial contribution to the global body of knowledge and has a potential to influence the clinical practice. On the other hand, primary research conducted by the PhD students is often insignificant and difficult to publish in internationally visible journals (21). Some authors convincingly argued that inadequately produced and reported research is an “avoidable waste” that should be prevented from occurring and that systematic reviews are the main method of the prevention (22). In response to these arguments, a high-profile medical journal The Lancet recently announced that they will ask authors of all research reports submitted after August 1, 2010 to put their work into the context of what has been done before, by either reporting their own, up-to-date systematic review or citing a recent systematic review done by others (23).

In some fields of biochemistry and molecular biology, primary research tends to be relatively inexpensive: bioinformatics, chemometrics, bio­stat­i­sti­cs, and biochemical applications of quantum chemistry are just few examples. Producing a Cochrane systematic review also imposes a relatively small financial burden, which can be a vital advantage for the candidates in low- and middle-income countries. Basically, expenses of producing a Cochrane review are related to administrative work (e.g. paper copying, teleconferencing) and library services (e.g. obtaining full texts of articles), but the latter can often be reduced with the assistance of Cochrane Review Groups’ Trial Search Coordinators. Training costs can vary substantially – basic training is mostly provided by the Cochrane Centers or Branches and offered at a low price or free of charge and Cochrane Handbook for Systematic Reviews of Interventions is freely available online. However, one or more short visits to collaborative centers or editorial bases abroad may prove beneficial, or even necessary for first-time review authors, and such visits may require more funding. In European countries, PhD students are eligible for different funds aimed to support their mobility (24), and Cochrane Centers or Branches can occasionally support their study visits abroad.

Relatively low financial costs make a Cochrane review particularly attractive for part-time PhD candidates, who are not research fellows involved in a financed research project and have to combine their regular work with their doctoral study. At the same time, producing a Cochrane review is usually very time-consuming and labor intensive, which may pose a serious challenge to “part-time” PhD candidates.

There are several other challenges in accepting Cochrane reviews as PhD theses. In theory, any PhD thesis should result primarily from the work of the candidate, but in Cochrane review teams the workload can be rather evenly distributed, so it is possible that some coauthors will not be directly involved in all the steps of the review process. However, the first author is usually the one who does the majority of work, so the Cochrane reviews should be accepted as PhD theses only if a candidate is the first author. The peer review process conducted by the Cochrane Review Groups is concerned primarily with the quality and relevance of the end product – protocol or review, and not with the level of contribution by individual coauthors. The candidates’ contribution to Cochrane reviews should be assessed by the assigned PhD committee, which needs to make sure that candidates thoroughly understand all the steps and procedures in the making of a Cochrane review and that they actually performed the majority of these procedures themselves. For this to happen, PhD committees have to include at least one experienced Cochrane reviewer, which may not be easy to achieve in countries where only a few people have been involved in the work of The Cochrane Collaboration.

Another problem is related to the so-called empty reviews – the ones that do not include any primary study because of narrowly defined inclusion criteria or a paucity of research on the topic. Without any primary studies to include and analyze, a systematic review may still be a valuable contribution to the literature, but does not entail enough learning opportunities and engagement for the candidate to be accepted as a PhD thesis. The similar objection may be raised with regard to the updates of Cochrane reviews. A decision on the minimum number of studies that should be included in a review to justify its acceptance as a PhD thesis is not a straightforward one and may require careful judgment and negotiation on a case by case basis.

An important downside of accepting Cochrane systematic reviews as PhD thesis is the fact that such a program does not provide candidates with technical experience and laboratory skills, which are especially important for medical biochemists.

Croatian Branch of Italian Cochrane Center (CBICC) was established in 2008 to promote evidence-based medicine, The Cochrane Collaboration and The Cochrane Library, and to encourage Croatian healthcare workers to become authors of Cochrane systematic reviews (25). One of the main goals of the CBICC business plan was to enable Croatian doctoral students to do a Cochrane systematic review as a PhD thesis. This was deemed an important goal since doing a systematic review early in a scientific career may encourage Croatian doctoral students to make important contributions for clinical practice and to stay involved with The Cochrane Collaboration throughout their careers, not only by creating but also by using and promoting the use of systematic reviews.

While survey among biochemists about their use of The Cochrane Library has not been done, a study of physicians in Croatia was published recently and showed that only 10% of contacted physicians used The Cochrane Library and 9% had read the systematic reviews from The Cochrane Library (26). Usage of The Cochrane Library was much lower among family physicians, compared to physicians from clinical hospitals (26). As much as one third of the contacted physicians responded that they would be interested in learning a methodology for doing Cochrane reviews; among them were many of those who previously said that they did not hear about The Cochrane Library, which may indicate a social desirability bias (26).

Concurrent with this study, an initiative for adopting a ‘Cochrane PhD’ was started in 2009 with letters mailed to all Croatian medical schools, Faculty of Natural Sciences in Zagreb and Faculty of Pharmacy and Biochemistry in Zagreb, inviting them to approve a Cochrane systematic review as a potential doctoral thesis. Currently, the idea is formally accepted by two out of four Croatian medical schools - medical school in Split and Osijek (27). The same suggestion was sent to Medical School in Mostar, Bosnia and Herzegovina, as a liaison medical school, where it was also accepted.

At the School of Medicine in Split, a ‘Cochrane PhD’ was adopted with the following conditions: a candidate needs to be the first author, and the systematic review in question cannot be an ‘empty’ review (i.e. a review without any primary study included) or an updated review (i.e. a systematic review originally written by another group of authors). Under the scheme of a ‘Cochrane PhD’, a student may officially apply for a doctoral thesis when a protocol of the review is published in The Cochrane Library, and may defend the thesis after the full text of a systematic review is published in The Cochrane Library. Candidates need to translate in Croatian their Cochrane systematic reviews, which will be archived in Croatian libraries, as is the custom with all theses.

Other than having a Cochrane systematic review as PhD thesis, there is a possibility for Croatian doctoral studies to enhance their curricula by adopting educational activities organized by entities of The Cochrane Collaboration. The 2^nd Croatian Cochrane Symposium, organized by the CBICC in June 2010, was included in the regular doctoral program of “Evidence-Based Medicine” at the School of Medicine in Split. Besides participating in lectures and workshops of the Symposium, students were also encouraged to take part in the continuing education course “Basics of the Cochrane Systematic Reviews”, which was developed by the CBICC, and offered free of charge on the web site of Interactive Medical Education Center (InterMeCo). This online course has been approved by Croatian Medical Chamber as a continuing medical education (CME), and licensed physicians may get 8 CME points if they pass the test successfully. However, the online Cochrane course is not intended solely for physicians; CME points are simply a way of motivating physicians to take part in it. Any school in Croatia is welcome to adopt this course and give students academic credits for it.

Although a full integration of the CBICC’s educational programs, such as Croatian Cochrane Symposium or CME online course, into doctoral study curricula may be the easiest approach, the expertise offered by the CBICC can also be used in other ways. For example, the director of the CBICC was invited to teach the principles of systematic reviews on the doctoral course “Evidence-Based Medicine” at the School of Medicine in Zagreb. Training programs of other Cochrane entities, such as the Cochrane Canada’s webinar series (28), could also be conveniently incorporated in the existing doctoral courses.

Croatian Society of Medical Biochemists, with its keen interest in the activities of The Cochrane Collaboration, is very active in promotion of evidence-based medicine among biochemists in Croatia. If also formally recognized and accepted by the University of Zagreb Faculty of Pharmacy and Biochemistry, this “Cochrane PhD” initiative might provide an excellent opportunity for biochemists to create their research projects within the Cochrane Collaboration.

It is important for biochemists to know that seven years ago The Cochrane Collaboration has developed a database of systematic reviews of diagnostic test accuracy. There are three entities of The Cochrane Collaboration responsible for these systematic reviews: the Diagnostic Test Accuracy Working Group, the Regional Support Units and the Diagnostic Test Accuracy Editorial Team. Diagnostic test accuracy is the ability of a diagnostic test to show the difference between patients with and without the disease (29). In such a study, the results of the studied test are compared to the reference standard determined in the same patients. The methodology for making diagnostic test accuracy studies has progressed recently, but some challenges are still remaining, such as quality and availability of primary studies, and presentation of results in reviews (30). Since this is relatively recent and still developing field of studies, biochemists in Croatia may become actively involved in evolving reviews of diagnostic test accuracy and in such way contribute to medical practice worldwide.

 

 

Systematic reviews have a number of advantages over research assignments usually given to PhD students in postgraduate education. The Cochrane Collaboration provides training and infrastructure for production of systematic reviews, and therefore doctoral students should consider a “Cochrane PhD” when deciding about their doctoral thesis, while their supervisors and institutions should take administrative steps to accept Cochrane systematic review as a legitimate PhD thesis.

 

 

Dario Sambunjak is financially supported by the Croatian Ministry of Science, Education and Sports (grant No. 216-1080314-0245 to Matko Marušić).