|Ahead of print publication
The critical appraisal of randomized controlled trials published in an Indian journal to assess the quality of reporting: A retrospective, cross-sectional study
Sandeep Kumar Gupta, Ravi Kant Tiwari, Raj Kumar Goel
Department of Pharmacology, Heritage Institute of Medical Sciences, Varanasi, Uttar Pradesh, India
|Date of Submission||26-Oct-2019|
|Date of Decision||05-Feb-2020|
|Date of Acceptance||11-Feb-2020|
|Date of Web Publication||08-Jan-2021|
Sandeep Kumar Gupta,
Heritage Institute of Medical Sciences, Varanasi - 221 311, Uttar Pradesh
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Background: Although randomized controlled trials (RCTs) are the highest levels of evidence, they might not necessarily be of good quality. Hence, RCTs should always be appraised critically. Critical appraisal is the corroboration of evidence by methodically studying its validity, reliability, and applicability.
Objective: The primary objective of this study was to do a critical appraisal of the RCTs published in Indian Journal of Pharmacology (IJP) from 2011 to 2016. The secondary objective was to scrutinize how adequately the published RCTs adhere to the Consolidated Standards of Reporting Trials (CONSORT) declaration.
Materials and Methods: The present study included all RCTs published as full-text articles in IJP from January 2011 to December 2016. The identified RCTs were critically appraised using the critical appraisal checklist based on CONSORT 2010 guidelines and its extensions.
Results: According to this analysis, 75% (95% confidence interval [CI]: 0.56–0.87) of the articles had given details about the sample size calculation. Nearly 89.29% (95% CI: 0.72–0.96) of the articles described the method for generating random allocation sequence, but only 35.71% (95% CI: 0.20–0.54) of the articles described allocation concealment method. Almost 35.71% (95% CI: 0.20–0.54) of the trials reported results as per the principle of the intention to treat (ITT). Nearly 21.43% (95% CI: 0.10–0.39) of the studies reported CIs in the present study.
Conclusion: Allocation concealment method, analysis of the data based on the ITT principle, and reporting CIs were found to be underreported in this study. There should be more emphasis on reporting of allocation concealment, ITT analysis, and CI.
Keywords: Confidence intervals, critical appraisal, intention-to-treat principle, quality of reporting, randomized controlled trials, the Consolidated Standards of Reporting Trials statement
|How to cite this URL:|
Gupta SK, Tiwari RK, Goel RK. The critical appraisal of randomized controlled trials published in an Indian journal to assess the quality of reporting: A retrospective, cross-sectional study. Perspect Clin Res [Epub ahead of print] [cited 2021 Jan 17]. Available from: https://www.picronline.org/preprintarticle.asp?id=306576
| Introduction|| |
One of the most important skills a physician needs in the era of evidence-based medicine is the skill to scrutinize scientific publication critically. Critical appraisal is a systematic way of reading, comprehending, elucidating, and pinpointing the limitations of and determining the adequacy of the results of scientific publications. Critical appraisal weighs up how valid, reliable, and valuable the research will be. It requires thoroughly scrutinizing research evidence to evaluate its validity, results, relevance, impact, and applicability before coming to any conclusion. Critical appraisal encourages sound decision-making based on the best available evidence. It helps us determine how accurate a piece of research is (validity), how genuine the result is (reliability), and how pertinent it is to our patient (applicability).
In general, randomized controlled trials (RCTs) provide authentic results that could apprise future research or clinical practice., However, trials carried out with insufficient methodological approaches are associated with inflated treatment effects. There is ample evidence in the public domain to prove that the standard of reporting of published RCTs is not optimum., The Consolidated Standards of Reporting Trials (CONSORT) statement helps in the comprehensive and unambiguous reporting of trials and facilitates their critical appraisal and analysis. Clinicians must rely on the scientific publications to keep themselves up-to-date on recent developments on new therapies as well as for new information on old therapies. RCTs might not be necessarily of good quality, hence they should always be appraised critically. Moreover, very less information is available about the reporting quality of RCTs published in Indian journals.
The primary objective of this study was to do a critical appraisal of the RCTs published in Indian Journal of Pharmacology (IJP) from 2011 to 2016. The secondary objective was to scrutinize how adequately the published RCTs adhere to the CONSORT statement.
| Materials and Methods|| |
This was a cross-sectional and retrospective study. It was based on the critical appraisal and assessment of reporting quality of published RCTs in IJP as per the CONSORT statement. This research relied exclusively on information freely available in the public realm, hence ethical sanction was not required.
The present study included all RCTs published as full-text articles in IJP from January 2011 to December 2016.
Studies were included if they were described within the paper as a RCT or claimed to use random assignment for participants. Studies were only eligible if they were controlled trials with two comparators.
Animal experiments, systematic review/meta-analyses, pharmacoeconomic studies, drug safety studies, pharmacokinetic/pharmacodynamic studies, drug utilization studies, and cross-sectional studies were excluded. Cohort studies and case series were also excluded. RCTs published as short communication and letters to the editor were not included in the present analysis because of brief information and word limitation.
The eligible articles were identified by screening of titles, abstracts, and methodology.
Assessment of reporting quality
The identified RCTs were critically appraised using the CONSORT statement. A critical appraisal checklist was prepared based on the CONSORT 2010 guiding principle and its extensions [Table 1]. The critical appraisal of a RCTs published in IJP with reference to its methodology was done to assess its validity, reliability, and applicability.
Descriptive statistics was utilized in this study. The lower and upper bounds of the 95% confidence interval (CI) for the proportions were calculated. The data were analyzed by means of the SPSS (Statistical Package for the Social Sciences), version 16; IBM Corporation, Chicago, Illinois, USA.
| Results|| |
A total of 1102 articles published from January 2011 to December 2016 in the IJP were screened for eligibility. The process used to select potentially relevant studies for inclusion in our study is depicted in [Figure 1]. From the total of 1102 articles, 596 were omitted after screening of titles and abstracts. A total of 506 studies underwent further evaluation. Of these, 478were excluded after full-text review.
|Figure 1: Flow diagram of citations through the retrieval and the screening process|
Click here to view
Of the 28 included articles, 21 (75%, 95% CI: 0.56–0.87) mentioned “randomization” in the title. In 28 (100%, 95% CI: 0.87–1) articles, the abstract was structured; these articles described the scientific rationale, gave details of specific objectives or hypotheses, gave a description of trial design, and gave details of eligibility criteria for the participants. Twenty-five (89.29%, 95% CI: 0.72–0.96) articles gave details about setting and locations. Twenty-seven (96.43%, 95% CI: 0.82–0.99) articles defined outcome measures.
Twenty-one (75%, 95% CI: 0.56–0.87) articles mentioned how sample size was determined. Twenty-five (89.29%, 95% CI: 0.72–0.96) articles mentioned about the method for random allocation sequence generation. Ten (35.71%, 95% CI: 0.20–0.54) articles mentioned about allocation concealment method. Sixteen (57.14%, 95% CI: 0.39–0.73) articles were blinded studies. Twenty-eight (100%, 95% CI: 0.87–1) studies described statistical methods for outcome assessment.
Ten (35.71%, 95% CI: 0.20–0.54) articles gave participant flow diagram. Twenty-six (92.86%, 95% CI: 0.77–0.98) articles gave details for each group, including the number of randomly assigned participants. Sixteen (57.14%, 95% CI: 0.39–0.73) studies mentioned the dates of recruitment and follow-up. Twenty (71.43%, 95% CI: 0.52–0.84) studies gave a table displaying the baseline characteristics of each group. “Intention-to-treat (ITT)” analysis was done in ten (35.71%, 95% CI: 0.20–0.54) studies. The appraised effect size was mentioned in 28 (100%, 95% CI: 0.87–1) articles, but the precision of effect size (CI) was mentioned in only six (21.43%, 95% CI: 0.10–0.39) studies. Adverse effects in each group were mentioned in 28 (100%, 95% CI: 0.87–1) studies.
Trial limitation was addressed in 19 (67.86%, 95% CI: 0.49–0.82) studies. Generalizability could not be evaluated because it was challenging to objectively evaluate them. Registration number and name of trial registry were mentioned in nine (32.14%, 95% CI: 0.17–0.50) studies. Sources of funding were mentioned in eight (28.57%, 95% CI: 0.15–0.47) studies [Table 1].
| Discussion|| |
The three keys of critical appraisal are validity, reliability, and applicability. When assessing validity, both internal and external validity should be assessed. When critiquing a study design and its subsequent findings, evaluation of the internal validity of the research paper is required. The aptness of the study results to support an association between the intervention and the outcome is internal validity. When assessing internal validity, emphasis is given to reporting of items such as randomization method, concealment of allocation, sample size calculation, baseline data reporting, estimated effect size precision, and ITT analysis reporting.
In this study, it was found that 75% of the articles provide details about the sample size calculation. In a survey by Zhang et al., 2016, no trials reported about adequate sample size calculation. In the same study by Zhang et al., 2016, with another set of journals, only 12.5% of the trials described the sample size calculation. In a study by Shi et al., 2014, only 0.4% of the trials reported adequate sample size. Nearly 75%of the articles were identified as randomized in title in this study. In a study by Choi et al., 2014, only 10.7% of the articles mentioned “randomization” in their title. In a study by Shi et al., 2014, only 0.3% of the trials reported “randomized control trial” in the title. Almost 89.29% of the articles described method for generating random allocation sequence, but only 35.71% of the articles described allocation concealment methods in this study. In a study by Pratoomsoot et al., 2015, 47.89% of the RCTs reported the methods for generating the random sequence allocation, and 29.58% of the study implemented a random allocation sequence. In a study by Zhai et al., 2015, the allocation sequence generation was described in 35.4% of the trials and concealment of allocation was described in 28.5% of the trials. In a study by Zhang et al., 2016, 33.4% of the trials reported about generation of allocation sequence, but only 3.7% of the trials reported about concealment of allocation. In a review of Dias et al., 2006, 49% of the trials were found not to have provided details of the randomization method and of all the trials assessed, only 10% had described adequate forms of allocation concealment. In a survey by Chen et al., 2014, randomization and allocation concealment were reported in 67% and 49% of the RCTs, respectively.
Only 35.71% of the trials have reported results as per the principles of ITT concept in this study. Aberrations from this principle can lead to biased results. In a study by Zhai et al., 2015, 23.8% of the trials used ITT analysis. In a survey by Chen et al., 2014, 44% of the trials reported ITT analysis. In the study by Partsinevelou and Zintzaras, 2009, 87.5% of the trials did not give any data about “ITT” analysis. In this study, 57.14% of the studies reported adequate blinding and 71.43% of the studies reported baseline data. In a study by Pratoomsoot et al., 2015, approximately 31% of the RCTs clearly reported about blinding. In a study by Zhai et al., 2015, blinding was reported in 57.0% of the trials. Blinding and baseline characteristics were reported in 51% and 97% of the RCTs, respectively, in a study by Chen et al., 2014. External validity means generalizability (representative of the study sample). As there is no validated scale to measure external validity, it was challenging to objectively evaluate them.
Reliability of the result refers that if the study were conducted again, the result would be the same. It is usually interpreted as the accuracy of measurement. It discourses the treatment effect magnitude and the estimation of the treatment effect precision. The size of the treatment effect is often expressed as the average difference between groups on some objective outcome measures. CIs are the most common measures of precision. There are certain weaknesses of P values: not truly compatible with hypothesis testing, never meant to be the sole indicator of significance, and no consideration of effect size. There are many factors which influence P values such as sample size, chance, effect size, and statistical power. Comparatively more information on the precision of the inference is provided by the 95% CI. Though all the studies reported P values in the study, only 21.43% of the studies reported CIs. Similarly, in an analysis of orthopedic research by Vavken et al., 2009, only 22% of the studies reported CIs.
Limitation of this research is that RCTs from only one Indian journal (IJP) were included. Another limitation was small sample size. RCTs' quality was not evaluated directly as the data were not corroborated from the corresponding authors. This study focused on the methodological specifics, which may differ from the quality of the actual study. Assessing clinical trial protocols along with contacting the investigators for more data may ameliorate quality appraisal.
| Conclusion|| |
This study demonstrated that underreported items from the CONSORT statement included allocation concealment method and analysis of the data based on the “ITT” principle. Apart from that, reporting of CIs was found to be less. There should be stress on allocation concealment, ITT analysis, and reporting of the CI to augment the validity of the trials. The results of this study would aid in improved observance to the CONSORT statement. Meticulous reporting of trials will help clinicians to endorse a new therapy or modify therapies currently in practice.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Das K, Malick S, Khan KS. Tips for teaching evidence-based medicine in a clinical setting: Lessons from adult learning theory. Part one. J R Soc Med 2008;101:493-500.
Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: Can we do better? Ann Surg 2006;244:663-7.
Choi J, Jun JH, Kang BK, Kim KH, Lee MS. Endorsement for improving the quality of reports on randomized controlled trials of traditional medicine journals in Korea: A systematic review. Trials 2014;15:429.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al
. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012;11:MR000030.
Schulz KF, Altman DG, Moher D; for the CONSORT group. CONSORT 2010 statement++ updated guidelines for reporting parallel group randomized trials. J Pharmacol Pharmacother 2010;1:100-7.
] [Full text]
Partsinevelou A, Zintzaras E. Quality of reporting of randomized controlled trials in polycystic ovary syndrome. Trials 2009;10:106.
Zhang J, Chen X, Zhu Q, Cui J, Cao L, Su J. Methodological reporting quality of randomized controlled trials: A survey of seven core journals of orthopaedics from Mainland China over 5 years following the CONSORT statement. Orthop Traumatol Surg Res 2016;102:933-8.
Shi C, Tian J, Ren D, Wei H, Zhang L, Wang Q, et al
. Methodological reporting of randomized trials in five leading Chinese nursing journals. PLoS One 2014;9:e113002.
Pratoomsoot C, Sruamsiri R, Dilokthornsakul P, Chaiyakunapruk N. Quality of reporting of randomised controlled trials of herbal interventions in ASEAN Plus Six Countries: A systematic review. PLoS One 2015;10:e108681.
Zhai X, Wang Y, Mu Q, Chen X, Huang Q, Wang Q, et al
. Methodological reporting quality of randomized controlled trials in 3 leading diabetes journals from 2011 to 2013 following CONSORT statement: A system review. Medicine (Baltimore) 2015;94:e1083.
Dias S, McNamee R, Vail A. Evidence of improving quality of reporting of randomized controlled trials in subfertility. Hum Reprod 2006;21:2617-27.
Chen B, Liu J, Zhang C, Li M. A retrospective survey of quality of reporting on randomized controlled trials of metformin for polycystic ovary syndrome. Trials 2014;15:128.
Karpouzis F, Bonello R, Pribicevic M, Kalamir A, Brown BT. Quality of reporting of randomised controlled trials in chiropractic using the CONSORT checklist. Chiropr Man Therap 2016;24:19.
Gupta SK. The relevance of confidence interval and P value in inferential statistics. Indian J Pharmacol 2012;44:143-4.
] [Full text]
Vavken P, Heinrich KM, Koppelhuber C, Rois S, Dorotka R. The use of confidence intervals in reporting orthopaedic research findings. Clin Orthop Relat Res 2009;467:3334-9.