Greater than Placebo: A Critical Review of Homeopathy Research

Naturopathic Perspective

Paul Theriault, BSc, ND, VNMI

Discussions of homeopathy within the naturopathic profession have emerged recently, which have been predicated on the assumption that there is no evidence for homeopathy. This assumption is false. The evidence base, when taken as a whole and evaluated in light of the way homeopathy is practiced, is positive. Homeopathic methodology does indeed contain many aspects that make it difficult to evaluate homeopathy in a clinical trial; however, these problems are slowly being overcome through improved clinical methodologies. Other lines of evidence supporting homeopathy include laboratory studies and historical records of homeopathy’s use, particularly in infectious diseases, but will not be the subject of this article. Rather, this discussion will focus on meta-analyses of homeopathy research, taken as a whole.

Homeopathy has been evaluated by means of clinical trials for hundreds of years. However, the nature of homeopathic prescribing, the deeply detailed nature of case-taking, the therapeutic nature of case-taking itself, and the multiple adjustments and refinements of potency needed during therapy – all of which make placebo control difficult – combine together to make it challenging to undertake clinical trials for genuine, pure homeopathy. Methodological refinements have indeed been made to the clinical trial procedure over the past several decades, but they have not been completely successful, often creating outcomes in which the controls of a clinical trial result in data that are inferior to what one would expect from clinical practice.

Earlier Reviews

Throughout the years, homeopathy has been examined by multiple clinical trials to determine its efficacy. Many of these trials have been negative. Until recently, the compilation of the data from homeopathy trials had not been done in a way that examined the validity of homeopathy as a whole. In recent decades, the volume of clinical trials – and the development of meta-analysis as a methodology – has enabled the clinical trials to be evaluated en masse. This has resulted in more positive outcomes.

The first meta-analysis of homeopathic clinical trials was performed in 1991 by Kleijnen et al and published in the British Medical Journal.1 It produced, among the small amount of clinical trials available at the time, a positive result.

The next meta-analysis was published in the Lancet in 1997 by Linde et al.2 Of 185 trials found, 119 met inclusion criteria, and 89 of these had adequate data for inclusion and analysis. Rather than differentiate methodologies, they pooled studies of all varieties of homeopathy, regardless of methodology. Linde et al delivered an odds ratio (OR) of 2.45 (with a 95% confidence interval of 2.05-2.93) – a strongly positive result. When corrective factors for publication bias were added, the OR decreased to 1.78 (1.03-3.1) – weaker but still positive. Analyzing only high-quality trials resulted in an OR of 1.66 (1.33-2.08).2 Linde was repeatedly criticized for lumping data of varying quality together, and for lumping together trials of individualized and non-individualized homeopathy; consequently, further studies were conducted.

Ernst also conducted several reviews of homeopathy during this period. His first, published in 1998,3 purported to reanalyze Linde’s 1997 meta-analysis, in light of criticisms that it evaluated many conditions, included non-individualized trials, and examined lower-potency preparations that could potentially contain molecules of the original substance. Ernst’s analysis of what he viewed as higher-quality studies produced an OR of 0. Hahn4 has criticized Ernst’s review as including only 5 studies, and thus being far less valid than Linde’s original 1997 analysis despite its flaws.

Questionable Methodologies

In 1998, Linde conducted a review of clinical trials examining individualized homeopathy.5 He encountered a number of methodological and quality issues; nevertheless, pooling the data yielded a positive result, with an OR of 1.62 (95% CI 1.17-2.23). However, when the analysis was restricted to higher-quality trials, no significant effect was seen, although a positive trend was observed, with an OR of 1.12 (0.77-1.44). As discussed above, the many issues in clinical trials of homeopathy at the time make this conclusion not terribly surprising, though this problem has been remedied in more current research. Linde explored this in a subsequent article in 1999,6 in which he examined the effect of trial quality on effect size in homeopathy studies. He found that higher trial quality often produced smaller effect sizes. This is important, as many critics seized upon this as proof of lack of efficacy of homeopathy, forgetting that other research7 has shown this effect to be consistent throughout medical research (ie, not a problem unique to homeopathy).

In 2000, Ernst published another meta-analysis,8 in which he criticized Linde’s 1998 analysis as flawed due to the weaknesses of the studies used. Ernst extrapolated trends based on Linde’s 1999 study of trial quality, and suggested a linear trend (when in fact the data is asymptotic); using this approach, trials with a perfect quality would theoretically move towards results of 0. This tactic is strongly criticized by Hahn4 as prioritizing imagined and extrapolated data over real data – particularly bizarre considering he had access to the real data.

The next meta-analysis of homeopathy was performed in 2000 by Cucherat et al and published in the European Journal of Pharmacology.9 They took 118 trials, judged 16 of them to meet inclusion criteria, and then combined p values to form a grand p value – an extremely unusual method of analysis and one which Hahn4 notes is likely to deliver the least favorable statistical result for homeopathy. Despite this odd method, the results of the analysis were positive, with a grand p value of <0.000036. Studies with less than 10% dropouts had a slightly lower p value of 0.0084 (ie, 8.4 out of a 1000 chance of being due to placebo), and studies with less than 5% dropout had a p value of 0.082 – marginally above the generally accepted threshold for statistical significance.9 However, this standard was extraordinarily rigorous, with dropout rates of less than 20% being commonly accepted as adequate.10

The next meta-analysis of homeopathy as a whole was by Shang et al, published in the Lancet in 2005.11 This study took an extraordinarily bizarre approach of choosing 110 randomized, controlled, double-blind trials, and then “matching” them to 110 comparable allopathic trials; some high-quality homeopathy studies were excluded solely due to an inability to find comparable allopathic trials. The data in these 110 trials were not combined to create an odds ratio. Instead, the data from all but 21 trials of homeopathy were excluded due to unspecified quality measures. The authors then excluded, for unclear reasons, all but 8 trials, and came up with an insignificant OR of 0.88 (0.65-1.19). Fascinatingly, the identities of these 8 studies were not listed in the original publication, but were released several months later, after the media circus around this result had died down.11

These results have been heavily criticized by homeopaths and statisticians, with the overall opinion that this trial should not have been published. These arguments are summarized by the Homeopathy Research Institute; however, perhaps the most significant arguments belong to Ludtke and Hahn. Ludtke12 analyzed Shang’s review, concluding that the results from Shang’s analysis were highly dependent on the subsection of trials selected, with his results being almost entirely due to a single, large, non-individualized trial of homeopathy. Hahn4 comments that the funnel plot tool used by Shang was completely inadequate to evaluate treatment effects in different medical conditions, and that in order to reach his conclusion that homeopathy lacks effect beyond placebo, 90% of clinical trials must be excluded. In fact, it has been determined by Ludtke12 that if Shang had included all 21 higher-quality trials, even using his bizarre selection criteria, he would have found a statistically significant effect that was greater than placebo.

More Recent Reviews

The next major meta-analysis of homeopathy was by Mathie et al in 2014.13 It included all available randomized controlled trials (RCTs) and rated all of them using several quality assessment measures. Unfortunately, again likely due to methodological issues, no trial was rated as having no risk of bias, in all domains. However, those trials that had low or unclear risk of bias in 1 domain were analyzed. Twenty-two trials had extractable data and were thus subject to analysis, creating a significant OR of 1.53 (1.22-1.91). Restricting the analysis to trials rated as reliable evidence produced an even higher OR of 1.93 (1.16-3.38). This is wonderfully explained by Mathie, himself, in a presentation available on YouTube.14

At the present time (May 2019, as of this writing), Mathie’s analysis still stands as the most current, complete, and unbiased of all reviews of homeopathy. The low quality of evidence does merit caution, but the results clearly indicate that the existing data do suggest that homeopathy has an effect greater than placebo. This is the current state of the evidence from clinical trials, and to deny this is to go against the existing data.

Ernst has criticized the results as excluding one of his own trials. Mathie graciously explained on Ernst’s blog his exclusion criteria, yet redid the analysis with the addition of Ernst’s data (despite not meeting the criteria), and arrived at the same results. This is published in a PDF separate from Mathie’s original article.15

The next, and probably lowest quality of analysis, is the 2015 NHMRC review,16 commissioned by the Australian government to evaluate the evidence of homeopathy. This review had a number of methodological problems, which have been summarized expertly by the Homeopathy Research Institute (HRI); much gratitude to HRI for presenting their points so succinctly. I will summarize them as follows17:

  • NHMRC conducted the review twice
  • The first review, and even its existence, was not disclosed to the public
  • The NHMRC, upon questioning, responded that the first review was low quality despite being conducted by the individual responsible for developing NHMRC’s guidelines on how to conduct evidence reviews
  • FOI (Freedom of Information) requests confirm that a member of NHMRC, Fred Mandelsohn, confirmed the first review to be high quality, stating, “I am impressed by the rigor, thoroughness and systematic approach given to this evaluation …. Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.”
  • NHMRC stated their results were based on over 1800 studies, when in fact they were based on only 176
  • NHMRC has used a method which has never been used in any other review, declaring that only trials of over 150 participants would be accepted, excluding the vast majority of high-quality homeopathic trials, which due to lack of funding tend to be smaller, and despite the fact that the NHMRC routinely conducts studies of fewer than 150 participants
  • The above rules resulted in exclusion of 171 of 176 studies, leaving only 5 to be used as the basis of the study
  • The chair of the second review, Peter Brooks, signed a “conflict of interest” form declaring himself “free from any association with any organization whose interests are either aligned or opposed to homeopathy” when he was a member of the anti-homeopathy lobby group “Friends of Science in Medicine”
  • The NHMRC review included no homeopaths or experts in homeopathy research, despite the NHMRC guidelines requiring such an inclusion

Rachel Roberts of the HRI presents this set of circumstances expertly in a presentation delivered at HRI.18

The evidence against the NHMRC in frank academic bias and misconduct is so strong that the HRI is pursuing a complaint with the relevant ombudsman in Australia. The initial investigation has found sufficient evidence to warrant a full investigation of NHMRC’s conduct, at the expense of the ombudsman, which is ongoing.19

Another meta-analysis of homeopathy was conducted by Mathie et al in 2017.20 This review focused on non-individualized, randomized, double-blinded control trials of homeopathy, and found 75. Forty-eight of these trials had a high risk of bias, 23 were uncertain, and 3 had a low risk of bias and were thus listed as reliable. The standard mean deviation (SMD) was the measure used. Fifty-four trials had extractable data, and the pooled data showed a SMD of -0.33 (95% CI -0.44, -0.21), which was statistically significant. Adjusting for publication bias, this was adjusted to a still significant -0.16 (95% CI -0.46, -0.09). Reliable data resulted in an insignificant result of -0.18 (95% CI -0.46, 0.09). This result has gone unremarked upon in the greater skeptical literature on homeopathy, despite both major negative reviews of homeopathy (Shang; NHMRC) failing to differentiate these 2 types of studies.

In 2018, Mathie et al performed another meta-analysis of RCTs of homeopathy, this time non-placebo-controlled. These studies examined homeopathy in comparison to another treatment, homeopathy alongside another intervention, or homeopathy compared to a no-treatment group.21 Eleven RCTs were found, 10 of which had a high risk of bias, and 1 of which had a high risk of bias solely due to participant blinding – something inherent to the study’s design. For 4 heterogeneous designs based on comparative treatments, the pooled odds ratio was non-significant. In 1 remaining trial, homeopathy was found to be non-inferior to fluoxetine for treatment of depression. For trials examining homeopathy alongside another intervention, there was a statistically significant mean SD favoring homeopathy. This review – due to a lower quality of evidence, the small number of trials, and trial heterogeneity – precludes any definitive conclusions.21

Mathie’s most recent meta-analysis, in 2019,22 examined non-randomized, other-than-placebo-controlled trials of homeopathy. Seventeen RCTs were found, with 10 containing data extractable for analysis. Fourteen trials showed a high risk of bias, and 3 had an unknown risk of bias. Heterogeneity prevented much comparison; however, 3 trials that were able to be compared had a non-significant pooled effect size.

Existing meta-analyses of homeopathy as a whole (and the NHMRC report, often lumped into this category) are summarized in Table 1.

Table 1. Meta-analyses of Homeopathy

Kleijnen, 19911

 All types of homeopathy (eg, single remedy vs combination). Methodological quality assessed; 105 trials. Results: Positive trend, regardless of type of homeopathy; 81 trials were positive, 24 showed no effect.

Linde, 19972

 All types of homeopathy. Out of 185 trials, 119 met inclusion criteria; 89 of these had extractable data. Results: OR = 2.45 (95% CI 2.05-2.93).

Ernst, 19983

 Individualized homeopathy; 5 trials determined to be high-quality. Results: OR = 0.

Linde, 19985

 Individualized homeopathy; 32 trials, 19 of which had extractable data. Results: OR = 1.62 for all trials (95% CI 1.17-2.23). Only high-quality trials produced no significant trend.

Cucherat, 20009

 All types of homeopathy; 118 trials, 16 of which met inclusion criteria. Used unusual method of combining p values. Results: All trials = p< 0.000036. Less than 10% dropouts: p<0.084; less than 5% dropouts (higher standards than most trials considered reliable): p<0.08 (non-significant).

Shang, 200511

 All types of homeopathy; only 8 trials selected from 21 high-quality trials of 110 selected with unusual criteria. Results: OR = 0.88 (0.65-1.19). Result strongly disputed by statisticians.

Mathie, 201413

 Individualized homeopathy; of the analysis pooled data from 22 higher-quality, individualized, double-blind RCTs. Results: OR = 1.53 (1.22-1.91) for all trials pooled; OR = 1.93 (1.16-3.38) for the 3 reliable trials.

NHMRC, 201516

 Out of 176 studies, 171 were excluded, leaving only 5 for the study. Investigators used unprecedented methods, did not combine data, and are currently under investigation for outcome shopping. Results: Negative results.

Mathie, 201720

 Non-individualized homeopathy; very few higher-quality trials. Results: For 54 trials with extractable data, SMD = -0.33 (-0.44, -0.21). When these were adjusted for publication bias, SMD = -0.16 (-0.46,-0.09). The 3 high-quality trials had non-significant results: SMD = -0.18 (-0.46, +0.09).

Mathie, 201821

 Individualized, other-than-placebo-controlled trials; 11 trials found, 8 with extractable data. Results: 4 heterogeneous comparative trials showed a non-significant difference. One trial in this group was positive. Three heterogeneous trials with additive homeopathy showed a statistically significant SMD. No definitive conclusion possible due to trial heterogeneity, poor quality, and low number of trials.

Mathie, 201922

 Non-individualized, other-than-placebo-controlled trials; 17 RCTs found, 14 with high risk of bias. Results: Significant heterogeneity prevented much comparison; 3 comparable trials showed a non-significant SMD.

Conclusions

In recent decades, homeopathy has been examined via a number of clinical trials, the number of which now allow meta-analysis. As we can see from the study findings, the type of homeopathy research (ie, individualized vs non-individualized, placebo-controlled vs non-placebo-controlled) can have a strong influence on the results, although trial quality also has a strong effect.

All meta-analyses performed in at least a somewhat open and rigorous manner have found statistically significant effects. This suggests that homeopathy has a greater-than-placebo effect, or at least a strong trend in that direction, when using data from the totality of homeopathy research, or from individualized, placebo-controlled trials. The meta-analyses with questionable methodology, one of which is undergoing government investigation for academic irregularities, have produced negative results, which have been demonstrated to be a direct result of their exclusion of vast swathes of the homeopathic clinical trial literature (based on arbitrary and unexplained criteria), as well as of their failure to differentiate – as Mathie has done – different types of homeopathic research.

The clinical data are flawed. Issues with methodology used in homeopathy RCTs, combined with a lack of research funding, have produced a lack of high-quality trials and data. However, the data we do have point towards homeopathy as having an effect greater than that of placebo.

There can be no argument with this conclusion, aside from possible new data emerging. Anyone who disputes this is going against the existing set of the highest-quality evidence on homeopathy.

References:

  1. Kleijnen J, Knipschild P, ter Riet G. Clinical trials of homeopathy. BMJ. 1991;302(6772):316-323.
  2. Linde K, Clausius N, Ramirez G, et al. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet. 1997;350(9081):834-843.
  3. Ernst E. Are highly dilute homeopathic remedies placebos? Perfusion. 1998;11(7):291-292.
  4. Hahn HG. Homeopathy: meta-analysis of pooled clinical data. Forsch Komplementmed. 2013;20(5):376-381.
  5. Linde K. Melchart D. Randomized controlled trials of homeopathy: a state of the art review. J Altern Complement Med. 1998;4(4):371-388.
  6. Linde K, Scholz M, Ramirez G, et al. Impact of study quality on outcome in placebo-controlled trials of homeopathy. J Clin Epidemiol. 1999;52(7):631-636.
  7. Hempel S, Miles J, Suttorp MJ, et al. Detection of Associations Between Trial Quality and Effect Sizes (Internet). Jan 2012. Agency for Healthcare Research and Quality (US). Report No: 12-EHC010-EF.
  8. Ernst E, Pittler MH. Re-analysis of previous meta-analysis of clinical trials of Homeopathy. J Clin Epidemiol. 2000;53(11):1188.
  9. Cucherat M, Haugh MC, Gooch M, Boissel JP. Evidence of clinical efficacy of homeopathy. A meta-analysis of clinical trials. HMRAG. Homeopathic Medicines Research Advisory Group. Eur J Clin Pharmacol. 2000;56(1):27-33.
  10. National Heart, Lung, and Blood Institute. Study Quality Assessment Tools. NIH Web site. https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools. Accessed May 12, 2019. Accessed May 12, 2019.
  11. Shang A, Huwiler-Müntener K, Nartey L, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet. 2005;366(9487):726-732.
  12. Lüdtke R, Rutten AL. The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. J Clin Epidemiol. 2008;61(12):1197-1204.
  13. Mathie RT, Lloyd SM, Legg LA, et al. Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev. 2014;3:142.
  14. Mathie RT. Systematic review and meta-analysis of randomized, placebo-controlled, trials of individualised homeopathic treatment. Homeopathy Research Institute. 2nd HRI International Homeopathy Research Conference; Rome, 2015. [YouTube presentation] Available at: https://youtu.be/3KMN7P6EaJk. Accessed May 12, 2019.
  15. Mathie RT, Lloyd SM, Legg LA, et al. Meta-analysis of randomised controlled trials (RCTs) of individualised homeopathy: sensitivity of results to using original authors’ ‘primary outcome measure’. Available at: https://www.britishhomeopathic.org/wp-content/uploads/2015/01/BHA-16-Jan-2015.pdf. Accessed May 12th 2019.
  16. National Health and Medical Research Council. Australian Government. NHMRC Statement: Statement on Homeopathy. March 2015. Downloadable PDF available at: https://www.nhmrc.gov.au/about-us/publications/homeopathy#block-views-block-file-attachments-content-block-1. Accessed May 12, 2019.
  17. Homeopathy Research Institute. The Australian report. Available at: https://www.hri-research.org/resources/homeopathy-the-debate/the-australian-report-on-homeopathy/. Accessed May 12th, 2019.
  18. Roberts R. The Australian report – an in-depth analysis of the highly influential 2015 overview report on homeopathy. June 9, 2017. 3rd HRI International Homeopathy Research Conference; Malta, 2017. [YouTube presentation] Available at: https://youtu.be/dWKHFsRJhWk. Accessed May 12, 2019.
  19. Roberts R. NHMRC Commonwealth Ombudsman Investigation. Homeopathy Research Institute. [YouTube presentation] Available at: https://www.youtube.com/watch?v=oTNwT53qaGc&t=2s. Accessed May 12, 2019.
  20. Mathie RT, Ramparsad N, Legg LA, et al. Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev. 2017;6(1):63.
  21. Mathie RT, Ulbrich-Zürni S, Viksveen P, et al. Systematic Review and Meta-Analysis of Randomised, Other-than-Placebo Controlled, Trials of Individualised Homeopathic Treatment. Homeopathy. 2018;107(4):229-243.
  22. Mathie RT, Fok YYY, Viksveen P, et al. Systematic Review and Meta-Analysis of Randomised, Other-than-Placebo Controlled, Trials of Non-Individualised Homeopathic Treatment. Homeopathy. 2019;108(2):88-101.

Paul Theriault, BSc, ND, VNMI, graduated from CCNM in 2010. He maintains a practice in Calgary that is dedicated to treating chronic infections, elimination of blockages to healing, digestive problems, and autism using homeopathy, constitutional hydrotherapy, drainage, and auricular acupuncture. Dr Theriault is the author of A New Era: Homeopathy and the Human Spirit (a book about triturated remedies in homeopathy) and The Table of Animals (a series of books systematizing the animal remedies in homeopathy). He is one of the first to complete an advanced training in Vitalistic Naturopathic Medicine from the Naturopathic Medicine Institute and receive the VNMI designation, and is currently completing his DHANP from the Homeopathic Academy of Naturopathic Physicians. Website: drpaultheriault.com.

Related Posts

Upcoming Events

View Calendar
Scroll to Top