Oxford Neurological Society
Skip to main content

Managing the Uncertainty: Introduction to Evidence Based Medicine PART 3

Is Hypnotherapy Effective and Why Charcot Was Wrong About Hysteria?

It was a remarkable symposium of neurological masterminds of the time: Jean-Martin Charcot, accompanied by Joseph Babinski, Pierre Marie, Georges Gilles de la Tourette, and other discoverers of famous neurological disorders were observing a truly bizarre spectacle.

Gentlemen,’ began the Napoléon of neuroses, ‘You may know that I initially believed hysteria to be a neurological disorder, which can be an inherited flaw of the nervous systems.

But I have now concluded that hysteria, or non-insane hysterio-epilepsy is indeed a psychological disease!’, the whispering of approval went throughout the audience of the concilium

Having prepared the instruments, he then attended to demonstrate his treatment – the marvellous hypnotherapy. A middle-aged woman was put in a trance, and assumed the arc en circle or ‘the hysteric’s classic posture’, the same as depicted on the charcoal drawing at the back of the demonstration room.

Babinski held the woman still, as the great Charcot was practising his treatment. Resting on the table were a reflex hammer and what is thought to be a Duchenne electrotherapy apparatus.

 


 

Although this  demonstration is thought to have taken place in 1887, hypnotherapy supporters still think it is a valid and efficient treatment of many neurological and psychiatric disorders. It is widely used by alternative medicine practitioners, despite the scepticism being as lively as it was in the 19th century.

Is hypnotherapy the right treatment? Does it work? Should it be used to treat patients or indeed offered to patients on the NHS?

Today, you’ll learn how to assess that and what evidence can be used to support either side of the argument.

 

banner0

 



How to assess the accuracy of the study?

 

The accuracy of the study will largely depend on its design, and sometimes on its execution. The process of designing a study is a very cumbersome and ungrateful procedure, but it is so for a reason: the data collection, generalisability, statistical analysis and ethical approval will all depend on a robust and valid study design.

 

PROBLEM 1 – HOW TO DEFINE ACCURACY?

 

One of the most important parts of a study design is to define the population of the study, the sample size and its characteristics.

You may need to consider a number of factors: age, occupation, educational background, employment, geographical location, past medical history, race, marital status, etc. can all be important in your study.

Which factors you want to include will depend on what you want to measure.

For example, if you want to study smoking patterns of high school pupils, interviewing females over 65  would not make sense.

In this example, you may want to limit the inclusion criteria, i.e. the requirements someone must meet to be included in the study, to certain age and education background, with exclusion criteria, i.e. the requirements that will trigger an exclusion from the study, such as “dropped out of school” “no longer in full-time education” or “not smoking”.

You also want to measure these demographic variables to pick up confounding or irregular patterns that may distort your results. Please, don’t worry about it now – I’ll write more about confounding later.

Some variables may be quite obvious (e.g. age, education), others may require deeper analysis (e.g. specific medical conditions, score on tests, medical imaging results).

There is also an important factor of the practical feasibility of the study. The more participants you want to include, the larger the samples size will become. The logistics, organisation and funding of a very large study may be difficult to control: assessing 16,000 high school pupils all over the country is much more difficult than conducting a study on 500 students in a local school area.

Click here to see the first option

 

PROBLEM 2 – HOW TO DEFINE PRECISION

 

Precision, on the other hand, is much easier to define. It is also less descriptive: you can generally assess the precision of a given value by reading its confidence interval.

Narrow confidence intervals suggest a very precise measurement. Broader CIs can be indicative of a low precision.

 

Consider this example:

RESULTS SECTION

The mean age of the take-up of smoking is 15 95% CI(14-16).

 

This means that the average age, based on the data collected is 15. But how certain are you that this mean is “true”? Confidence intervals come to aid: the CI95% means that you are 95% confident that the true mean age of the take-up of smoking in this population lies between 14 and 16.

 

NB: It is very important to remember that the 95% confidence applies to the ENTIRE INTERVAL. It means that the true mean could be 15, but it also could be 14 and 1 month or 15 and 11 months or even 15 and 2 days.

Any value in that interval is EQUALLY likely to be the “true” mean. Furthermore, you are 95% confident, so there is still a chance of 5 in a 100 (or 1 in 20) that your true mean will lie OUTSIDE of that interval.

 

This means that, statistically, one in every 20 measurements will be unreliable, because it will lie outside of the interval, and it can really be any number at all: in this case, 1 year 2 months or even 99years and 1day.

 

There is always a degree of uncertainty.

 

You can never be 100% sure about something in science, statistics or medicine. You may, however, be very sure that on the balance of probabilities, you are indeed right.

This is why studies with multiple measurements should be treated with caution. If you report 40 different measurements, statistically 2 of them will be “false”. And that’s just confidence intervals and means.

Click here to see the first option

 

PROBLEM 3: Accuracy versus Precision

In summary: accuracy is mainly inferred from the study design, which you can find in the methods section of an article, and gives an indication of how generalizable the study may be. Large and comprehensive studies are more accurate than small samples.

Precision is assessed by analysing the confidence intervals in the results section. Narrow CIs indicate more precise results than very wide intervals.

You may say that conducting large studies is always better because it guarantees a good accuracy.

However, we must consider many problems large studies have: you may not get enough funding, the discrepancies between reporting methods in different centres can be huge, it’s difficult to manage large datasets and protect the anonymity of participants, you can lose many people in follow-up etc. etc.

All this can distort your results and get you large variability between them. And this, in turn, will most likely broaden your CIs and lower your precision.

 

You can already see that in real life, some very precise results can be inaccurate, and some very accurate results may be imprecise.

 

Therefore, once you do a power and sample size of study calculation (a statistical method of estimating how many participants you need), you should aim to recruit this many participants, as inflating the number of results can have an adverse effect on your precision (and your mental wellbeing as a principal investigator trying to manage the organisational mayhem of a large study)

Equally, having a very small sample may mean the result will be very accurate, but cannot be generalizable to other populations e.g. an inner city smoking cessation efficiency results may not be applicable to the same smoking cessation programme in rural Wales. What works perfectly on children may not have the same effect in adults, etc.

Before answering the question of the efficacy of hypnotherapy, let’s assess a couple of studies in terms of precision and accuracy.

 

2016-08-20_14-30-32

Source: ELKINS ET AL. 2010

This is a table from METHODS section on how the demographics of subjects included in this Intensive Hypnotherapy for Smoking Cessation study. Please pay attention to the characteristics of the study and control groups.

Click here to see the first option

Consider this output from a mathematical application (Opex Resources)

Don’t worry about the multitude of results, just focus on the ones you are familiar with.

111

Click here to see the first option

 

PROBLEM 4: Confidence intervals

 

Confidence intervals can often invalidate the conclusions about the study.

Although the result itself (e.g. a mean) can support your conclusion, the mere 95% CI can make the result meaningless.

Consider the example from previous hypothetical problem. You want to measure the smoking patterns in pupils and your average take-up age is 15. But is this enough to formulate a conclusion that “Pupils take up smoking at school”?

What if the 95% CI were [3-55]? With the same mean, you can be 95% confident that the true mean lies between the age of 3 and the age of 55. The age of 3 is before going to school and the age of 55 is long after leaving it.

Therefore, you cannot say that “Pupils take up smoking at school” as you are 95% confident that they take it up before going to school or even long after.

In this case, the very broad 95% CI effectively invalidated your conclusion. This doesn’t necessarily mean your whole study is rubbish. Your results can still be useful for other purposes.

Also, remember that in 5% of the cases, the age of take-up could have been even 1 or 99 (outside the 95% CI range)

So, dear Brutus, as much as you’d like to fault the stars of our means, it is the intervals wherein the truth can hide.

 

Can other measurements be invalidated by 95% CI?

Yes, many measurements have something called the NULL value. It means that, if this measurement is of this value, the effect simply isn’t there.

Consider this example:

euro

The Fee et al (1977) study compared hypnotherapy treatment vs. control group of placebo (dummy pill) alone in smoking cessation. The figure above is a graphical representation of the results, sometimes called Foster diagram (or blobbogram). It may be quite difficult at the first glance, but it is, in fact, very easy to read.

Look at the diamond at the bottom, and its corresponding risk ratio (RR) confidence intervals.

 

Now look at the x axis of the graph:

RR of more than 1 Favours treatment (Hypnotherapy is better than placebo)

RR of less than 1 Favours control (Placebo group is better than hypnotherapy)

RR of 1 indicates that there is no difference between the groups.

 

Let’s start from the value itself: the RR is 0.83 (less than 1) so you may think that the PLACEBO group is better than HYPNOTHERAPY (i.e. hypnotherapy doesn’t work)

However, the 95% CI indicate that you are 95% confident that the true RR lies between 0.27 (favouring placebo) and 2.58 (favouring hypnotherapy).

 

What’s more, the null value, i.e. RR of 1, is INSIDE the 95% CI.

 

This means that you are equally confident that the hypnotherapy is better than placebo, that placebo is better than hypnotherapy and that there is no difference at all.

What does it say about the study? Basically, you cannot infer any firm conclusion about which therapy is better.

So far, I’ve been guiding you through the Platonian cave of Forest diagrams. Now, equipped with your παιδεία about the graphs, can you tell the nature from the shadow?

This analysis compares hypnotherapy to drugs for smoking cessation. RR is 1.02 [0.31 – 3.33]

 

122

 

Click here to see the first option

 

Yes, the 95% CI includes all three possibilities, so you cannot say which one is best. Please note that the actual value of RR is irrelevant in deciding that. We are only considering the 95% CI. It doesn’t matter that the RR is “close” or “far” from 1.

Another example shows a comparison between hypnotherapy with psychological treatment AND brief relaxation/advice with psychological treatment

 

2016-08-20_15-24-28

 

Click here to see the first option

 

In this example, although the 95% CI is very close to the null value of 1, and it ranges from a small benefit (1.07) to large benefit (16.70), all of these are beneficial, so we can conclude that the hypnotherapy with psychological treatment is more effective.

I guess that’s +1 for hypnotherapy after all.

Congratulations! You’ve interpreted your first blobbograms: the power of the Holy Grail of the evidence based medicine is now at your disposal!

 

banner5

 

That’s it for today. I hope you’ve enjoyed this lesson. Please do read other analyses from the same review (link), if you want to practice deciding in favour or against the treatment.

We’ll talk about the Risk Ratios and Odds Ratios in more detail later on.

 


 

banner3

 

ACCURACY:

The measurement of how close, on average, is the sample statistic (e.g. collection of means, medians, etc.) to the population parameter (the actual “true outcome) that it estimates. If the measurement is very close to the “true” parameter, the accuracy will be high

 

PRECISION:

The amount of variation in the sample statistic (e.g. collection of means, medians, etc.). If the samples are very varied, the sample statistic will be imprecise.

 

CONFIDENCE INTERVAL (CI):

The interval, where the “true” value of a parameter is between two values, with a given degree of probability (e.g. 95%). Thus, the 95% CI (0-2) means that you are 95% confident that the true value lies somewhere between 0 and 2.

 

GENERALISABILITY

The ability to apply the results or conclusions of the study to a larger population than defined and studied in the paper.

About the author


Max Brzezicki

Max Brzezicki

Passionate about evidence-based medicine and science, likes slicing meat, crushing rat brains, criminal & public law, foreign languages, rhetoric, history, classical studies and political thought. FNS since 2015.

Popular posts this week


11 thoughts on “Managing the Uncertainty: Introduction to Evidence Based Medicine PART 3

  1. Really interesting, Im just reading these and cant wait for another one, its just like reading a story but its arcually learning which is soo cool

  2. I’ve seen a diagram with accuracy and precision, but it wasnt like you explain. It was like you had a lot of points near a “true” variable that were very precise but not accurate. Given that you should think that accuracy is about getting the “true” result not getting the spectrum of people or narrowing the study scope. How do you explain this?

    1. Hi Deb, yeah, I know this diagram can be confusing indeed! Forget about the diagram for a moment. Just focus on two things: ACCURACY is how generalizable the study can be, so the more people are included the more accurate the results are. Thus, you will find information about accuracy mainly in the METHODS section.

      PRECISION, on the other hand, is a measure mainly inferred from your 95% CIs. If they are narrow, you can be very confident that the result is precise.

      I hope this explains the difference. I was easier for me to learn from these definitions when I was studying the EBM for the first time, hope it help you, too!

  3. It’s all very interesting and all but seriously hypnotherapy cannot be really working. It’s just all made up just to fit the story.

  4. Oh yeah here we go again, Max. Is anybody interested in this? Literally read none of these, or heard of anyone who read that. Top work.

Leave a Reply

Your email address will not be published.