A new study of e-cigarettes’ efficacy in smoking cessation has not only pitted some of vaping’s most outspoken scientific supporters against one of its fiercest academic critics, but also illustrates many of the pitfalls facing researchers on the topic and those – including policy-makers – who must interpret their work.
The furore has erupted over a paper published in The Lancet Respiratory Medicine and co-authored by Stanton Glantz, director of the Center for Tobacco Control Research and Education at the University of California, San Francisco, along with a former colleague – Sara Kalkhoran, now of Harvard Medical School, who is in fact named as first author but does not enjoy Glantz’s fame (or notoriety) in tobacco control and vaping circles.
Their research sought to compare the success rates in quitting combustible cigarettes of smokers who vape and smokers who don’t: in other words, to find out whether use of e-cigs is correlated with success in quitting, which might well imply that vaping helps you give up smoking. To do this they performed a meta-analysis of 20 previously published papers. That is, they didn’t conduct any new research directly on actual smokers or vapers, but instead tried to blend the results of existing studies to see if they converge on a likely answer. This is a common and well-accepted approach to extracting truth from statistics in many fields, although – as we’ll see – it’s one fraught with challenges.
Their headline finding, promoted by Glantz himself online as well as by the university, is that vapers are 28% less likely to stop smoking than non-vapers – a conclusion which would suggest that vaping is not just ineffective in smoking cessation, but actually counterproductive.
The result has, predictably, been uproar from the supporters of e-cigarettes in the scientific and public health community, especially in Britain. Among the gravest charges are those levelled by Peter Hajek, the psychologist who directs the Tobacco Dependence Research Unit at Queen Mary University of London, calling the Kalkhoran/Glantz paper “grossly misleading”, and by Carl V. Phillips, scientific director of the pro-vaping Consumer Advocates for Smoke-Free Alternatives Association (CASAA) in the U.S., who wrote “it is clear that Glantz was misinterpreting the data willfully, rather than accidentally”.
Robert West, another British psychologist and the director of tobacco studies at a centre run by University College London, said “publication of this study represents a major failure of the peer review system in this journal”. Linda Bauld, professor of health policy at the University of Stirling, suggested the “conclusions are tentative and sometimes incorrect”. Ann McNeill, professor of tobacco addiction in the National Addiction Centre at King’s College London, said “this review is not scientific” and added that “the information included about two studies that I co-authored is either inaccurate or misleading”.
But what, precisely, are the problems these eminent critics find in the Kalkhoran/Glantz paper? To answer some of that question, it’s necessary to go beneath the sensational 28%, and examine what was studied and how.
The dangers of meta-analysis
Meta-analysis is a seductive idea. If (say) you have 100 separate studies, each of 1000 individuals, why not combine them to create – in effect – a single study of 100,000 people, the results from which ought to be much less susceptible to any distortions that might have crept into an individual investigation?
(This could happen, for example, by inadvertently selecting participants with a greater or lesser propensity to give up smoking due to some factor not considered by the researchers – a case of “selection bias”.)
Of course, the statistical side of a meta-analysis is rather more sophisticated than just averaging out the totals, but that’s the general concept. And even from that simplistic outline, it’s immediately apparent where problems can arise.
If its results are to be meaningful, the meta-analysis has to somehow take account of variations in the design of the individual studies (they may define “smoking cessation” differently, for example). If it ignores those variations, and tries to shoehorn all results into a model that some of them don’t fit, it’s introducing its own distortions.
Moreover, if the studies it’s based on are inherently flawed in any way, the meta-analysis – however painstakingly conducted – will inherit those same flaws.
This is a charge made by the Truth Initiative, a U.S. anti-smoking nonprofit which normally takes an unwelcoming view of e-cigarettes, about a previous Glantz meta-analysis which comes to similar conclusions to the Kalkhoran/Glantz study.
In a submission last year to the U.S. Food and Drug Administration (FDA), responding to that federal agency’s call for comments on its proposed e-cigarette regulation, the Truth Initiative noted that it had reviewed many studies of e-cigs’ role in cessation and concluded that they were “marred by poor measurement of exposures and unmeasured confounders”. Yet, it said, “many of them have been included in a meta-analysis [Glantz’s] that claims to show that smokers who use e-cigarettes are less likely to quit smoking compared to those who do not. This meta- analysis simply lumps together the errors of inference from these correlations.”
It also added that “quantitatively synthesizing heterogeneous studies is scientifically inappropriate and the findings of such meta-analyses are therefore invalid”. Put bluntly, don’t mix apples with oranges and expect to get an apple pie.
Such doubts about meta-analyses are far from rare. Steven L. Bernstein, professor of health policy at Yale, echoed the Truth Initiative’s points when he wrote in The Lancet Respiratory Medicine – the same journal that published this year’s Kalkhoran/Glantz work – that the studies included in their meta-analysis were “mostly observational, often with no control group, with tobacco use status assessed in widely disparate ways” though he added that “this is no fault of [Kalkhoran and Glantz]; abundant, published, methodologically rigorous studies simply do not exist yet”.
So a meta-analysis can only be as good as the research it aggregates, and drawing conclusions from it is only valid if the studies it’s based on are constructed in similar ways to one another – or, at least, if any differences are carefully compensated for. Of course, such drawbacks also apply to meta-analyses that are favourable to e-cigarettes, such as the famous Cochrane Review from late 2014.
Other criticisms of the Kalkhoran/Glantz work go beyond the drawbacks of meta-analyses in general, and focus on the specific questions posed by the San Francisco researchers and the ways they tried to answer them.
One frequently-expressed concern has been that Kalkhoran and Glantz were studying the wrong people, skewing their analysis by not accurately reflecting the real number of e-cig-assisted quitters.
As CASAA’s Phillips points out, the e-cigarette users in the two scholars’ number-crunching were all current smokers who had already tried e-cigarettes when the studies on their quit attempts started. Thus, the study by its nature excluded those who had started vaping and quickly abandoned smoking; if such people exist in large numbers, counting them would have made e-cigarettes seem a much more successful route to smoking cessation.
A different question was raised by Yale’s Bernstein, who observed that not all vapers who smoke are trying to give up combustibles. Naturally, those who aren’t attempting to quit won’t quit, and Bernstein observed that when these people were excluded from the data, it suggested “no effect of e-cigarettes, not that e-cigarette users were less likely to quit”.
Excluding some who did manage to quit – and then including those who have no intention of quitting anyway – would certainly seem to affect the outcome of a study purporting to measure successful quit attempts, even though Kalkhoran and Glantz argue that their “conclusion was insensitive to a wide range of study design factors, including whether or not the study population consisted only of smokers interested in smoking cessation, or all smokers”.
But there is also a further slightly cloudy area which affects much science – not just meta-analyses, and not just these particular researchers’ work – and, importantly, is frequently overlooked in media reporting, as well as by institutions’ public relations departments.
This is the confidence interval, or CI.
Confidence intervals and best guesses
When a study draws numerical conclusions, it often provides a best stab at the exact answer, known as a “point estimate”, then immediately hedges this by stating a “95% confidence interval”: in effect, a range of numbers within we can be 95% confident the true answer will fall. (Strictly speaking, the CI means that if similar studies were conducted over and over again, the correct answer – in public health the one that accurately portrays the whole population, not just the sample used in this particular research – would fall within the CI 95% of the time. But “95% certain” is a reasonable shorthand.)
The crucial thing about the CI is this: it indicates that while the evidence leads in the direction of the point estimate being right, there’s always a possibility that it’s not, thanks to the inherent perils in drawing conclusions about whole populations (in this case, millions of people) from studies of comparatively small samples (hundreds or thousands). This kind of frankness is widely accepted in science, and doesn’t in itself ring warning bells.
Yet clearly, the wider the confidence interval, the less faith can be placed in the point estimate. Even though values toward the edges of the CI are less plausible than those nearer the centre, such as the point estimate itself, a wide CI still admits the possibility that the point estimate might be some distance from the truth.
And in the Kalkhoran/Glantz meta-analysis, the CIs are sufficiently wide to hint that the true answer to the question of e-cigarettes’ usefulness in cessation could be rather different from the one they proffer. They don’t, of course, deny this in their paper; the CIs (which we’ll look at more closely below) are there for anyone to read. But much reporting, and indeed hype, of studies like this meta-analysis tends to ignore them.
So what do their figures say, exactly?
The two researchers use another statistical technique, the “odds ratio” (OR), to express their conclusions. An odds ratio illustrates how the presence of a given factor (an “exposure”) affects the likelihood of a particular thing happening (an “outcome”) – in this case, how the presence of e-cigarette use influences the likelihood of the user stopping smoking. An OR over 1.0 means that the outcome being studied becomes more likely when the exposure is present; an OR below 1.0 means that it becomes less likely.
In the Kalkhoran/Glantz meta-analysis, the attention-grabbing “28% less likely to stop smoking when using e-cigs” was derived from a point estimate of 0.72 for the odds ratio (an OR of precisely 1.0 would mean no effect either way, and 0.72 is 28% less than 1.0).
If spot-on correct, that would indeed imply that e-cigarettes had a notable negative effect on quit attempts, leaving aside for the moment the other flaws identified in the research.
However, the 95% CI for this OR was 0.57-0.91, meaning that e-cigs could be anything from very unhelpful in quitting (0.57) to a minor impediment (0.91). While this surely does suggest that e-cigs are unlikely to have a positive effect – since even the upper boundary of the CI is below 1.0 – it also means that we cannot be super-confident about the 0.72 and its corollary, the “28% less likely”. At a minimum, we should restrict ourselves to saying “quite a bit less likely” rather than putting such an exact figure on it.
Halving the 28%
What’s more, once one of the less technical criticisms of the study has been addressed and those people uninterested in quitting have been excluded from the data – leaving only those who are at least trying to quit – the OR rises to 0.86, meaning that the best-guess “negative” effect of e-cigs is already reduced from 28% to 14%.
Even more tellingly, the 95% CI for this figure is 0.60-1.23, indicating that vaping could indeed be a very bad move for those trying to give up smoking (if an OR toward the bottom end of the CI range happens to reflect reality)…but could also be a positive boon (if anything approaching the top-end OR is right).
This absolutely does not, in itself, mean the Kalkhoran/Glantz conclusion has to be incorrect: the confidence intervals for “all smokers” and “smokers interested in quitting” overlap a great deal, and the OR for e-cigarettes’ helpfulness – even to those definitely wanting to quit – could still mean they are worse than useless.
Moreover, note that the very wide CI for the group interested in quitting means we should place even less trust in the point estimate for that group than we should for the group of all smokers. (It could be so big because wide CIs generally go with small samples, and narrower CIs with larger samples, bearing out the common-sense principle that the more people you study, the closer your answer should be to the reality in the population as a whole. It’s like tossing a coin: two tosses may well not give you a head and a tail, but 100 will probably give you close to 50/50.)
Even so, the numbers don’t rule out the possibility that vaping works in cessation. On their own these statistical question marks might be less concerning, since the point estimate is after all the best guess, but combined with other reservations raised about the meta-analysis, they ought to make us wary.
Data and the real world
Kalkhoran and Glantz do acknowledge in their paper that the picture is subtler than their bottom-line figures, not to mention the University of California’s publicity, might suggest.
Importantly, they note – as do some of their critics – that studies which treat all people who have ever vaped as if they were a single group, without distinguishing the regular users from the occasional experimenters, can be misleading.
After all, a few drags on a friend’s cigalike at a party months ago are hardly likely to make any difference to the outcome of a quit attempt, yet “the definition of e-cigarette use in all but two of the studies could have included people who only used e-cigarettes once”, they recognize.
This is a problem that has been commented upon several times before, for example by a group of Rutgers University scholars back in 2014. It was conceded, too, by a recent Swiss study on vaping and smoking among young men, which found little noteworthy linkage – either positive or negative – between vaping and cessation or reduction. (This research also suffered from the problem of a small sample leading to a very wide confidence interval, and thus gives conclusions that can’t be wholly relied upon. It has been criticized by Phillips for more fundamental problems in its approach, as well, although praised by Glantz.)
Kalkhoran and Glantz also nod to the importance of patterns in e-cigarette usage, for example research showing that regular users (and especially tank users) are more likely to quit successfully than other vapers.
And they propose that e-cigs’ status as consumer products rather than pharmaceutical ones might, perhaps counterintuitively, diminish their value in cessation.
Some research has suggested that nicotine replacement therapy (NRT) seems to be more effective when prescribed than when available over the counter on demand, possibly because in the latter scenario it is not accompanied by professional medical advice, users are less motivated casual purchasers, and compliance (taking your medicine regularly) is poorer.
Could that apply to e-cigs, too – meaning that their low score as cessation aids on the meta-analysis is partly a result of the way they’re regulated and sold, rather than an innate inability to do the job?
Kalkhoran and Glantz put forward some ideas for improving future research in the field. Their suggestions include a “focus on determining standard definitions of e-cigarette use; evaluating the association of different extents of use and different devices with smoking cessation; conducting more randomized clinical trials comparing e-cigarettes to standard therapies such as NRT; evaluating the effect of e-cigarette use on factors such as motivation to quit; and distinguishing e-cigarette users by their reasons for using the products”.
Few, even among their sworn enemies, would disagree with those proposals – and they might even lead to e-cig research which the pro and anti camps can both accept as meaningful.
In the meantime, however, it’s likely that the spats over studies like this one will continue.
Especially in the field of public health, science is often not as clear-cut as it superficially seems. That ambiguity means researchers can’t just be robots, but have to make judgments on their conclusions – and where a topic as controversial as e-cigarettes is concerned, those judgements rarely go unchallenged.
Kalkhoran and Glantz’s paper, entitled “E-cigarettes and smoking cessation in real-world and clinical settings: a systematic review and meta-analysis”, was funded by the National Institutes of Health (NIH), National Cancer Institute, and FDA Center for Tobacco Products.
What This Means: If Stanton Glantz were to suggest that the Earth orbits the Sun, there are e-cig advocates who would surely jump to criticise his methodology, so we should not automatically take every condemnation of his work at face value without digging deeper.
But some critics of this study are credible in the extreme. And it seems undeniable, given not only Glantz’s track record as a pioneer of aggressive tobacco control but also the shifting sands that lie beneath some of the apparently firm conclusions of this meta-analysis, that he and his colleague did manage to find the answers they wanted to find.
That does not, of course, mean their results are wrong. They could even happen to be right, and it’s worth bearing in mind that anecdotal evidence of e-cigarettes being helpful in cessation is also flawed, by being anecdotal: it could well be wrong on a population level, too. Confirmation bias, the tendency to see what you want to see, works both ways.
And yet: useful science is about replicable processes leading to results everyone can agree on, more than it is about results just happening to be right, and the concerns raised about this study do tend to undermine its conclusions – certainly to cast considerable doubt on the precision of the 28%, even if the general thrust of the findings is more firmly supported by the data.
Kalkhoran and Glantz’s most important sections may be those that discuss the weak points of current research on e-cigarettes and smoking cessation, then suggest future work to address them. Moving beyond the scientific sphere, it is to be hoped that this study will not unduly influence policy-makers. While e-cigs are undoubtedly purchased by some consumers as aids to smoking cessation or reduction, and while (as Kalkhoran and Glantz imply) they might even be more efficacious in that role for some people if they were pharma-licensed, e-cigarettes are used for other reasons too. Their effectiveness in quit attempts is certainly important – but it is not the sole criterion on which they should be judged.