Abstract:
The health impacts of second hand smoke (SHS) exposure are well documented and international evidence has shown that smoke free laws can have a positive impact on health outcomes. On March 2004, the Republic of Ireland became the first country in the world to implement a national workplace smoking ban. Italy banned smoking in all indoor places in January 2005.
There are different types of studies examining the effect of smoking ban and their quality should be assessed based on the type of data presented in the study and the questions to be answered by the analysis. Evaluation of health policy can be challenging as there are different interpretation concerning which approaches are best suitable for the evaluation.
I have selected two studies with different design assessing the effects of legislative smoking ban on morbidity and mortality from exposure to SHS. One study uses an interrupted series design and the other uses before-and-after methods with no control group. Both studies reported significant improvement in health outcomes after comprehensive bans implementation.
This essay will discuss the importance of study design, focusing on threats to internal validity. It will also critical appraise the studies methods, exploring different quality assessment tools appropriate for such type of studies.
Introduction:
Healthcare System is a living system therefore needs to be evaluated based on its components interaction, functional perspectives, and the success of performance. The management of any healthcare system is typically directed through a set of policies and plans. According to the World Health Organization (WHO), health policy refers to decisions, plans, and actions that are undertaken to achieve specific health care goals within a society.1
In order to achieve this goal, the first step is problem identification and issue recognition.2 Once the intervention is formulated and implemented, the following step is evaluation. It is of uttermost importance to effectively examine the relationship between the practice of public health and population outcomes. As we move further and endeavor to improve the performance of the healthcare system and its ability to respond effectively, we need research on the continued longer or shorter term impact of the health policy.
The term ‘passive smoking’ was first used in the scientific literature in a paper published in the Lancet in 1974.3 The smoking impact not just on smokers, but on others around them, radically transformed the ethical basis of public health policy on smoking.4, 5 As the evidence on passive smoking have grown, it evolved the status of smoking from individual perspective on health risk into a public health problem that could justify interventionist policies designed to protect others from harm.6
A significant body of international evidence now exists which demonstrates that smoking bans are effective in reducing exposure to second hand smoke (SHS). In 2008, Cesaroni et al7 published a before-and-after study evaluating the effect of the Italian smoking ban on population rates of acute coronary events. In 2013, Stallings-Smith et al8 evaluated the correlation between smoking ban implemented in Ireland and the reduction in cardiovascular, cerebrovascular, and respiratory mortality using interrupted time-series analysis.
In the first part of this essay I will provide a background and overview to the tobacco control intervention. The second part will provide a brief overview to the importance of study design, risk of bias and confounders as threats to internal validity, and explore some of the quality assessment instruments. The third part will look specifically into selected studies and critically appraise their validity, underlying their strengths and weaknesses.
Section 1: Tobacco Control and smoking bans
Among the global health policies, tobacco control under the treaty Framework Convention on Tobacco Control – World Health Organization (WHO – FCTC) represents a paradigm-shift in developing a regulatory strategy to address addictive substance, in contrast to previous drug control treaties.9
The aim of the policy was to reduce the harms of passive smoking and to provide a supportive environment for people who want to stop smoking.10 Numerous debates were triggered for the usefulness of the policy related to countries specificity.
Protect nonsmokers
Smoke-free Positive health impact
places
Increase tobacco cessation
Reduce consumption
Reduce initiation
Tobacco is the second major cause of mortality in the world and currently responsible for the death of about one in ten adults worldwide. WHO estimates that six million people die every year from tobacco related disease and 600,000 due to the effect of passive smoking.
Countries are at different stages in controlling and addressing tobacco use. This is related significantly to their political and socioeconomic conditions. Ms Kristina Mauer-Stender, Program Manager for Tobacco Control in WHO Regional Office for Europe stated that tobacco use is not a choice: ‘It is a powerful addiction. The true choice is between tobacco and health.’
One step for controlling this addiction is the awareness of noxious effect of smoking not only for smokers themselves, but also for the people close to them. There is strong evidence that passive exposure to tobacco smoke is harmful to human health.10
The first epidemiological research into passive smoking by monitoring the health of nonsmoking women married to heavy smoking husbands was conducted by Takeshi Hirayama back in 1981.11 The objective of the study was to measure the wife’s risk of developing lung cancer related to the husband’s smoking habit. The study followed 91 540 non-smoking housewives aged 40 and above, who answered to detailed questions about smoking habits. Wives of heavy smokers were found to have a greater risk of developing lung cancer comparing to wives of non-smokers. Moreover, the effect of passive smoking were compared with the effect of direct smoking, indicating that the effect of passive smoking is about one-half to one third that of direct smoking in terms of mortality ratio or relative risk.11
His findings were later challenged by the tobacco industry in an attempt to suppress the association between exposure to environmental tobacco smoke and health damage.12
Thirty-six years later, we continue to study the relationship between exposure to secondhand smoke and health impairment such as respiratory and cardiovascular diseases. However, the remaining question is how can we avoid the distortion of scientific findings? To meet this challenge it had become crucial to rethink our approach and place quality at the heart of our evaluation. Moreover, correctly choosing the applicability of different tools for quality assessment is of equal importance.13
Correctly choosing
the applicability of the tools is very important;
Correctly choosing
the applicability of the tools is very important;
Even though much progress has occurred regarding research to support the implementation, enforcement, and evaluation of smoke free-laws, there is still room for improvement.10 Different types of studies have been published examining evidence into the effects of passive smoking and the associated health risks including heart and pulmonary diseases. Studies need to be critically evaluated in order to prove their validity, to identify the risks and benefits and to try understanding how policies are transferred around the world.2 Yet there are limits to study interpretation and this can be challenging. Unfortunately, researchers often stray from the core principles of study design required to provide valid evidence. The result is that policymakers and the public do not always know what to believe.14
Section 2: Domains of evidence-based policy
Selection of study design
Brownson proposed three domains of evidence based policy: process for understanding approaches to increase the probability of policy adoption, content for identifying elements likely to be effective, and outcome for documenting the potential impact of intervention.15
Policy evaluations (outcomes of the intervention) are critical to the understanding of the impact of policies on community- and individual-level behavior changes. Documenting the effects of implemented policies is equally important in supporting evidence-based policy.15 Evidence informed decision-making assumes people make better decisions if informed by best available evidence. Policy evaluations may employ both quantitative (i.e., data in numerical quantities) and qualitative (involving non numerical observations) methodologies and may make use of ‘natural experiments’.
One of the key game changers in evaluation is represented by the study design. Study design plays an important role in the quality, execution, and validity to answer questions of interest. Each study design has its own inherent strengths and weaknesses, and there can be general biases triggered by the study design per se.
In order to assess the effectiveness of an intervention, the ultimate goal is to provide evidence about the level of internal and external validity for both categories of study design: experimental and observational.
Internal validity refers to what degree the study’s results could be attributed to the intervention evaluated, while external validity refers to generalisability of the study’s results outside of the intervention context.
The golden standard for evaluating the effectiveness of an intervention has been represented by randomized controlled trials (RCTs). RCTs are essential for evaluating the efficacy of clinical interventions, where the causal chain between the agent and the outcome is relatively short and simple; moreover the results may be safely extrapolated to other settings.16
In the absence of RCTs, in the situation of complex causal chains, healthcare practitioners and policy-makers rely on non-randomised studies to provide evidence of the effectiveness of healthcare interventions. However, there is controversy over the validity of non-randomised evidence, related to the existence and magnitude of selection bias.
Despite the fact that interrupted time series is considered a valuable study design, the strongest, quasi-experimental approach for evaluating longitudinal effects of health policies and programs targeted at the population level17,18 their internal validity should be evaluated.
Confounders and bias represent important threats to the validity of studies evaluated. The most common and dangerous threats to research validity is history bias (ie, researchers’ failure to consider relevant events or change that precede an intervention or co-occur while it is in progress).19 Before-and-after studies that fail to control for history could represent a threat for policy makers.
Quality assessment instruments
Numerous tools exist to assess methodological quality, or risk of bias in RCTs and non-randomised trials. Also, the choice of quality assessment instruments could be a laborious task owing the differences of tools. Instruments are considered from a compendium of methods that best suit the study design. A number of tools were developed by academic organizations, and some were developed by only a small group of researchers. Instruments for assessing the methodological quality of randomized controlled studies were most abundant. The best available tool for assessing RCTs is Consolidated Standards Of Reporting Trials20, a 25 item checklist and flowchart specifically designed for RCTs to standardize reporting of key elements including design, analysis and interpretation of the RCTs.
In order to evaluate non-randomised studies, from a total of 213 potential quality assessment tools identified, Deeks et al21 selected 6 tools as being suitable for use in a systematic review:
• Cowley quality assessment tool for use in a systematic review of total hip replacement with 13 questions for the quality assessment of comparative studiesa
• Downs scale to assess the methodological quality of randomised and non-randomised studies with a list of 27 questions to measure study qualityb
• Newcastle–Ottawa designed for use in an epidemiological systematic review, containing eight itemsc
• Reisch tool to facilitate the evaluation of the design and performance of therapeutic studies and to teach critical appraisal skills with a list of 57 itemsd
• Thomas tool to assess the methodological quality of studies including 21 itemse
• Zaza tool designed for use as part of a data collection instrument for systematic reviews with 22 quality itemsf
The most relevant aspect for selecting these tools is how well they address internal validity and particularly integrates the four core items: selection bias, performance bias, attrition bias, detection bias.
The author’s conclusion infers that overall the tools were poorly developed and even the six considered potentially suitable for quality assessment require some modification to meet the prespecified criteria.21
For my research I chose an innovative tool proposed by W.Zingg et al: Integrated quality criteria for review of multiple study designs (ICROMS).22
As the purpose of my research is to uncover and identify the strengths and weaknesses of two non-controlled studies, I considered ICROMS as being the most suitable for my critical appraisal. The decision was triggered by the studies’ different design and their implications. I based my critical appraisal on ICROM as being a comprehensive tool which includes both established criteria for RCTs and incorporates criteria for non-controlled studies.
ICROMS allows attributing points to a study when it successfully meets each criterion – 2 points if a criterion is met; 0 if it is not met, and 1 if it is unclear whether or not a criterion is met. The global quality score for each study is obtained by summing the points attributed to each criterion. Moreover, a Decision Matrix is provided to facilitate transparent decision, specifying the minimum requirements for each study type.
Section 3: Critical appraisal based on ICROMS
Studies validity
The main objective of both studies was to evaluate the effectiveness of the implementation of comprehensive smoking bans. Reliable primary outcome were measured: Stallings-Smith et al assessed association between ban (implemented in March 2004) and mortality rates (cardiovascular, cerebrovascular, and respiratory), and Cesaroni et al investigated the effect of ban (implemented in January 2005) on acute coronary events.
Stallings-Smith et al selected an interrupted time series (ITS) design and Cesaroni et al opted for before-and-after design. Both are appropriate designs for smoking ban evaluation as large data sets were investigated, covering national mortality data for the studied period (Stallings-Smith et al) and Rome residents between 35 and 84 Years (Cesaroni et al).
Smoking ban was implemented in all regions of both countries simultaneously. Thus, the authors were unable to compare their results against a control population unaffected by the ban. Furthermore, it was not possible to blind participants as the intervention was a national public policy and smoking is visible. In order to confirm principal diagnose, International Classification of Diseases (ICD) coding was used in both studies.
Quasi-experimental study designs are particularly useful when a randomised trial is infeasible or unethical.15 The major strengths are represented by ability to control the effect of secular trends, readiness to analyse the unintended consequences of interventions, extremely clear and easy-to-interpret graphical results. It should be noted, however, that there are also important threats to validity in ITS analyses.
Sources of bias
As presented by Hussein Naci (Health Policy Evaluation, session 3), six biases are potential threats to the internal validity of studies: history, maturation, testing and instrumentation, statistical regression, selection, and attrition.
For ITS studies, three are important: history, followed by changes in instrumentation and selection.
Both studies in scope addressed most threats to internal validity except for a special form of history bias (co-intervention or co-occurring event). New diagnostic criteria and new prescribing practice in Italy, and respectively coal bans implementation across Ireland could be considered as history bias.
Analysis performed were sufficiently rigorous for both studies: Stallings-Smith et al used time-series epidemiologic assessment utilizing Poisson regression adjusted for time trend, season, influenza, and smoking prevalence. Cesaroni et al used Poisson regression analyses on the calendar-time axis adjusted by particulate matter (PM10) air pollution, temperature, influenza epidemics, time-trend, and total hospitalization rates.
Furthermore, the authors tried to address limitations. Stallings-Smith et al identified a few limitations such as: 1. missing direct adjustment for weather due to the small number of weekly mortality events remaining after stratification by age, gender, and region, and 2. air quality improvements. Cesaroni et al mentioned as major weakness the absence of control population.
The conclusions of the studies were clear and justified. Immediate and consistent evidence of reduce mortality is reported, with a significant number of deaths likely prevented (Stallings-Smith et al). Statistically significant reduction in acute coronary events in adult population observed after the smoking ban in Italy (Cesaroni et al).
Strengths and weaknesses:
Despite the different study design, both are methodologically robust studies, incorporating large data sets (national data registries) facilitating modelling and regression analyses and adjusting for non-linear trends and confounders. Among the strengths of both studies is the large sample covering national mortality data for Stallings-Smith et al and respectively residents of Rome between 35 and 84 Years (~2.7 million) for Cesaroni et al. Both studies investigate the impact of smoking ban on several types of acute respiratory and cardiovascular events, including other that were less explored. In addition, studies use a robust Poisson regression analysis to better assess the specific effect of the smoking ban and interaction with influenza epidemics. Studies adjusted for air quality, pollution, influenza rates and seasonality in an effort to reduce confounding and influence on health outcomes.
Limitations in both studies are the absence of random sequence generation and blinding of participants. The smoking ban was implemented in all regions of both countries simultaneously thus, the authors were unable to compare their results against a control population unaffected by the ban.
The use of data from a single region (Rome) represents a study limitation for Cesaroni et al, and there is a need to study a larger number of regions to assess the impact of smoking ban in health outcomes.
Potential factors that could have led to bias include: implementation of the new diagnostic criteria for acute myocardial infarction and changes in use of statins during the study period (Cesaroni et al). As Stallings-Smith et al analyzed retrospectively administrative hospital data, inaccurate coding of ICD-10 diagnoses at admission is possible and could potentially lead to misclassification bias. In 2007, a 43% decrease in pneumonia/influenza mortality was observed compared to 2006. Therefore, 2007 data were excluded. Furthermore, biases identified for Stallings-Smith et al were related to air quality improvement due to concomitant coal bans implementation across Ireland during the study period and concurrent advertising ban for tobacco.
Conclusion:
Health policy should be evidence-based in order to understand how policy was designed, executed and its dynamics, weaknesses and limitation, and ultimately the policy’s potential of reiteration. Evidence-based public health requires further refinement of protocols for conducts and evaluates the study designs. Similar to the Cochrane Collection for RCTs, specifically designed tools are needed for evaluating large-scale public health interventions.
Smoke free environment provide an example of how research can be translated into policy.10 The implementation of a legislative smoking ban has been associated with improved quality of life for both active smokers and second or third hand smokers. The smoking ban impact on longer- term needs to be further reinforced based on more robust research. Additional research is needed to reach final conclusions where evidence is only suggestive of causality. Documenting the smoking harmful effect in studies should be more consistent and should use validation methods.23
Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. I have identified a variety of methodological assessment tools for different types of study design. Yet, study quality could be described as a subjective concept, open to different interpretation depending on the reader.
In conclusion, formal approaches to assessing quality are required when it comes to evaluate the effectiveness of an intervention. In spite of the existing wide range of quality assessment tools, further efforts are needed since there is currently a lack of such tools universally accepted. In addition, it is aslo important to make the distinction between the methodological quality and the reporting quality. If researchers ought to meet both challenges, they need to be more rigorous in their study design and more diligent in their reporting.13