Introduction
In the United States, colorectal cancer (CRC) is one of the most common cancers and cause of death, while CRC incidence increases with age.[1] Surgery is one of the main treatments for colorectal cancer.[2] In the recent healthcare structure, multiple reimbursement models for surgical specialties have emerged. Although different in implementation, most of these reimbursement models share the need for evaluating quality of care through measured outcomes.[3] The Centers for Medicare and Medicaid Services (CMS) stated its goal of having 50% of Medicare payments tied to quality or value through Medicare Incentive-based Payment (MIP) or alternative payment models.[4] This goal supports a patient-centered quality system where patients are informed about their options and can make treatment decisions based on best information.
Preoperative risk assessment calculators may be valuable tools for determining patients at high risk for morbidity and mortality. Identifying the probability of perioperative adverse events may facilitate shared decision making by allowing the patient to choose a lower-risk intervention or proceeding with surgery with a better understanding of the potential risks.[5]
The American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) was developed to provide risk assessment and includes a database that analyzes data from preoperative patient risk factors and postoperative morbidity and mortality. This information is then used to evaluate surgical quality.[6] ACS-NSQIP also offers a Risk Calculator (RC) which provides patient-level risk evaluations for specific adverse outcomes based on the current database.[7] In addition, the surgical RC has been recommended by the ACS as an important tool for surgeons and patients during preoperative interactions and discussion of treatment.[8] Further evaluation of the RC for its accuracy in predicting mortality and morbidity as a component of a value-based payment model is needed.
Previous study regarding the ACS-NSQIP RC demonstrated good calibration and discrimination characteristics. In addition, the RC performed comparably to colorectal procedural-specific risk calculators, although this was not expanded to other subspecialties.[9] Subsequent studies, however, re-examining procedural-specific validity have not shown similar validity. Arce et al. studied the predictive accuracy of the ACS-NSQIP RC compared to observed outcomes in head and neck cancer patients undergoing microvascular reconstruction with free flaps, and suggested the ACS-NSQIP RC is not a useful metric for risk stratification in the studied population.[10] The ACS-NSQIP RC also has limitations when predicting outcomes for procedures which have inherently higher risks for specific complications. Although studies have shown the ACS-NSQIP RC to have good internal validity, external validation studies performed at institutional level on surgical subspecialties have concluded the opposite, that the ACS-NSQIP RC lacked accuracy when applied to selective populations.[7-9]
As a selective population, patients undergoing rectal cancer surgery also present challenges due to the inherent nature of the disease and related procedures.[11] Value-based payment models rely partly on administrative data gathered from national databases. Accuracy of predicted patient risks become increasingly important when transitioning to new payment models. If higher risk patient characteristics are not risk adjusted and accurately reported, there may be significant financial consequences.
A paucity of studies specifically examines the ACS-NSQIP RC ability to accurately predict mortality in patients undergoing primary rectal cancer surgery. The goal of this study is to compare the accuracy of the ACS-NSQIP RC in predicting overall morbidity and 30-day mortality within various age groups in primary rectal cancer surgery.
Methods
Data Source: Data were obtained from the 2012-2015 ACS-NSQIP, which collects data from patients undergoing surgery across a wide range of specialties from approximately 700 hospitals in the United States and Canada. These data are compiled directly from patients’ medical records by trained staff at each institution and include a wide variety of pre-operative patient characteristics, operative variables, and 30-day outcomes. Previous studies have utilized ACS-NSQIP data for analysis of proctectomy outcomes. [11]
Inclusion Criteria: Patients were included in the study based on ICD9/10 codes for rectal cancer (154.0, 154.1, C20) and CPT codes for proctectomy (44145, 44146, 44147, 44207, 44208, 45110, 45111, 45112, 45114, 45119, 45395, 45397, 44207, 44208, 44395, 44397). Since ACS-NSQIP does not differentiate years of age above 90, these patients were excluded from the study. There were no further restrictions for inclusion.
Age Classification: In accordance with previously published studies, patient age was categorized as 18-64, 65-79, and 80-89 years.
Study Outcomes: The primary outcomes of interest in this study were: actual 30-day mortality, ACS-NSQIP predicted 30-day mortality, actual morbidity, and ACS-NSQIP predicted morbidity. As categorized in previous studies, actual morbidity was defined as occurrence of at least one of the following complications: organ space infection, pneumonia, unplanned intubation, pulmonary embolism, ventilator requirement at >48 hours, progressive renal insufficiency, acute renal failure, cerebrovascular accident, cardiac arrest, myocardial infarction, deep venous thrombosis, sepsis, septic shock, and return to operating room.
Statistical Methods: Descriptive statistics (means/standard deviations, frequencies/ranges) were calculated to characterize the study population based on demographics, preoperative comorbidities, preoperative labs, and factors related to surgery (emergency vs. elective, wound class, operating time). Normality of continuous data was assessed via Kolmogorov-Smirnov test. Unadjusted differences in patient characteristics between age categories were evaluated using Chi-square test for categorical variables and analysis of variance (ANOVA) for continuous variables. Pearson correlation coefficients were computed to determine the accuracy of predicted and actual mortality overall and in each age category. ANOVA was utilized to evaluate the statistical significance of the unadjusted relationship between actual versus predicted mortality overall and between the three age categories. Logistic regression models were also constructed to estimate this relationship while adjusting for key covariates. Regression models were adjusted for gender, BMI, race, smoking, and functional status. An age x predicted mortality interaction term was included in regression models to determine the extent of effect modification of the relationship between predicted and actual mortality by age category. Statistical significance was defined as p<0.05. All analyses were performed in SAS version 9.4 (Cary, NC, USA).
Results
9,289 patients were included in the analysis. The age distribution of the sample was as follows: 18-64 (n=5,674), 65-79 (n=2,899), 80-89 (n=716). As described in Table 1, older age was associated with female gender, lower BMI, more diabetes, less smoking, less disseminated cancer, more transfusions, and longer length of stay (p<0.05). There was no difference in emergency surgery frequency between age categories (p=0.10). While the frequency of some CPT codes varied meaningfully by age (44207, 45110, 45395, 45397; p < 0.05), most CPT codes included in the analysis did not differ widely across age categories.
Table 2 provides the predicted and actual mortalities across age categories. As expected, both predicted and actual mortality increased with age (p<0.0001) (Figure A) The overall correlation between predicted and actual mortality across the entire study sample was weak (r=0.20). The correlation was weakest from age 18-64 (r=0.07), strongest from age 65-79 (r=0.25), and in between from age 80-89 (r=0.13). Predicted mortality was overestimated in the 18-64 group and underestimated in both the 65-79 and 80-89 groups.
These relationships between predicted and actual mortality were robust to adjustment for covariates (BMI, race, smoking, functional status, and CPT codes that varied substantially by age in bivariate analyses [44207, 45110, 45395, and 45397]) in logistic regression modeling of actual mortality. The age x predicted mortality interaction term was statistically-significant in the regression models, further signifying the varying accuracy of the predicted mortality with actual mortality by age (p<0.05).
Table 3 provides the predicted and actual morbidities across age categories. Both predicted and actual morbidities increased with age (p<0.0001) (Figure B). The correlation between predicted and actual mortality across the entire study sample was weak (r=0.14). The correlation did not vary substantially by age, with age 18-64 (r=0.14) and both 65-79 and 80-89 (r=0.13). Predicted morbidity was considerably overestimated across all age groups.
The relationships between predicted and actual morbidity were also robust to adjustment for the same set of covariates in logistic regression modeling of actual morbidity. The age x predicted morbidity interaction term was not statistically-significant in the models, thereby reflecting the consistency in overestimation of the predicted morbidity across age groups.
Discussion
Currently, CRC is one of the most common cancers and cause of death in the United States [1]. Increasing age has been shown to be a risk factor for postoperative complications after colorectal surgery for cancer [12, 14-15]. The ACS NSQIP RC has been recommended as a useful tool for preoperative discussion with the patient, including goals of care [13]. In addition, due to its strong ability as an outcomes predictor, there is the possibility it may eventually be implemented as part of the MIP program. As such, it is important that any tool be relatively accurate in predicting risks, especially mortality.
The goal of this study was to evaluate the predictive accuracy of the ACS-NSQIP RC for mortality rate and serious postoperative complications in patients who had surgery for primary rectal cancer by comparing predicted and actual patient outcomes. The study found while both predicted and actual mortality increased with age, the overall correlation between predicted and actual mortality increase was low. Correlation remained low after stratification into different age groups; degree of correlation with predicted mortality in older age groups was at least twice as high as the youngest age group. Predicted mortality was also underestimated in the remaining older age groups. These results remained true after adjusting for covariates via logistic regression analysis. Meanwhile, overall serious complications were overestimated in each age group.
In this study, the ACS-NSQIP RC was found to underestimate overall and age stratified mortality and overestimate serious comorbidities in all age groups after primary rectal cancer surgery. In addition, there is variation in accuracy of the RC between different age groups as demonstrated by the underestimation of mortality in older age groups and overestimation of mortality in the younger age group. Prior studies [16-17, 24-27] reported while RC was a capable tool for overall morbidity and mortality risk assessment, it is less accurate when used for individual quality outcomes or when applied to a more focused subset of a patient population as seen in various subspecialties. Keller DS, et al. [7] found in elective colorectal surgery the RC correlated well for predicting complications in general but predictive accuracy for identifying actual occurrences was poor, except mortality, which had good accuracy. Adegboyega TO, et al [27] found the RC underestimated specific and overall complication rates in colorectal surgeries. Bergquist, JR et al [28] showed that the RC failed to predict surgical site infection in an independent prospective database of colorectal surgeries performed at the Mayo Clinic. Of note, a paucity of literature was found regarding predictive accuracy of RC in the older population undergoing primary rectal cancer surgery.
Overestimation of serious comorbidities in all age groups may also be due to improving care since the start of the ACS-NSQIP. Cohen ME, et al [29] found reductions in adverse events after surgery in hospitals participating in ACS-NSQIP with the magnitude of quality improvement increasing with time of participation. While the ACS-NSQIP database is updated continuously, the RC itself may still be incorporating older data with much higher complication rates, skewing overestimation of comorbidities. A similar study could be done after the next ACS-NSQIP RC update to examine whether more current data would change predictive accuracy.
Further refinement of the RC algorithm is needed for calculating predicted risk in older patients undergoing primary rectal cancer surgery. The oldest patient population subgroup faces unique challenges such as cumulative loss of physiologic reserve in nearly all organ systems which may require further adjustment of the RC risk modifiers. [30] McMillan, MT et al, [31] previously compared the ACS-NSQIP RC and a procedural-specific model for pancreaticoduodenectomy which incorporated intraoperative factors and demonstrated better morbidity and mortality predictive accuracy with the latter. A similar model may be applicable to the older patient population as well. Colorectal surgery is also associated with higher inherent risks of complications, often independent of surgical technique and quality of patient care [32-33]. Due to these factors, special consideration by the surgeon is needed when using the RC for older patients undergoing surgery for primary rectal cancer. Finally, there is the ongoing ACS-NSQIP Geriatric Surgery Pilot program which expands the surgical database to include information that reflects the unique needs of older adults. This expanded database was created to look at geriatric-specific risk factors to improve the ability of surgeons to predict poor outcomes and to provide the best possible care focusing on quality outcomes. It is hoped that a RC focused on the older surgical population may be better suited for risk prediction.
Limitations of this study include being retrospective along with its inherent shortcomings. Other limitations include accuracy of the ACS- NSQIP and the RC with potential for errors in data collection and coding, which has been previously described [9, 12]. In addition, there was no adjustment for surgeon experience as this was not reported. The number of actual mortalities is low relative to sample size which may limit reliable estimates of predictive accuracy. Surgical RC parameters were not re-estimated for validation data sets, which my increase effects of case-mix homogeneity from model inadequacy.
Conclusion:
ACS-NSQIP RC predicted mortality risk estimates in older patients appear to underestimate overall mortality and overestimate overall serious morbidity risk. Despite this, ACS-NSQIP RC remains a useful tool for the surgeon in preoperative patient education; however, accuracy is still lacking as a predictive tool. Surgeons should be mindful of limitations of risk calculators when discussing surgery with patients. Further refinement of the ACS-NSQIP RC is needed prior to being employed as part of a quality- based healthcare metric.