Metrics Marked as Most Important

In response to feedback that the original 5-tier outcome assessment focused too much on posttransplant outcomes when pretransplant outcomes, eg, transplant rate and pre-transplant mortality rate, may have a greater impact on overall patient survival, SRTR added 5-tier assessments for both survival on the waiting list (based on the standardized pre-transplant mortality rate ratio) and getting a transplant faster (based on the standardized transplant rate ratio). Methods for calculating the 5-tier assessments can be found here. An additional suggestion was to highlight the metric of most importance to survival after listing. In response to these suggestions, SRTR undertook an analysis to determine which of the three metrics to highlight. The updated version of the website includes the icon  above one of the three columns with tier assignments for heart, kidney, liver, and lung searches as shown in the figure.

SRTR estimated the association of the 5-tier evaluations for survival on the waiting list, getting a transplant faster, and 1-year organ survival at listing, with candidate mortality after listing. This analysis directly assesses the utility of each metric for informing patients of potential differences between programs in mortality after listing. The analysis also enables comparison of the relative importance of each metric on the scale in which the 5-tier evaluations are interpreted. For this example, we included candidates listed between July 12, 2011, and July 16, 2014. The tier rating for each metric at the time of listing was determined through archived program-specific reports (PSRs). Candidates were then followed prospectively for overall survival after listing, censoring follow-up on December 31, 2016. Cox proportional hazards models assessed the effect of a 1-tier increase in each metric on overall survival after listing. We adjusted for every factor in the SRTR risk-adjustment models for pre-transplant mortality in the January 2018 PSR release. Tier assignment (1-5) for each metric was included in each model to determine the relationship of the tier to survival after listing. The analysis was completed only for heart, kidney, liver, and lung candidates.

The hazard ratios (HRs) associated with a 1-tier increase in each of the three metrics are shown in Table 1. Considering the row of results for kidney candidates, we see that a 1-tier increase in the survival on the waiting list  tier, ie, changing from a tier-1 to a tier-2 program, or a tier-2 to a tier-3 program, etc., is associated with a 3% reduction in the hazard of death after listing (HR 0.97, 95% confidence interval 0.95-0.98); similarly, a 1-tier increase in the standardized getting a transplant faster ratio is associated with a 5% reduction in the hazard of death after listing (HR 0.95, 0.94-0.96). Finally, a 1-tier increase in the tier for first-year organ survival after transplant was not shown to affect survival after listing (HR 1.00, 0.99-1.01). From these results, we conclude that the getting a transplant faster tier has the greatest impact on a candidate’s hazard of death after listing and assign the  icon to that column for kidney program search results. Similarly, we conclude that the getting a transplant faster tier has the greatest impact for liver candidates (HR 0.90, 0.89-0.91) and heart candidates (HR 0.96, 0.93-0.99), while the tier for first-year posttransplant organ survival has the greatest impact for lung candidates (HR 0.95, 0.92-0.98).

Table 1. Effect of a 1-tier increase on risk of death after listing for kidney, liver, lung, and heart candidates.

The hazard ratios in Table 1 represent the effect of moving up 1 tier in performance. Table 2 translates these hazard ratios into the effect of moving 1 or more tiers. For example, a 4-tier difference in the getting a transplant faster tier for kidney programs, ie, changing from a tier-1 to a tier-5 program, is associated with a 19% reduction in the hazard of death after listing.

Table 2. Effect of various differences in tier assignments on overall patient survival after listing.

Through a research project funded by the Agency for Healthcare Research and Quality, researchers at the Hennepin Healthcare Research Institute (Principle Investigators: Ajay Israni, MD, MS, and Cory Schaffhausen, PhD) conducted a randomized-controlled trial of various versions of the website with the goal of determining which version was most successful at guiding users to the column of greatest importance to overall survival after listing. Using a factorial study design, they conducted a randomized trial of website versions that included various combinations of the impact icon, the table of national rates, and the disclaimer that past performance does not necessarily reflect current or future outcomes. More than 190 participants were recruited per website version. They were randomly shown one version of the website and asked to choose from three hypothetical hospitals. The goal was to guide participants to select the program with the highest transplant rate (consistent with the current icon placement for heart, kidney, and liver programs); 47% percent chose the hospital with the highest transplant rate when the icon was present, compared with 39% when the icon was not present. Between the impact icon, the table of national rates, and the disclaimer, the impact icon was the only version that succeeded in guiding the most users to choose the program with the highest transplant rate. When the impact icon was not present, users tended to choose the program with the highest 1-year organ survival.

These analyses provide information on the relative utility of each metric to patient decision making, and the associations with candidate mortality after listing are arguably more important than eventual posttransplant survival. The public reporting should guide users to the metrics associated with candidate mortality after listing, especially the metrics with the strongest associations (ie, the getting a transplant faster evaluation for kidney, liver, and heart transplantation).