What’s the best way to measure transplant program performance?

Improvement

What’s the best way to measure transplant program performance?

Richard Formica, M.D., on a new proposal to improve transplant program performance monitoring.

Richard Formica

“As a community, we have to take this seriously. We are the right ones to make these changes—we know and understand the system and are in the best position to make these improvements.”

Richard Formica, M.D.

Why this project?

What are the goals?

What will the new metrics be?

Everyone involved in organ donation and lifesaving transplantation wants an efficient system focused on patient safety, increasing transplant and maximizing organ utilization.

But what is the best way to measure transplant program performance?

The Organ Procurement and Transplantation Network’s Membership and Performance Standards Committee is currently at work on a proposal to improve transplant program performance metrics, and intends to release it for public comment in August of this year.

The MPSC aims to develop a more patient-centered and holistic approach to monitoring transplant program performance. The committee hosted a national webinar on June 2 to address the proposed changes and help educate the community so that they are able to provide feedback during the next cycle of public comment.

At-a-glance: the MPSC transplant program performance project
  • The current single metric approach needs to be improved—transplant programs are graded solely on whether a graft functions for a full year.
  • The MPSC is proposing to use both pre- and post-transplant metrics to monitor performance, in a more patient-centered approach.
  • These changes present a paradigm shift, but will utilize metrics that already exist, and that members can influence through quality improvement.
  • The metrics will be risk-adjusted—this allows the MPSC to account for differences in patients and donors in order to make a more valid comparison of other factors, such as treatment practices. Learn more about risk adjustment.
  • The new approach will not result in more programs having to interact with the MPSC.
  • The transplant community gets to provide feedback during fall 2021 public comment and help shape the proposed new metrics.
  • The SRTR’s 5-tier outcome assessments are not part of the proposal.
  • There is nobody better positioned to develop more holistic performance monitoring than the professionals who work every day to save lives through transplantation.

Why is the MPSC taking on this project?

Richard Formica, M.D., chair of the MPSC Performance Monitoring Enhancement subcommittee, delivered the June 2 webinar presentation to the community, and explains there has been growing conversation over the past decade to improve monitoring of transplant program performance.

The OPTN is required to monitor member performance, he says, including threats to patient health and public safety. The OPTN also must develop processes to evaluate, assess and monitor member performance over time for compliance with bylaws and policies. “As a community, we have to take this project seriously,” says Formica. “We are the right ones to make these changes—we know and understand the system and are in the best position to make these improvements.”

Formica is Director of Transplant Medicine at Yale University School of Medicine, where he is also Medical Director of Adult and Pediatric Kidney Transplantation and Medical Director of Pancreas Transplantation. A member of the MPSC since 2018, Formica has a long record of peer support, and has previously chaired the Kidney Transplantation Committee. He is also immediate past president of the American Society of Transplantation.

icon of webinar with person on computer monitor talking
Watch Formica’s webinar presentation on the MPSC’s proposed metrics.

Goals of the performance monitoring enhancement project

Currently the MPSC uses a single metric—one-year patient and graft survival—to monitor transplant program performance. Formica says the broader transplant community feels the single one-year metric is outdated and can be a disincentive to transplantation.

“We as transplant professionals want to transplant more people, and we’re trying to make changes that will allow programs to feel like they can accept more organs.”

The goals of the performance monitoring enhancement project are:

  • To develop a holistic review of member performance throughout all phases of transplant.
  • To identify real-time patient safety issues.
  • To provide support and collaboration to transplant programs with identified opportunities for improvement.
  • To ensure maximum support for increasing the number of transplants, promoting equitable access to transplantation and fostering innovation.

When asked what makes a good metric, Formica says “it should be something that is important, and that can be reliably measured.” In his eyes, a good metric should be something that the monitored entity—in this case, transplant programs—can impact.

What else is important? “We don’t want to let perfect be the enemy of progress,” he says. “This process will be regularly reviewed and updated.”

The proposed metrics

For many years, the MPSC’s monitoring of transplant programs has relied on a single data point—one-year post-transplant patient and graft survival (whether a transplanted organ works for a full year).

As Formica outlined in the June 2 webinar, the MPSC proposes to use four different pre- and post-transplant metricsall risk-adjusted—to monitor transplant program performance:

  • Offer acceptance: A program’s rate of accepting organ offers relative to the expected based on similar offers across the nation.
  • Waitlist mortality: A program’s waitlist mortality rate relative to the mortality rate experienced by similar patients across the nation.
  • 90-day graft survival hazard ratio: In the first 90 days post-transplant, a graft is counted as failed if there has been graft failure or death.
  • One-year graft survival conditional on 90-day graft survival hazard ratio: After the patient has been released to longer-term post-transplant care, a graft is counted as failed if there has been graft failure or death.

Resource sheet for the MPSC's proposed metrics

Circle diagram showing 4 quadrants the proposed metrics. Left half is waitlist management with 2 segments: Offer acceptance ratio and waitlist mortality ratio. The right half post-transplant outcomes with 2 segments: 90-day graft survival and conditional one-year graft survival.

Until now, the MPSC has not monitored pre-transplant metrics. The committee acknowledges that transplant programs may worry about their ability to control these metrics, and wants programs to understand they have the ability to improve their pre-transplant metrics through waitlist management and by increasing offer acceptance. Formica says that organ offer acceptance is actually the one metric that programs truly have complete control over—chance plays a minimal part, and the program controls the denominator and the numerator.

“Programs have to be able to alter the course of the process, and anyone being monitored has to be able to have their actions impact the outcome,” stresses Formica. Perhaps most importantly, he says, “any metric used should be understandable. We have to educate the community, particularly about risk adjustment.”

icon of webinar with person on computer monitor talking
Watch Formica’s webinar presentation on the MPSC’s proposed metrics.

The importance of risk adjustment

Risk-adjusting transplant metrics is crucial because it helps the MPSC account for differences in patients and donors in order to make a more valid comparison of other factors, such as treatment practices. This allows transplant programs to make what they feel are appropriate clinical decisions for the patients they serve. Formica says not adjusting monitoring metrics for risk can contribute to risk-averse behavior and can result in unintended consequences such as underutilization of organs. “We want to avoid reinforcing well-meaning behaviors that are intended to lessen risk but in actuality can reduce the number of transplants.”

Formica says a simple example of risk adjustment is already made for age in relation to the current one-year post-transplant patient survival metric for kidney patients. “Looking at the age of transplantation, data shows that one-year patient survival decreases for patients 50 years and older. You would then expect patients older than 50 to have lower survival than patients younger than 50—so risk-adjustment helps account for that in the monitoring metrics.”

Risk-adjusting metrics means that transplant programs are not unduly penalized for transplanting older patients or accepting more complex organs in order to increase utilization and save more lives.

Performance monitoring needs to emphasize peer support and self-improvement

While some in the community may worry that more metrics means a greater likelihood of being identified for performance review, Formica and his colleagues strongly disagree. “I know there is concern, but our intention is not to create an environment where programs feel like they are more at risk.” He wants to reassure the community that the MPSC has designed the metrics so that there will not be an increase of interactions about performance. “The vast majority of programs are doing fine and don’t need to have any MPSC involvement,” says Formica.

This proposed framework would identify a similar number of programs for review across four metrics, and other programs who fall into what the committee calls the “yellow zone” would have optional self-improvement. There are many resources available to members that can assist them with process improvement—quality assurance and performance improvement (QAPI) services such as education on available data, process mapping and coaching sessions, peer mentoring, and collaborating through case studies.

Formica says it’s important for the transplant community to hold themselves accountable for all of the decisions they make, including not accepting organs for their patients. “The transplant waiting list is a dangerous place to be. And waitlist mortality jumps dramatically after an organ offer has been declined for a kidney candidate,” he says, pointing to research by Sumit Mohan, M.D., and others about the association between declined offers and outcomes in kidney candidates.

“I think we want to get programs to realize that doing better—and having your colleagues help you do better—means moving away from a ‘center vs. center’ framework. If we change the paradigm, then we could start working to get more organs transplanted. We’re all in this thing together.”

The MPSC released their proposal to enhance the transplant program performance monitoring system for public comment on Aug.3, 2021. All proposals this cycle are open for public comment through Sept. 30, 2021, on the OPTN public comment page. During the two-month public comment period, the community is invited to provide feedback and input about the proposed changes to transplant program performance monitoring.

Do you have questions about the MPSC performance monitoring project?

Please contact Sharon Shepherd at Sharon.Shepherd@unos.org.

icon of webinar with person on computer monitor talking
Watch Formica’s webinar presentation on the MPSC’s proposed metrics.

What is the MPSC?

The OPTN Membership and Performance Standards Committee is a performance partner, providing feedback on and recommendations to members to help them improve.

MPSC: who they are and what they do

The MPSC:

  • Has oversight responsibilities aligned with the OPTN mission to maximize organ supply, ensure patient safety, and provide equitable access to transplantation, and it needs the best tools to do this work.
  • Reviews applications for membership in the OPTN, approvals of designated transplant programs and changes in OPTN member key personnel.
  • Works to safeguard the integrity of the transplant system and helps OPTN members reach their maximum potential. 
  • Monitors member performance using data from the Scientific Registry of Transplant Recipients.
Illustration of a key

“I think we want to get programs to realize that doing better—and having your colleagues help you do better—means moving away from a ‘center vs. center’ framework. If we change the paradigm, then we could start working to get more organs transplanted. We’re all in this thing together.”

Richard Formica, M.D.

Share This