What’s the best way to measure transplant program performance?

But what is the best way to measure transplant program performance?

The Organ Procurement and Transplantation Network’s Membership and Performance Standards Committee is currently at work on a proposal to improve transplant program performance metrics, and intends to release it for public comment in August of this year.

The MPSC aims to develop a more patient-centered and holistic approach to monitoring transplant program performance. The committee hosted a national webinar on June 2 to address the proposed changes and help educate the community so that they are able to provide feedback during the next cycle of public comment.

Why is the MPSC taking on this project?

Richard Formica, M.D., chair of the MPSC Performance Monitoring Enhancement subcommittee, delivered the June 2 webinar presentation to the community, and explains there has been growing conversation over the past decade to improve monitoring of transplant program performance.

The OPTN is required to monitor member performance, he says, including threats to patient health and public safety. The OPTN also must develop processes to evaluate, assess and monitor member performance over time for compliance with bylaws and policies. “As a community, we have to take this project seriously,” says Formica. “We are the right ones to make these changes—we know and understand the system and are in the best position to make these improvements.”

Formica is Director of Transplant Medicine at Yale University School of Medicine, where he is also Medical Director of Adult and Pediatric Kidney Transplantation and Medical Director of Pancreas Transplantation. A member of the MPSC since 2018, Formica has a long record of peer support, and has previously chaired the Kidney Transplantation Committee. He is also immediate past president of the American Society of Transplantation.

Goals of the performance monitoring enhancement project

Currently the MPSC uses a single metric—one-year patient and graft survival—to monitor transplant program performance. Formica says the broader transplant community feels the single one-year metric is outdated and can be a disincentive to transplantation.

The goals of the performance monitoring enhancement project are:

• To develop a holistic review of member performance throughout all phases of transplant.
• To identify real-time patient safety issues.
• To provide support and collaboration to transplant programs with identified opportunities for improvement.
• To ensure maximum support for increasing the number of transplants, promoting equitable access to transplantation and fostering innovation.

When asked what makes a good metric, Formica says “it should be something that is important, and that can be reliably measured.” In his eyes, a good metric should be something that the monitored entity—in this case, transplant programs—can impact.

What else is important? “We don’t want to let perfect be the enemy of progress,” he says. “This process will be regularly reviewed and updated.”

Until now, the MPSC has not monitored pre-transplant metrics. The committee acknowledges that transplant programs may worry about their ability to control these metrics, and wants programs to understand they have the ability to improve their pre-transplant metrics through waitlist management and by increasing offer acceptance. Formica says that organ offer acceptance is actually the one metric that programs truly have complete control over—chance plays a minimal part, and the program controls the denominator and the numerator.

“Programs have to be able to alter the course of the process, and anyone being monitored has to be able to have their actions impact the outcome,” stresses Formica. Perhaps most importantly, he says, “any metric used should be understandable. We have to educate the community, particularly about risk adjustment.”

The importance of risk adjustment

Risk-adjusting transplant metrics is crucial because it helps the MPSC account for differences in patients and donors in order to make a more valid comparison of other factors, such as treatment practices. This allows transplant programs to make what they feel are appropriate clinical decisions for the patients they serve. Formica says not adjusting monitoring metrics for risk can contribute to risk-averse behavior and can result in unintended consequences such as underutilization of organs. “We want to avoid reinforcing well-meaning behaviors that are intended to lessen risk but in actuality can reduce the number of transplants.”

Formica says a simple example of risk adjustment is already made for age in relation to the current one-year post-transplant patient survival metric for kidney patients. “Looking at the age of transplantation, data shows that one-year patient survival decreases for patients 50 years and older. You would then expect patients older than 50 to have lower survival than patients younger than 50—so risk-adjustment helps account for that in the monitoring metrics.”

Risk-adjusting metrics means that transplant programs are not unduly penalized for transplanting older patients or accepting more complex organs in order to increase utilization and save more lives.

Performance monitoring needs to emphasize peer support and self-improvement

While some in the community may worry that more metrics means a greater likelihood of being identified for performance review, Formica and his colleagues strongly disagree. “I know there is concern, but our intention is not to create an environment where programs feel like they are more at risk.” He wants to reassure the community that the MPSC has designed the metrics so that there will not be an increase of interactions about performance. “The vast majority of programs are doing fine and don’t need to have any MPSC involvement,” says Formica.

This proposed framework would identify a similar number of programs for review across four metrics, and other programs who fall into what the committee calls the “yellow zone” would have optional self-improvement. There are many resources available to members that can assist them with process improvement—quality assurance and performance improvement (QAPI) services such as education on available data, process mapping and coaching sessions, peer mentoring, and collaborating through case studies.

Formica says it’s important for the transplant community to hold themselves accountable for all of the decisions they make, including not accepting organs for their patients. “The transplant waiting list is a dangerous place to be. And waitlist mortality jumps dramatically after an organ offer has been declined for a kidney candidate,” he says, pointing to research by Sumit Mohan, M.D., and others about the association between declined offers and outcomes in kidney candidates.

“I think we want to get programs to realize that doing better—and having your colleagues help you do better—means moving away from a ‘center vs. center’ framework. If we change the paradigm, then we could start working to get more organs transplanted. We’re all in this thing together.”

The MPSC released their proposal to enhance the transplant program performance monitoring system for public comment on Aug.3, 2021. All proposals this cycle are open for public comment through Sept. 30, 2021, on the OPTN public comment page. During the two-month public comment period, the community is invited to provide feedback and input about the proposed changes to transplant program performance monitoring.

Do you have questions about the MPSC performance monitoring project?

Please contact Sharon Shepherd at [email protected].