Five Strategies for More Effective Assessments of State Demonstrations

What are state and federal policymakers looking for when they invest[i] in state health care demonstration projects? Results. Policymakers need evaluation data to justify investments to legislators and other stakeholders, and to accurately assess the effectiveness of the demonstrations. However, gauging the impact of state demonstrations through monitoring and evaluation requires a great deal of time and effort from both state and federal officials: collecting and reporting data, participating in evaluators’ interviews and site visits, and reading and processing monitoring reports require considerable staff capacity.

State and federal leaders grappled with these issues at a June 2015 meeting convened by the National Academy for State Health Policy (NASHP) with the support of The Commonwealth Fund, and a number of promising strategies emerged. More information on the strategies discussed at the meeting can be found in the brief, “Determining the Impact of State Demonstrations: Considerations for State and Federal Policymakers.”

  1. State and federal officials can work cooperatively to align program goals and reporting and evaluation requirements. Communication between state and federal officials early in the demonstration process may help align state and federal program and evaluation goals, as well as help balance states’ desire for flexibility with the need for accountability.

Federal officials suggest that states approach them with a reform goal, and then work collaboratively to jointly identify the most appropriate funding mechanism. For their part, states seek the flexibility to address state-specific needs in their demonstrations.

  1. A planning period at the start of demonstrations would help states better align performance measures and evaluation criteria with the goals of the program. Currently, demonstrations are often underway in states before an outside evaluator is hired, which stymies efforts to embed evaluation into the design of the demonstration.

A planning period could help states and federal agencies align performance measures and evaluation criteria, and ensure that both are consistent with the goals of the relevant programs. It could also help states finalize implementation protocols and prepare for ramp-up, and give evaluations the nimbleness needed to adapt to early program changes.

  1. Jointly developing a meaningful set of standardized core measures could help state and federal officials streamline monitoring and evaluation. State and federal officials could work together to jointly develop a modest set of core measures that are common across demonstrations, with additional demonstration-specific measures reported separately as needed.

A set of routinely reported common measures—perhaps using the Congressionally mandated set of adult and child Medicaid core measures as a starting point–would relieve some of the burden associated with collecting and reporting different measures for different programs, and may help to better align evaluation and performance measurement with the goals of the demonstration. 

  1. Giving states timely feedback can help improve demonstrations. When evaluation reports are issued after a lengthy time lag—sometimes after demonstrations are over—states cannot use the information to help improve programs.

Timing evaluations to drive learning and change would increase evaluations’ impact on state demonstrations. The federal emphasis on rapid cycle evaluation has the potential to help states improve their programs in response to well-timed data.

  1. Designing assessments that help policymakers answer practical questions of feasibility can help spread promising practices. State officials need to know whether they can use their existing resources to replicate successful demonstrations from other states.

One state official said, “I need to know not only if a demonstration worked, but if we can actually do it. Is the demonstration part of day-to-day work, or is it an additional layer?”

Opportunities exist for states and the federal government to work collaboratively to streamline monitoring, reporting, rapid cycle assessment, and longer-term evaluation. In the wake of significant changes to the U. S. health care system, policymakers are closely watching the outcomes of federal demonstrations to see what works and what doesn’t in the realm of payment and delivery reform. The stakes are high, and state and federal officials can work together to meet the challenge.

[i]For example, the ACA appropriated $10 billion dollars for the Center for Medicare and Medicaid Innovation (CMMI), which funds innovative payment and delivery models in states and health care facilities. According to a March 6, 2015 Congressional Research Service report, CMMI was appropriated $10 billion for FY2011-FY2019 and $10 billion for each subsequent ten-year period.