Why even innovators struggle to define and measure innovation

Gabriel Perna | November 24, 2020

Measuring innovation is not easy for any organization, but it’s especially difficult for leaders in health care. Metrics around clinical and financial success are often intertwined and innovation can be completed through either technological advancements or simple process improvement. It’s hard to directly correlate “innovation” as the cause of organizational progress, although many have tried.  

“When you talk about innovation, it’s not a simple metric like quality improvement. You have to play the infinite game. You have to be in there for the long-term making impact. People who try to measure it probably don’t understand innovation. It’s a cultural change. It’s difficult to measure because it’s complex. It’s not like, ‘X numbers of dollars saved from X number of projects,” says Anthony Chang, MD, pediatric cardiologist and Chief Intelligence and Innovation Officer at CHOC Children’s Orange County.  

Yet despite this vague nature, top health care executives all have their own philosophies and theories on how they measure innovation. Some are willing to throw the word “innovation” aside and others have it as part of their core organizational beliefs.  

Health Evolution spoke to several innovation leaders about how to measure innovation, how they ensure it’s part of the culture, what’s on their innovation competency checklist and more. In part one of a two-part series, we feature responses from these two innovation leaders: 

Sara Vaezy, chief digital strategy officer at Providence, based in Seattle 

Peter Fleischut, MD, senior vice president and chief transformation officer at NewYork-Presbyterian 

Part 2 of this series.

What are some qualitative or quantitative metrics you use to measure innovation?  

Sara Vaezy: We’re not super fond of the word innovation because many times that word is a slush fund for science projects. When I say science, I don’t mean clinical research. I mean experimental things that do not have any specific, measurable ROI associated with them. Innovation becomes this extremely vague term that doesn’t yield much. And then a lot of organizations shut down their “innovation efforts.” It also introduces a lot of vagueness when you use the term innovation. Is it clinical innovation? Business innovation? Product innovation? One of the first things we did was put some parameters around what we’re talking about. 

For Providence, there’s a lot of innovation that occurs across the organization. There’s plenty of clinical innovation, research innovation, etc. [Providence Chief Digital Officer Aaron Martin]’s role, which initially was a leader of strategy and innovation across the organization, has been reframed to be much more focused around digital innovation deliverables. Even that can be a vague term, so we’re more specific around it. Digital is the efforts that we have around digital enablement and product development for our organization that are patient or consumer facing from brand discovery all the way through care delivery. We have metrics around digital transactions, engagement and enabling care delivery through digital means and tools.   

We have specific things we’re on the hook for in Providence Ventures, such as MOIC and return on capital invested. We have invested in 19 companies to date, four of which have exited and 15 that are active. Of those 15, all of them are working with us in an operational capacity throughout the organization. This all factors into how Providence Ventures is evaluated. From a digital development standpoint, our teams get rated on how much we’re driving growth, how many digital transactions we’re enabling, and how we’re promoting digital engagement. Monthly active users and metrics like that. We also have product development activities, which yield new companies and we get graded on how many companies we spin out and the notion of ROI by the equity stake we ultimately have in these companies.  

It’s a complicated way of saying, you can’t just say innovation if you have expectations on return or expectations of meaningful value for the organization, whether that’s clinical or financial. The first step is to define it and put parameters around it. We don’t have a Chief Innovation Officer, we have a Chief Digital Officer, a Chief Clinical Officer and others who own different aspects of innovation.  

Peter FleischutOne of the reasons we changed my role from Chief Innovation Officer to Chief Transformation Officer is we wanted to transform as an organization. So, this meant, for example, measuring standardization across our ten sites, whether it’s a community hospital or a large academic center. We had the same standards, the same pathways, the same protocols across the enterprise.   

When it comes to new care delivery, it was virtual … but it was what is the percentage of our total care being done virtually? What is being done remotely? What are our touch points with the patients remotely? We measure metrics based on operating where we want to be. What percentage of patients are virtual? What percent are in the remote monitoring program? And with that, financial metrics.  

It’s critical we have innovation because we want to make sure the innovation fundamentally causes us to transform. It’s led us to look at operating metrics that we measure to enable that transformation. We look at the whole organization globally and measure our effectiveness on the utilization of telemedicine. The return on telemedicine. Those types of things.  

If you surveyed innovation officers or transformation officers, they wouldn’t jump first to measuring standardization or how many order sets have been standardized. But to me, when we cover four boroughs, 10 sites of care, that sort of thing is critical when it comes to transformation. If you do that step correctly, it enables so much more. It enables you to do virtual care faster.   

There really has to be a strategic lens applied to what are you trying to accomplish, otherwise you get caught up in the very narrow day-to-day use cases.

Sara Vaezy, Providence

What are the biggest challenges you have in measuring innovation? 

Fleischut: The regulatory reimbursement environment and licensure issues are changing nonstop. When it comes to transformative efforts, there are a ton of barriers and they are changing at any given moment. Look at what happened this year. You change the ability to receive reimbursement, the regulatory restrictions, the platforms utilized, and you see a massive adoption of telemedicine. Once those barriers were lifted, it unlocked a huge amount of potential.  

Vaezy: The first big problem is that you can’t measure something that you don’t define and understand. One interesting thing is health care has a lot of data, but we have less information and fewer insights outside the clinical realm. We have tons of data but it’s very hard to access, it’s very hard to gain insight out of it and then it’s hard to design interventions to address it. I think that’s one of the biggest challenges and then innovation just turns into pet projects or sub-initiatives that don’t have actual delivery expectations.  

The other thing that’s really challenging, especially now, is we’re operating in a very low-margin environment and we sometimes over index on cost efficiency and miss other larger opportunities. You’re missing the forest for the trees. There really has to be a strategic lens applied to what are you trying to accomplish, otherwise you get caught up in the very narrow day-to-day use cases.   

How do you ensure innovation is part of the culture?  

Vaezy: First and foremost, the CEO has to be onboard. Without [Providence CEO Rod Hochman] we would never have done what we’ve been able to do. Rod and [Providence COO Mike Butler] have been the key enablers of culture supporting innovation and what we’re trying to do. And then they hire folks at senior positions to make it a reality. Then there is funding with expectations for delivery. And then the organization starts to expect something out of these groups.  

The last piece is having organizational metrics that inform compensation and incentive compensation at board level metrics. We have our leadership incentive plan. That plan has five metrics that inform the size of the leadership compensation pool for the entire organization. One of those metrics is around digital engagement. Our digital transaction target covers 20 percent of incentive compensation. Everyone in the organization rallies around that metric. That’s been an important enabler of what we do. 

Fleischut: I think it’s engagement. Measuring engagement of employees is the most important metric. In challenging times, dealing with the stressors of the pandemic, engagement is critical. Having an engaged workforce that understands what’s being done is critical. Overall, employee engagement is the biggest indicator of success. 

If you were to create a checklist on competencies for innovation readiness—what’s on it? 

FleischutFlexibility, being nimble, and the culture. At the end of the day, culture is the most important factor. When we started investing in telemedicine years ago, people would ask about the technology and the platform we were using. While that’s important, it’s such a small percentage of success. It’s really about the people and process.  

Vaezy: Everything we’ve talked about is at the top of the list: Having the right leadership, defining the right parameters, creating the culture, etc. But also do you understand the long-term role of innovation as it relates to organization? That will inform how you think about it. Are you thinking about it from a market-based perspective? We do because we have an ultimate goal of commercializing technology in addition to enabling our own health system. That perspective introduces a lot of complexity because there’s a healthy tension between serving Providence and building out a product for the market. You need to have clarity around those things in order to move forward because you’re stuck in limbo. Clarity around what you’re trying to achieve through this work is a big part of readiness, otherwise you’ll be trying a bunch of random stuff and you won’t be heading in the right direction.  

Lastly, are you hiring the right people? How does your talent acquisition strategy fit into all this? We’re lucky we’re in Seattle and we have this huge pool of talent. We can do product development and digital innovation. If we were elsewhere and we didn’t have that pool of talent, we would have to get more creative in recruiting and retaining talent. Evaluating your market and what kinds of resources are available in your market is really important.  

Part two of this series will be published next week 

About the Author

Gabriel Perna, Senior Manager, Digital Content

Gabriel Perna is the Senior Manager of Digital Content at Health Evolution. He brings 10+ years of experience in covering the intersection of health care and business. Previously, he was at Chief Executive, Physicians Practice and Healthcare Informatics. You can reach him via email or on Twitter at @GabrielSPerna